sebdelsol
commented
2 years ago No it's not :
- Headless is often detected, there's a disclaimer in the readme about that. You're lucky it works in your case... for now.
- Don't spoof your user agent, it's very easily detected especially OS and browser discrepancies.
- Refrain from using all those unneeded options : you don't want a broken and/or detectable config.
- If not headless you should keep
--start-maximizedthough because it looks less suspicious to have a maximized viewport... unfortunately you spelled it wrong. - Please learn how to use xpath instead of copying unreliable absolute path from the Chrome DevTools.
EDIT :
- Please use Selenium explicit waits it's more efficient.
My test code, it scrapped those 1000 items without a hitch :
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
import undetected_chromedriver as uc
if __name__ == "__main__":
options = uc.ChromeOptions()
options.headless = True # you're lucky headless works for this site... for now
driver = uc.Chrome(options=options)
wait = WebDriverWait(driver, 5)
url = "https://spb.vseinstrumenti.ru/instrument/shurupoverty/akkumulyatornye-dreli/"
x_items = '//div[@class="listing-grid"][1]//div[contains(@class, "product-tile grid-item")]'
x_item_infos = '//div[@class="column-right"]'
x_item_available = './/ul[contains(@class, "product-delivery")]/li[1]/span/span[1]'
x_item_href = './/div[@class="image"]/a'
x_item_name = './/div[@class="title"]'
no_page = 1
page = ""
while True:
driver.get(f"{url}{page}")
try:
items = wait.until(EC.presence_of_all_elements_located((By.XPATH, x_items)))
except TimeoutException:
break
for i, (href, name) in enumerate(
[
(
item.find_element(By.XPATH, x_item_href).get_attribute("href"),
item.find_element(By.XPATH, x_item_name).text,
)
for item in items
]
):
driver.get(href)
infos = wait.until(EC.presence_of_element_located((By.XPATH, x_item_infos)))
available = infos.find_elements(By.XPATH, x_item_available)
available = available[0].text if available else "not available"
print(f"{no_page}-{i} - {name} - {available}")
no_page += 1
page = f"page{no_page}/"
driver.quit()
realdronos
Hello, there is a server on ubuntu with python and UC + google chrome. Until the last browser update, the script on the server was working fine. After the update the script began to give an error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[6]/div[3]/div/div/div/div/main/div/div[2]/div[3]/div/div/div/ul/li[3]"} (Session info: headless chrome=103.0.5060.53)
Put headless False and took screenshot - white screen. I suspect that the site began to detect UC. As a solution I want to downgrade Chrome to a previous version, so the question is where to find previous versions of uUC and how to install correct version?
Current versions: undetected-chromedriver - 3.1.5.post4 selenium - 4.1.3 Google Chrome - 103.0.5060.53 chromedriver - https://chromedriver.storage.googleapis.com/index.html?path=103.0.5060.53/
ps. Script on the computer starts and runs without any problems.
Code for test: