Closed STARKILLER-1 closed 10 months ago
Greetings. Let's go through your code and give you some solutions to work with...
You've got multiple recursive calls happening with initialize_driver()
, which is certainly going to cause various issues. (See the code below, where I highlighted it in two places.)
The officially-supported way of doing multithreading in UC Mode is by using pytest
(via pytest-dist
). Since it looks like you're setting multiple simultaneous proxies, you'll need to use --multi-proxy
/ multi_proxy=True
with that. (See https://github.com/seleniumbase/SeleniumBase/issues/1832)
Depending on your Ubuntu configuration, you may need to use --xvfb
/ xvfb=True
instead of using headless mode. Headless mode is the default on Ubuntu, but there is a --headed
/ headed=True
option to override.
You may be overcomplicating your selenium-stealth
setup. You can integrate it directly into the custom setUp()
method of your tests like this: (A BaseCase
format lets you use proper multithreading with pytest
/ pytest-xdist
.)
from seleniumbase import BaseCase
from selenium_stealth import stealth
class BaseTestCase(BaseCase):
def setUp(self):
super().setUp()
stealth(self.driver,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
webgl_vendor="Intel Inc.",
renderer="Intel Iris OpenGL Engine",
fix_hairline=True,
)
Then have your test classes inherit BaseTestCase
instead of BaseCase
. (See SeleniumBase/help_docs/syntax_formats.md#sb_sf_02)
SeleniumBase methods have automatic-waiting. You should never be using the external implicitly_wait
, WebDriverWait
, or EC.presence_of_element_located
anywhere in your code. Use the built-in methods instead. For the raw driver
formats, see examples such as SeleniumBase/examples/raw_login_driver.py, SeleniumBase/examples/raw_driver_manager.py, and SeleniumBase/examples/offline_examples/test_extended_driver.py. For the pytest
formats, see any example test that starts with test_
or ends with _test
in the SeleniumBase/examples folder.
Also note that you have unused code in your examples. You declared options
with options = webdriver.ChromeOptions()
, but never used it. You shouldn't be using that at all since SeleniumBase options are passed via pytest
command-line args, or via method args, eg. driver = Driver(uc=True)
.
Finally, selenium-stealth
is an external repo with its own issues. SeleniumBase won't magically fix those issues if you encounter them while using both frameworks together. To determine between the two, try examples with just SeleniumBase (but not selenium-stealth), and see if those issues still happen.
@mdmintz I found the issue point driver = sb.Driver(browser='chrome', headless=False, uc=True, proxy=proxy)# not working on ubuntu 22.04 driver = sb.Driver(browser='chrome', headless=False, uc=True)# working on ubuntu 22.04 the question was about the proxy with auth (definitely working) but the raw driver cannot connect to them. Somehow "proxy =" argument is not working on ubuntu (selenium-wire works with it. But it's not so good as SeleniumBase =) Do you know easy way to connect SeleniumBase raw Driver to proxy?
Are your proxy strings in a valid format?
host:port
or user:pass@host:port
?
Are you using multithreading / multiple proxies? (Then use pytest
and multi_proxy=True
)
from parameterized import parameterized
from seleniumbase import BaseCase
BaseCase.main(__name__, __file__, "-n3")
class ProxyTests(BaseCase):
@parameterized.expand(
[
["user1:pass1@host1:port1"],
["user2:pass2@host2:port2"],
["user3:pass3@host3:port3"],
]
)
def test_multiple_proxies(self, proxy_string):
self.get_new_driver(
undetectable=True, proxy=proxy_string, multi_proxy=True
)
self.driver.get("https://browserleaks.com/webrtc")
self.sleep(30)
Or if not using multithreading and not using multiple proxies at the same time, here's a way to do it using the Driver()
:
driver = Driver(uc=True, proxy="user:pass@host:port")
@mdmintz Nope, just a single driver at a time My proxies are strings Example_proxy="XXXX:XXXX@XX.XX.XX.XXX:8000" I'm confused.. Never had issues with those proxies.
Do you have a stack trace? It'll make it easier to debug.
Ok, give me a minute, i will make a sample script and test it
XXXXXXXXXXXXXXX/test# python3 main.py
Proxy: XXXXXX:XXXXX@XX.XX.XX.237:8000
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/browser_launcher.py", line 3342, in get_local_driver
driver = undetected.Chrome(
File "/usr/local/lib/python3.10/dist-packages/seleniumbase/undetected/__init__.py", line 305, in __init__
super().__init__(options=options, service=service_)
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
super().__init__(
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py", line 56, in __init__
super().__init__(
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 205, in __init__
self.start_session(capabilities)
File "/usr/local/lib/python3.10/dist-packages/seleniumbase/undetected/__init__.py", line 430, in start_session
super().start_session(capabilities)
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 289, in start_session
response = self.execute(Command.NEW_SESSION, caps)["value"]
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 344, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:9222
from chrome not reachable
Stacktrace:
#0 0x5598162d2933 <unknown>
#1 0x559815fac546 <unknown>
#2 0x559815f9821a <unknown>
#3 0x559815fe372c <unknown>
#4 0x559815fdad59 <unknown>
#5 0x55981601e77a <unknown>
#6 0x5598160153d3 <unknown>
#7 0x559815fe7e64 <unknown>
#8 0x559815fe8c4e <unknown>
#9 0x559816298558 <unknown>
#10 0x55981629c4a0 <unknown>
#11 0x5598162a697c <unknown>
#12 0x55981629d0b8 <unknown>
#13 0x559816268cdf <unknown>
#14 0x5598162c1048 <unknown>
#15 0x5598162c1219 <unknown>
#16 0x5598162d1ac3 <unknown>
#17 0x7f31683c3ac3 <unknown>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/champcommerce/test/main.py", line 38, in initialize_driver
driver = sb.Driver(browser='chrome', headless=False, uc=True, proxy=proxy)
File "/usr/local/lib/python3.10/dist-packages/seleniumbase/plugins/driver_manager.py", line 453, in Driver
driver = browser_launcher.get_driver(
File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/browser_launcher.py", line 1524, in get_driver
return get_local_driver(
File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/browser_launcher.py", line 3702, in get_local_driver
driver = webdriver.Chrome(service=service)
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
super().__init__(
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py", line 56, in __init__
super().__init__(
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 205, in __init__
self.start_session(capabilities)
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 289, in start_session
response = self.execute(Command.NEW_SESSION, caps)["value"]
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 344, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
(session not created: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
#0 0x55a6bd2b8933 <unknown>
#1 0x55a6bcf926f7 <unknown>
#2 0x55a6bcfc5a55 <unknown>
#3 0x55a6bcfc1bbc <unknown>
#4 0x55a6bd00477a <unknown>
#5 0x55a6bcffb3d3 <unknown>
#6 0x55a6bcfcde64 <unknown>
#7 0x55a6bcfcec4e <unknown>
#8 0x55a6bd27e558 <unknown>
#9 0x55a6bd2824a0 <unknown>
#10 0x55a6bd28c97c <unknown>
#11 0x55a6bd2830b8 <unknown>
#12 0x55a6bd24ecdf <unknown>
#13 0x55a6bd2a7048 <unknown>
#14 0x55a6bd2a7219 <unknown>
#15 0x55a6bd2b7ac3 <unknown>
#16 0x7f5ccfb19ac3 <unknown>
def initialize_driver(account_number, cookies_file, url, proxy, attempt=1):
if attempt > 5:
return None
try:
print(f"Proxy: {proxy}")
driver = sb.Driver(browser='chrome', headless=False, uc=True, proxy=proxy)
driver.get(url)
time.sleep(3)
with open(cookies_file, 'r') as f:
cookies = json.load(f)
current_domain = driver.current_url # get the current domain
for cookie in cookies:
if 'sameSite' in cookie and cookie['sameSite'] not in ['Strict', 'Lax', 'None']:
del cookie['sameSite']
if 'domain' in cookie:
if current_domain.find(cookie['domain']) != -1:
driver.add_cookie(cookie)
driver.refresh()
time.sleep(3)
try:
wait = WebDriverWait(driver, 2)
wait.until(EC.presence_of_element_located((By.XPATH, "//*[contains(normalize-space(), 'Enter or login in')]")))
if driver:
driver.quit()
return initialize_driver(account_number, cookies_file, url, proxy, attempt + 1)
except Exception as e:
send_telegram_notification(e)
return driver
except Exception as e:
send_telegram_notification(e)
if driver:
driver.quit()
return None
Same simple raw driver
I think you missed a few important parts from my original answer:
Depending on your Ubuntu configuration, you may need to use --xvfb / xvfb=True instead of using headless mode. Headless mode is the default on Ubuntu, but there is a --headed / headed=True option to override.
So, don't use headless=False
. Try xvfb=True
instead. And if that doesn't work, headed=True
.
And your script still has a recursive call in it, which could lead to multiple drivers, which may crash because you weren't using pytest
to spin up your multiple browsers.
And remember that SeleniumBase methods have automatic-waiting. You should never be using the external implicitly_wait, WebDriverWait, or EC.presence_of_element_located anywhere in your code. Use the built-in methods instead.
I have made some tests with print. Turning on proxy kills further script running.It freezes for a while and then gives an error =( I will make all your recommendations, after script goes further then driver = sb.Driver(browser='chrome', headless=False, uc=True, proxy=proxy) And i'm sorry, I copied the code incorrectly headless =True (I have no gui) Thank you! I can give you my proxy for test, if you want
Chrome headless mode doesn't support extensions. If you have to run on a headless machine, use headed=True
or xvfb=True
. (By default, it was probably using headless2=True
, which is Chrome's newer headless mode.)
I understand, but without proxy headless works. Sorry if I misunderstood
OMG. It worked! driver = sb.Driver(browser='chrome', headless2=True, uc=True, proxy=proxy)
Yes, because proxy settings are set via a Chrome extension that gets generated at runtime. If you don't use proxy with auth, then you don't have a Chrome extension, and therefore regular headless mode works fine. (Hence the reason for Chrome's newer headless mode, which is set with headless2
.)
There's a popular StackOverflow post about it: https://stackoverflow.com/a/73840130/7058266
@mdmintz Thank you so much! I hope you won't see me with my stupid questions soon=). I will rewrite all my code from selenium-wire to your framework, I like your Tracebacks they are so detailed and helpful
There's a popular StackOverflow post about it: https://stackoverflow.com/a/73840130/7058266
Yes i used it once but in original selenium(some elements were not displayed on screenshots via old headless mode). Gosh...
There's a popular StackOverflow post about it: https://stackoverflow.com/a/73840130/7058266
driver = sb.Driver(browser='chrome', headless2=True, uc=True, proxy=proxy) is unstable and works 1/10 times =( UPD: 1/100 headed=True- not working xvfb=True - not working headless2=True- not working
After a quick refresher, I recalled that the Driver()
format doesn't have the Xvfb launcher for Linux like the other SeleniumBase formats. You'll need to add that in separately: (https://github.com/mdmintz/sbVirtualDisplay)
from sbvirtualdisplay import Display
from seleniumbase import Driver
display = Display(visible=0, size=(1440, 1880))
display.start()
driver = Driver()
try:
driver.open("seleniumbase.github.io/demo_page")
driver.highlight("h2")
driver.type("#myTextInput", "Automation")
driver.click("#checkBox1")
driver.highlight("img", loops=6)
finally:
driver.quit()
display.stop()
Thank you. This is a stable solution
I'm sorry for may be being stupid, but..
Bug only on ubuntu 22.04. On Windows 11 no issues!
I have been struggling with this problem for several days:
THIS CODE WORKS ON UBUNTU 22.04:
THIS CODE ALWAYS FAILS ON UBUNTU 22.04:
I was debugging alot. On Windows 11 no issues! Only this code can go through Cloudflare on a target site
Thank you!