seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
5.3k stars 973 forks source link

How to access Browser from host when using Docker? #3010

Closed ttraxxrepo closed 2 months ago

ttraxxrepo commented 2 months ago

First off - very cool project!

I took a look at the docker docs here https://seleniumbase.io/integrations/docker/ReadMe/#1-install-the-docker-desktop and was able to spin up the container and pass the test.

How do I go about accessing the GUI and running in non-headless mode so I can see the browser? …is there a port that just needs to be mapped and then accessible at hostip:port ?

thanks in advance!

mdmintz commented 2 months ago

Thank you. There's the Remote Debugger: (--remote-debug / remote_debug=True)

https://github.com/seleniumbase/SeleniumBase/blob/9981b5f690712c3bb919ec75f5bad7c917601506/README.md?plain=1#L687

Then navigate to chrome://inspect/#devices to see the device. I'm not sure how well that's accessible when using Docker. There may be better solutions available on StackOverflow.

ttraxxrepo commented 2 months ago

@mdmintz thanks for the quick response - what exactly is this chrome debugger?

is that a server (with a default port?) that gets started and then accessed via browser? …a vnc server?

mdmintz commented 2 months ago

https://developer.chrome.com/docs/devtools/remote-debugging/local-server

ttraxxrepo commented 2 months ago

Thanks @mdmintz I actually misrepresented my question - I'm trying to access the browser, not the GUI.

Found another issue https://github.com/seleniumbase/SeleniumBase/discussions/2926 discussing a similar topic, but the current approach there is not working

mdmintz commented 2 months ago

Run a script from the Docker shell, but first add a breakpoint() in there so that you can take control of the browser from the command-line.

ttraxxrepo commented 2 months ago

Apologies for all the questions on this but I'm still struggling with this and have been at it for a few days now. I've gone through all the documentation but can't seem to find a straightforward of example of how to get to the browser through docker.

Could I please kindly request an update to https://seleniumbase.io/integrations/docker/ReadMe/ showing how to do this?

mdmintz commented 2 months ago

There's nothing Docker-specific about breakpoint() / Pdb debugging. The Docker ReadMe explains how to "enter Docker and stay inside the shell" - docker run -i -t seleniumbase (in SeleniumBase/integrations/docker). Then use standard Python debugging to pause your script at a certain point. Then you can control the browser from the command-line (although you might not be able to see it).

ttraxxrepo commented 2 months ago

I don't have an issue with using breakpoint() or controlling the browser via docker commands. The issue I am having is how to actually view the browser when I am running in docker via xvfb.

So the question is how can I see the browser that is running in docker (i.e. through noVNC) on my host machine?

Selenium grid does this where you can see click the session video icon in a browser on the local machine so view the browser thats running in the node, however I cannot use selenium grid because there is no support for UC mode

mdmintz commented 2 months ago

I save screenshots and then view them through the file viewer in Docker Desktop.

Go to Containers -> Files.

Screenshot 2024-08-13 at 9 32 01 AM

It's a 2-step process:

  1. Use the Python debugger to control the browser / save screenshots.
  2. Use Docker Desktop to view the screenshots.

Maybe there's an easier way.

This situation is likely the same for regular Selenium / Docker users, so this isn't a SeleniumBase-specific task.

ttraxxrepo commented 2 months ago

Thanks @mdmintz - while it may be ideal to have a live view, I think the screenshot approach should be enough for me to at least understand what's going on in the browser when there is an error so I can fix the test.

I am trying to run in test mode (and in docker) but it seems like the auto-screenshots are not working. Could you please help me understand below why the screenshot cannot be taken?

To be clear, I understand the error I am getting in my script, and probably need to add more time to wait for page to load, etc. but I am just curious why the screenshot is not able to be taken (WARNING: Unable to get screenshot!)

Also, what is the directory that the screenshots are saved to, and is there a way to get the filename into a variable when it's taken? I'm thinking of setting up some type of callback where when theres a failed test and a screenshot taken, sending a ping to telegram/discord with the screenshot embedded so I can see right away what happened.

Also I'd ideally mount the directory to host so I don't have to go into docker file viewer every time

thanks again

    with SB(uc=True, incognito=True, test=True, slow=True, demo_sleep=3) as sb:
        sb.uc_open_with_reconnect(url)
        sb.uc_click('a[href="/users/accounttype"]')
        sb.uc_click('a[href="/users/login"]')
        sb.type("input#sc-login-username", username)
        sb.type("input#sc-login-password", password)
        sb.uc_click("button.btnLogin")
        # Assert that we are logged in
        sb.assert_element("div.sc-brand")
        logging.info("Logged in successfully")
========================= {threading.py:973:SB} starts =========================
WARNING: Unable to get screenshot!
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/plugins/sb_manager.py", line 995, in SB
    yield sb
  File "/bots/scrape.py", line 19, in run
    self.login(sb)
  File "/bots/scrape.py", line 34, in login
    sb.uc_click('a[href="/users/login"]')
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/browser_launcher.py", line 4521, in <lambda>
    driver.uc_click = lambda *args, **kwargs: uc_click(
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/browser_launcher.py", line 537, in uc_click
    element = driver.wait_for_selector(selector, by=by, timeout=timeout)
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/sb_driver.py", line 117, in wait_for_selector
    return page_actions.wait_for_selector(self.driver, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/fixtures/page_actions.py", line 1790, in wait_for_selector
    return wait_for_element_present(
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/fixtures/page_actions.py", line 429, in wait_for_element_present
    timeout_exception(NoSuchElementException, message)
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/fixtures/page_actions.py", line 254, in timeout_exception
    raise exc(msg)
seleniumbase.common.exceptions.NoSuchElementException: Message:
 Element {a[href="/users/login"]} was not present after 7 seconds!
=================== {threading.py:973:SB} failed in 3007.46s ===================