tfmotu commented 2 years ago

Hi! firstly let me say you thanks. This is a good solution to me. Since last month my script do not work, some errors are shown and I not sure if here is an issue. Here go the logging information from geckodriver and from python execution. Thanks in advance. Python script logging info:

File "/usr/local/lib/python3.9/dist-packages/yt_videos_list/logic.py", line 134, in load_page wait.until(EC.element_to_be_clickable((By.XPATH, channel_heading_xpath))) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/wait.py", line 71, in until value = method(self._driver) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 297, in call element = visibility_of_element_located(self.locator)(driver) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 128, in call return _element_if_visible(_find_element(driver, self.locator)) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 415, in _find_element raise e File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 411, in _find_element return driver.find_element(*by) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element return self.execute(Command.FIND_ELEMENT, { File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/errorhandler.py", line 241, in check_response raise exception_class(message, screen, stacktrace, alert_text) selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: None Message: Dismissed user prompt dialog: This site is asking you to sign in.

geckodriver logging info: 1643352442766 geckodriver INFO Listening on 127.0.0.1:38755 1643352442786 mozrunner::runner INFO Running command: "/usr/bin/firefox" "--marionette" "-headless" "-no-remote" "-profile" "/tmp/rust_mozprofilea68MXZ" *** You are running in headless mode. [GFX1-]: glxtest: Unable to open a connection to the X server [GFX1-]: glxtest: libEGL initialize failed [GFX1-]: No GPUs detected via PCI 1643352446042 Marionette INFO Marionette enabled [GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /tmp/rust_mozprofilea68MXZ/search.json.mozlz4", (void 0))) 1643352452123 Marionette INFO Listening on port 43291 1643352452302 RemoteAgent WARN TLS certificate errors will be ignored for this session 1643352456889 Marionette WARN Ignoring event 'pageshow' because document has an invalid readyState of 'uninitialized'. 1643352460748 Marionette INFO Stopped listening on port 43291 JavaScript error: resource:///modules/Interactions.jsm, line 209: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIUserIdleService.removeIdleObserver]

tfmotu commented 2 years ago

HI, I'm not sure but there is an attempt to sign in that I suppose that it is not resolved... See attached image. The error and the screen is shown after cookie consent page. If I access to the page directly no sing in page is shown. Thanks again. error_signin

shailshouryya commented 2 years ago

Hi!

Thanks for filing this issue! I noticed a similar problem with the program clicking the wrong button after being redirected to the cookie consent page a while back, and added a change to address the issue in commit 62464aa9cf803c8f8b45bbffcb748f691595bf1c. However, in my case the program was clicking a button to select the display language (English, Spanish, etc.) and the program would time out on the cookie consent page itself (it did not even get past the "Submit" part). The change made above fixed that issue, but the problem you're facing seems slightly different.

In any case, it might be worth it to try running the program with the updated code locally (this change was made after the last release, so it will be available in release 0.6.4) to see if the change addresses your problem too. After you clone/fork the repo and have it locally, try running

cd yt-vidoes-list/python
pip3 install .           # should install changes from local code instead of using code from release 0.6.3

then try rerunning the program again. If this fixes the problem, then everything should be okay (these changes will eventually be available with the PyPI package in release 0.6.4), but if this does not fix the problem, you could try checking out the links I found below (included below the first horizontal line divide).

Another simple workaround could be to set the cookie_consent argument to True during initialization, and see if this fixes the error:

from yt_videos_list import ListCreator
lc = ListCreator(driver='firefox', cookie_consent=True, scroll_pause_time=0.8)
lc.create_list_for(url='https://www.youtube.com/channel/UC8butISFwT-Wl7EV0hUK0BQ')

This technically does not address the error you're seeing since the program will just accepts cookies instead of blocking them (which is the default behavior), but this might be good enough to unblock you temporarily until we figure out a proper fix.

I haven't seen the specific error or stack trace that you included, so I looked up potential causes and solutions and found a couple other posts that faced a similar issue. You might have already seen these, but I'll include them here just in case so that if you missed any of them, or if someone else is facing a similar issue, they can also see the posts and use them as a starting point for debugging:

Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIWindowsRegKey.open]. How to resolve this error please
- slightly different cause of error here: nsIWindowsRegKey.open instead of nsIUserIdleService.removeIdleObserver
- suggests doing a clean reinstall by deleting the Firefox program folder before (re)installing a fresh copy of the current Firefox release
- warns about losing personal data if Firefox profile folder is removed (post seems to suggest deleting just the program folder, and not the profile folder) - check out post for more info
0x80004005 (NS_ERROR_FAILURE) [nsIDOMHTMLFormElement.submit]
- also slightly different cause of error here: nsIDOMHTMLFormElement.submit instead of nsIUserIdleService.removeIdleObserver
- suggests checking for a pop-up blocker and addressing the pop-up
- code in the post is in Java, but a potential way to handle something like this in python would be to run something like the following (taken from here)
```
### if the pop-up is a new window ###
driver.switch_to.window(driver.window_handles[1])
driver.close() # closes currently active window
driver.switch_to.window(driver.window_handles[0]) # switch back to main window
```

if the pop-up is more like a text box overlaid on top of the page

driver.switch_to.alert.accept()

this might address the following part of your stack trace

selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: None Message: Dismissed user prompt dialog: This site is asking you to sign in.

- [GeckoDriver is unable to execute AutoIT script](https://github.com/mozilla/geckodriver/issues/1915)
  - cause of error ***is actually*** a match here: `nsIUserIdleService.removeIdleObserver`
  - however, the poster seems to be trying to run AutoIT, which is a [Windows specific scripting language](https://www.autoitscript.com/site/autoit/), and did not post any follow-up information, so the thread is kind of limited
- Another post I found, but probably not as relevant here:
  - [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIXMLHttpRequest.send]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrome://bolototoolbar/content/soapclient.js :: anonymous :: line 89" data: no\]](https://answers.microsoft.com/en-us/windows/forum/all/exception-component-returned-failure-code/7bd2e208-c1f1-4948-b567-795fb5faeb99)
    - different cause of error here as well: `nsIXMLHttpRequest.send` instead of `nsIUserIdleService.removeIdleObserver`
    - thread does not really have much debugging info, but links to [0x80004005 (NS ERROR FAILURE) and other firefox errors](https://helpful.knobs-dials.com/index.php/0x80004005_(NS_ERROR_FAILURE)_and_other_firefox_errors), which ***does*** have some useful debugging tips

Searching for `nsIUserIdleService.removeIdleObserver` did not return many relevant posts, but here are the results from different places:
- [DuckDuckGo](https://duckduckgo.com/?q=nsIUserIdleService.removeIdleObserver) (seems to have the closest matches)
- [Bing](https://www.bing.com/search?q=nsIUserIdleService.removeIdleObserver) (similar to results above)
- [Google](https://www.google.com/search?hl=en&q=nsIUserIdleService.removeIdleObserver)

<br>
<br>
<br>

---

If none of the above works, it might be worth debugging more and looking at the full stack trace - the code right now includes just the error message and hides the full stack trace for the original error, so you see

===>ERROR!<===\

There was a problem running the selenium webdriver!

Try running the program again with headless=False if you are currently running the program with headless=True

For further debugging, this was the exact error message (might also be blank): Alert Text: None Message: Dismissed user prompt dialog: This site is asking you to sign in.

the following was NOT in your attached image, but I think it shows up in the terminal console

File "/usr/local/lib/python3.9/dist-packages/yt_videos_list/logic.py", line 134, in load_page wait.until(EC.element_to_be_clickable((By.XPATH, channel_heading_xpath))) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/wait.py", line 71, in until value = method(self._driver) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 297, in call element = visibility_of_element_located(self.locator)(driver) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 128, in call return _element_if_visible(_find_element(driver, self.locator)) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 415, in _find_element raise e File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 411, in _find_element return driver.find_element(*by) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element return self.execute(Command.FIND_ELEMENT, { File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/errorhandler.py", line 241, in check_response raise exception_class(message, screen, stacktrace, alert_text) selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: None Message: Dismissed user prompt dialog: This site is asking you to sign in.



You did already include part of the stack trace under the "Python script logging info" part of your post (the last part in the code snippet above), but based on how I wrote the error handling, the stack trace (currently) shows only the **most recent error** that causes the program to crash, instead of the whole chain of errors (if there were multiple errors). For example, if there was an error that caused another error, and that error caused another error, I ***think*** the program only shows the last of the 3 errors, instead of all 3 errors.

Hopefully something here helps you debug the problem here, but in case nothing works, I'll add a change to see the full stack trace soon. I didn't know how to properly `raise` an exception `from` the original exception when I initially wrote the error handling logic, but I'll push up a change that fixes this soon and post an update here when I do.

tfmotu commented 2 years ago

Hi slow-but-steady, thanks for your time and great response. I test the new commit but sadly the error is there, and the other workaround does not solve the problem, I always use cookie_consent to True. I will wait to the next release in order to check it. As alternative path I will try to use a fresh container to try it again...At this moment all the work is done by a raspberry pi. I use your lib in my home in order to download the urls of youtube channels and users in order to block them with a squid proxy (Squid-in-the-middle SSL Bump). At this moment there are more that 100000 urls blocked ....my daughters will be happy if it does not work, ha,ha,ha... Again thanks for your work and for share it!

Only for info, here is the error after change, I think that is the same: =======> NOTE <=======

You are using yt_videos_list package version >= 0.6.0 (currently 0.6.3).

There are some formatting changes to the output files as of release 0.6.0 that may potentially interfere with your workflow, so please review https://github.com/slow-but-steady/yt-videos-list/releases/tag/v0.6.0 to see if you need to do anything.

There are also some changes to the return value from the create_list_from() method in release 0.6.1 that may also potentially interfere with your workflow, so please review https://github.com/slow-but-steady/yt-videos-list/releases/tag/v0.6.1 to see if you need to do anything.

Thanks!

Redirected to consent.youtube.com! Accepting consent for all cookies since program running with cookie_consent=True ...

Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/yt_videos_list/logic.py", line 131, in load_page wait.until(EC.element_to_be_clickable((By.XPATH, channel_heading_xpath))) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/wait.py", line 71, in until value = method(self._driver) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 297, in call element = visibility_of_element_located(self.locator)(driver) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 128, in call return _element_if_visible(_find_element(driver, self.locator)) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 415, in _find_element raise e File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/support/expected_conditions.py", line 411, in _find_element return driver.find_element(*by) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element return self.execute(Command.FIND_ELEMENT, { File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/errorhandler.py", line 241, in check_response raise exception_class(message, screen, stacktrace, alert_text) selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: None Message: Dismissed user prompt dialog: This site is asking you to sign in.

===>ERROR!<=== There was a problem running the selenium webdriver!

Try running the program again with headless=False if you are currently running the program with headless=True.

For further debugging, this was the exact error message (might also be blank): Alert Text: None Message: Dismissed user prompt dialog: This site is asking you to sign in.

shailshouryya / yt-videos-list

selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: None #17

if the pop-up is more like a text box overlaid on top of the page

this might address the following part of your stack trace

the following was NOT in your attached image, but I think it shows up in the terminal console