Headless usage on debian?

johns224 commented 6 years ago

Hi,

I'm trying to get xfinity-usage to run on a headless debian box. I have the various versions of things listed below. I do have the chromedriver and geckodriver in my path, and I know headlessness is probably the core of the problem here, but the error messages aren't giving me much to go on. I'd appreciate any help you can give.

Thanks,

Ross

Version

xfinity-usage 2.0.2 https://github.com/jantman/xfinity-usage

Installation Method

pip install xfinity-usage

Supporting Software Versions

ross@loki:~$ uname -v
#1 SMP Debian 3.16.7-ckt11-1+deb8u5 (2015-10-09)
ross@loki:~$ firefox --version
Mozilla Firefox 52.6.0
ross@loki:~$ chrome --version
Chromium 67.0.3372.0
ross@loki:~$ chromedriver --version
ChromeDriver 2.32 (undefined)
ross@loki:~$ geckodriver --version
geckodriver 0.19.1
ross@loki:~$ firefox --version
Mozilla Firefox 52.6.0
ross@loki:~$ chrome --version
Chromium 67.0.3372.0
ross@loki:~$ python --version
Python 2.7.12+

Actual Output

ross@loki:~$ xfinity-usage -b firefox
Traceback (most recent call last):
  File "/usr/local/bin/xfinity-usage", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 618, in main
    res = script.run()
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 119, in run
    self.browser = self.get_browser()
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 342, in get_browser
    browser = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 162, in __init__
    keep_alive=True)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status: 1

ross@loki:~$ xfinity-usage -b chrome
Traceback (most recent call last):
  File "/usr/local/bin/xfinity-usage", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 618, in main
    res = script.run()
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 119, in run
    self.browser = self.get_browser()
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 345, in get_browser
    browser = webdriver.Chrome()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
    desired_capabilities=desired_capabilities)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.32 (undefined),platform=Linux 3.16.0-4-amd64 x86_64)

ross@loki:~$ xfinity-usage -b chrome-headless
Traceback (most recent call last):
  File "/usr/local/bin/xfinity-usage", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 618, in main
    res = script.run()
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 119, in run
    self.browser = self.get_browser()
  File "/usr/local/lib/python2.7/dist-packages/xfinity_usage/xfinity_usage.py", line 370, in get_browser
    browser.set_window_size(1024, 768)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 1071, in set_window_size
    'windowHandle': windowHandle})
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: unable to connect to renderer
  (Session info: headless chrome=62.0.3202.89)
  (Driver info: chromedriver=2.32 (undefined),platform=Linux 3.16.0-4-amd64 x86_64)

jantman commented 6 years ago

@johns224 I haven't run this on a truly headless machine myself, but that's unusal, and I certainly consider it a bug if something requires a GUI/X server to run (this is one of my few projects that I actually run directly on my desktop).

I'll be able to look into this more later today, but my initial thoughts:

I've never tried the Firefox support. I have it in there for people who want to use it, but I haven't tried it and I'm pretty sure it requires X.
I know that chrome requires X.
chrome-headless is what I use (via cron). It should work, but I notice that you said you have Chromium 67 and chromedriver 2.32. According to the chromedriver downloads page that version is far too old. The current downloads page says that the latest chromedriver version, 2.36, only supports chrome 63-65. I don't know if it works with 67 or not, but I do know that you'll need at least the newest version of chromedriver.

If you could try using a chromium and chromedriver version that are known to be compatible, that should at least help.

billimek commented 6 years ago

@johns224 I'm facing the same issue (in docker) and searching for a good headless solution. Will update here if I find something and if you find something could you please report back as well?

johns224 commented 6 years ago

Thanks! Updating Chrome did it! Works like a charm with chrome-headless now:

ross@loki:~/bin$ chrome --version
Google Chrome 65.0.3325.162
ross@loki:~/bin$ chromedriver --version
ChromeDriver 2.36.540471 (9c759b81a907e70363c6312294d30b6ccccc2752)
ross@loki:~/bin$ xfinity-usage -b chrome-headless
Used XXX of 1024 GB this month.

jantman commented 6 years ago

Ok, I'm glad that worked for you @johns224

@billimek I haven't tried anything like this in Docker myself (that I can think of), but I know there are a few resources on Google that might help...

billimek commented 6 years ago

My issue is actually different, sorry for the confusion.

Mine is that it throws the following error (using the default phatomjs driver):

selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with xpath '//*[@ng-bind-html=\"usage.details.userMessage.monthlyUsageState\"]'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"150","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:43757","User-Agent":"Python http auth"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"xpath\", \"value\": \"//*[@ng-bind-html=\\\"usage.details.userMessage.monthlyUsageState\\\"]\", \"sessionId\": \"92448a70-29f2-11e8-969a-a7e0c5892c85\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/92448a70-29f2-11e8-969a-a7e0c5892c85/element"}}

ghostdriver.log shows:

[INFO  - 2018-03-17T14:51:53.172Z] GhostDriver - Main - running on port 35473
[INFO  - 2018-03-17T14:51:54.119Z] Session [bb9bbb50-29f2-11e8-ac31-ef03425707b6] - page.settings - {"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 xfinity-usage/2.0.2","webSecurityEnabled":false}
[INFO  - 2018-03-17T14:51:54.119Z] Session [bb9bbb50-29f2-11e8-ac31-ef03425707b6] - page.customHeaders:  - {}
[INFO  - 2018-03-17T14:51:54.119Z] Session [bb9bbb50-29f2-11e8-ac31-ef03425707b6] - Session.negotiatedCapabilities - {"browserName":"phantomjs","version":"2.1.1","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"linux-unknown-64bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"},"phantomjs.page.settings.userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 xfinity-usage/2.0.2"}
[INFO  - 2018-03-17T14:51:54.119Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: bb9bbb50-29f2-11e8-ac31-ef03425707b6
[ERROR - 2018-03-17T14:52:01.413Z] WebElementLocator - _handleLocateCommand - Element(s) NOT Found: GAVE UP. Search Stop Time: 1521298321402

  phantomjs://platform/console++.js:263 in erro

Assume it is related to the driver/browser but not sure.

billimek commented 6 years ago

Ok so my issue was that after logging in, it redirected to a special page to confirm notification settings or whatever (I haven't logged in via a real browser in a very long time). No issue. The errors went away after sorting that out upon logging in for real.

No issue here.

jantman commented 6 years ago

Ok. Yeah unfortunately special redirects or modals are an issue for a lot of screen-scraping of accounts.

Are you all OK if I close this?

johns224 commented 6 years ago

Close away. Thanks for your help!

jantman / xfinity-usage