Closed meingraham closed 5 years ago
Hey Mike,
Hmm, that's odd. So what the program tries to do is start a firefox, and then connect to it to control it. That connection refused is it trying to connect to firefox and for some reason either firefox is not running, or not accepting the connection from nest-scape.
Can you try running nest-scrape with the -s option? Maybe something like:
nest-scrape -v2 -s
That -s option should make firefox appear visibly and so you can see if there is something going wrong when it starts. It could be that it is expecting some initial setup; in my tests it displayed some sort of profile entry dialog for a few seconds, but then nest-scrape was able to navigate to the nest page anyway. Perhaps that's not working on your system.
-v 2 will set the max verbosity for debugging.
Let me know what you see. If it's not working I can add try and add some extra debugging to help narrow down what's going wrong.
On Sat, Jan 26, 2019 at 1:06 PM Michael Ingraham notifications@github.com wrote:
Jeff,
I KNEW someone would have devised this workaround to Nest's unresponsiveness in providing API access to the temperature sensor. Thanks!
I executed the command and it just sat there staring at me. So I ran it with verbosity:
nest-scrape -v 1 browser: running browser using cmd '/usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile' dial tcp 127.0.0.1:2828: getsockopt: connection refused browser: retrying connection in 1 second dial tcp 127.0.0.1:2828: getsockopt: connection refused browser: retrying connection in 1 second ...
The connection refused message is repeated until I interrupt it with Ctrl-C. Can you provide some guidance as to what the issue might be?
Thank you.
Mike
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jeffwilliams/nest-scrape/issues/1, or mute the thread https://github.com/notifications/unsubscribe-auth/AA_n1tn7kZ8K6SAXIJNtAXpUY68so8Yaks5vHJkxgaJpZM4aUTA_ .
OK, I gave that a try. Same result. It seems to fail immediately without having been able to invoke Firefox at all.
What Linux distribution did you develop nest-scrape under? I am using the latest Raspbian Stretch. Also, the information I found said that the only Firefox version available for Raspbian is Firefox-ESR. This is the description given on that version's page:
Firefox Extended Support Release (ESR) is an official version of Firefox developed for large organizations like universities and businesses that need to set up and maintain Firefox on a large scale. Firefox ESR does not come with the latest features but it has the latest security and stability fixes.
I do not know what underlying facilities your code makes use of. Could it be that it's not finding something due the fact that ESR "does not come with the latest features"? Are there additional debugging levels that I can set to see what the call is where it's failing to connect successfully?
Mike
I developed this on Debian Jessie.
What happens if you run the firefox command manually with those options? I.e.:
/usr/bin/firefox --marionette --headless --profile
/home/pi/.nest-scrape/firefox-profile
or alternately:
/usr/bin/firefox --marionette --profile
/home/pi/.nest-scrape/firefox-profile
Does it print any output? After running that last command, what is the value of the $? environment variable? I.e.:
/usr/bin/firefox --marionette --profile
/home/pi/.nest-scrape/firefox-profile
echo $?
Also what is the output of /usr/bin/firefox --version?
It's odd that firefox is failing with no output to stderr, and yet the Go exec call is not returning an error on first launch (i.e. it was able to find the binary and execute it).
On Sat, Jan 26, 2019 at 3:02 PM Michael Ingraham notifications@github.com wrote:
OK, I gave that a try. Same result. It seems to fail immediately without having been able to invoke Firefox at all.
What Linux distribution did you develop nest-scrape under? I am using the latest Raspbian Stretch. Also, the information I found said that the only Firefox version available for Raspbian is Firefox-ESR https://support.mozilla.org/en-US/kb/switch-to-firefox-extended-support-release-esr. This is the description given on that version's page:
Firefox Extended Support Release (ESR) is an official version of Firefox developed for large organizations like universities and businesses that need to set up and maintain Firefox on a large scale. Firefox ESR does not come with the latest features but it has the latest security and stability fixes.
I do not know what underlying facilities your code makes use of. Could it be that it's not finding something due the fact that ESR "does not come with the latest features"? Is there additional debugging levels that I can set to see what the call is where it's failing to connect successfully?
Mike
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jeffwilliams/nest-scrape/issues/1#issuecomment-457861560, or mute the thread https://github.com/notifications/unsubscribe-auth/AA_n1ulvRwtGBtw6tD6MZr-wAZsVeT4Mks5vHLRMgaJpZM4aUTA_ .
Jeff,
It does appear to run successfully when run interactively. Here are the results.
$ /usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile
1548592442824 Marionette INFO Listening on port 2828
$ echo $?
130
$ /usr/bin/firefox --version
Mozilla Firefox 52.9.0
Mike
Thanks Mike.
When you run firefox interactively like that, did it ask you to fill out any settings or anything, or could you immediately browse? If it did ask, does the script work after that?
Also, could you try running it interactively as you did, and in another terminal try connecting to that marionette port? I.e. in another terminal try:
telnet 127.0.0.1 2828
You should be able to connect and see something like the following:
$ telnet 127.0.0.1 2828
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
50:{"applicationType":"gecko","marionetteProtocol":3}
Then you can disconnect by typing CTRL-] (or whatever the escape character is in your telnet) and then typing close.
One thing that is odd is that the connection refused seems to be happening on the getsockopt
call in the Go net library:
dial tcp 127.0.0.1:2828: getsockopt: connection refused
where on mine it fails in the connect
call (until firefox is ready for connections):
dial tcp 127.0.0.1:2828: connect: connection refused
Let me see if there is any debugging I can add around there.
One other thing, can you try running firefox interactively as previously:
/usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile
Then once you see:
1548592442824 Marionette INFO Listening on port 2828
From another terminal can you try seeing what address the OS thinks the firefox marionette socket is listening on? This command should do it:
netstat -an -A inet,inet6
I am wondering if 127.0.0.1 is not the configured local loopback address for some reason, or firefox is listening on IPv6 (address ::1).
Jeff,
So... I had been running nest-scrape
from MobaXterm (an ssh client) - i.e., not from the Raspbian GUI Desktop. Your note last night where you asked
When you run firefox interactively like that, did it ask you to fill out any settings or anything, or could you immediately browse?
clued me in that Firefox
, even though invoked as headless, appears that it wants to render a browser window. Sure enough, invoking Firefox
from the command line does indeed start a browser window. So, I then ran nest-scrape
from a Terminal
... and progress... but not success... yet.
If I start with a fresh session (no Firefox running) and just try running nest-scrape
, I get the same repeating output:
$ nest-scrape -v2
browser: running browser using cmd '/usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile'
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
If I first invoke
$ /usr/bin/firefox --marionette --headless --profile ./.nest-scrape/firefox-profile
this opens a Firefox window (it does not request any more information of me - the window opens with the default Start tab)
If then, in another Terminal
, I invoke
$ nest-scrape -v2
The browser opens another tab with a directory listing of file:///home/pi/.nest-scrape/firefox-profile/
and in the Terminal
, the following output:
browser: running browser using cmd '/usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile'
[0,1,"WebDriver:NewSession",{"capabilities":null,"sessionId":""}]
2019/01/27 16:32:00 65:[0,1,"WebDriver:NewSession",{"capabilities":null,"sessionId":""}]
2019/01/27 16:32:00 [1,1,{"error":"unknown command","message":"WebDriver:NewSession","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\n - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,2,"newSession",{"capabilities":null,"sessionId":""}]
2019/01/27 16:32:05 55:[0,2,"newSession",{"capabilities":null,"sessionId":""}]
2019/01/27 16:32:05 [1,2,null,{"sessionId":"f26c1564-7aea-48dc-8eb8-d00466a59fda","capabilities":{"browserName":"firefox","browserVersion":"52.9.0","platformName":"linux","platformVersion":"4.14.79+","pageLoadStrategy":"normal","acceptInsecureCerts":false,"timeouts":{"implicit":0,"page load":300000,"script":30000},"rotatable":false,"specificationLevel":0,"moz:processID":4901,"moz:profile":"/home/pi/.nest-scrape/firefox-profile","moz:accessibilityChecks":false}}]
Navigating to site
[0,3,"WebDriver:Navigate",{"url":"https://home.nest.com"}]
2019/01/27 16:32:10 58:[0,3,"WebDriver:Navigate",{"url":"https://home.nest.com"}]
2019/01/27 16:32:10 [1,3,{"error":"unknown command","message":"WebDriver:Navigate","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\nDi - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
Checking if login is required
Waiting for thermostat location to load (element css selector .puck-item > a)
[0,4,"WebDriver:FindElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/27 16:32:12 84:[0,4,"WebDriver:FindElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/27 16:32:12 [1,4,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\ - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,5,"WebDriver:FindElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/27 16:32:16 84:[0,5,"WebDriver:FindElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/27 16:32:16 [1,5,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\ - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,6,"WebDriver:FindElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/27 16:32:18 84:[0,6,"WebDriver:FindElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/27 16:32:18 [1,6,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\ - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
Finding thermostat location (element with id .puck-item > a)
[0,7,"WebDriver:FindElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/27 16:32:20 84:[0,7,"WebDriver:FindElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/27 16:32:20 [1,7,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\ - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
Waiting for login field to load (element id email)
[0,8,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:20 60:[0,8,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:20 [1,8,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\ - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,9,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:21 60:[0,9,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:21 [1,9,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\ - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,10,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:22 61:[0,10,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:22 [1,10,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12 - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,11,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:24 61:[0,11,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:24 [1,11,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12 - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,12,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:25 61:[0,12,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:25 [1,12,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12 - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,13,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:26 61:[0,13,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:26 [1,13,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12 - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
[0,14,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:27 61:[0,14,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:27 [1,14,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12 - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
Finding login field (element with id email)
[0,15,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:28 61:[0,15,"WebDriver:FindElement",{"using":"id","value":"email"}]
2019/01/27 16:32:28 [1,15,{"error":"unknown command","message":"WebDriver:FindElement","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12 - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
Error: When finding login field by id email, got an error: WebDriver:FindElement
Login failed: When finding login field by id email, got an error: WebDriver:FindElement
[0,16,"Marionette:Quit",{"flags":["eForceQuit"]}]
2019/01/27 16:32:29 49:[0,16,"Marionette:Quit",{"flags":["eForceQuit"]}]
2019/01/27 16:32:29 [1,16,{"error":"unknown command","message":"Marionette:Quit","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nUnknownCommandError@chrome://marionette/content/error.js:444:5\nDispatcher.prototype.execute/req<@chrome://marionette/content/dispatcher.js:121:13\nTaskImpl_run@resource://gre/modules/Task.jsm:319:42\nTaskImpl@resource://gre/modules/Task.jsm:277:3\ncreateAsyncFunction/asyncFunction@resource://gre/modules/Task.jsm:252:14\nTask_spawn@resource://gre/modules/Task.jsm:166:12\nDisp - END - spatcher.prototype.onPacket@chrome://marionette/content/dispatcher.js:89:5\nDebuggerTransport.prototype._onJSONObjectReady/<@chrome://marionette/content/server.js -> resource://devtools/shared/transport/transport.js:483:11\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\nexports.makeInfallible/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://devtools/shared/ThreadSafeDevToolsUtils.js:101:14\n"},null]
$
So, nest-scrape
still can't successfully start up Firefox, but if it's already started, then it can use that session.
Perhaps there is something in there that gives you a clue as to what we can look at next.
Mike
It seems like we have two problems with nest-scrape now:
It can't start firefox properly.
It can't communicate properly with firefox when it's running.
So for problem #1, I believe nest-scrape should be able to start firefox from ssh (without a GUI) as long as -s is not specified, but in your original post it failed. Just to confirm I understand the situation, can you confirm the following?
a) Over ssh (i.e. from MobaXterm) if you run '/usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile' manually, does it start and remain running? That is, does it display "*** You are running in headless mode." and then retain control of the terminal (you don't get a prompt back) until you hit CTRL-C?
b) Over ssh, if you run 'nest-scrape -v 1' (without -s) is it able to connect after some time (maybe 1 minute)?
c) From the Raspbian GUI if you run 'nest-scrape -v 1' is it able to connect after some time (maybe 1 minute)?
d) From the Raspbian GUI if you run 'nest-scrape -v 1 -s' is it able to connect after some time (maybe 1 minute)?
Now for problem #2, I have a hunch that the firefox version from Raspbian (which seems to be 52.9.0) might have a different Marionette API than the version I used (64.0.2). The reason I think this is that the Go library I'm using tries to call the WebDriver:NewSession function in Firefox which fails with "unknown command", but then the library falls back to using an older call newSession which succeeds. Then later WebDriver:Navigate fails similarly... I suspect a number of functions got renamed in later versions.
I have a feeling that that version might be too old to support the Go library nest-scrape uses to talk Marionette with Firefox. The library seems to only test as early as firefox 56.0 (see the 'matix' line in this file):
https://github.com/njasm/marionette_client/blob/master/.travis.yml
Let me think about some possible options.
a. MobaXterm
$ /usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile
1548645820820 Marionette INFO Listening on port 2828
No other output ever came after over 5 minutes. I hit Ctrl-C to stop it. Before doing that, I looked at running processes (in another terminal session):
$ ps -ax | grep fox
5439 pts/0 Sl+ 2:20 firefox-esr --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile
b. MobaXterm
$ nest-scrape -v 1
browser: running browser using cmd '/usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile'
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
1548646307702 Marionette INFO Listening on port 2828
Navigating to site
Checking if login is required
Waiting for thermostat location to load (element css selector .puck-item > a)
Finding thermostat location (element with id .puck-item > a)
Waiting for login field to load (element id email)
Finding login field (element with id email)
Error: When finding login field by id email, got an error: WebDriver:FindElement
Login failed: When finding login field by id email, got an error: WebDriver:FindElement
While it was starting up
$ ps -ax | grep fox
5507 pts/0 Rl+ 0:04 firefox-esr --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile
5509 pts/0 Z+ 0:03 [firefox-esr] <defunct>
The failure left behind the Firefox process. I killed it manually before going to the next test.
Obviously I did not give the startup enough time in the initial invocations yesterday. During my testing & debugging, I'm running this on a Pi ZeroW so it is slow... which explains why it takes a long time to start up. Good call on
connect after some time (maybe 1 minute)
Once I get this to work, I'll put nest-scrape on my RPi 3. I just wanted this testing on a separate system until I was ready to put it into "production".
c. & d. Raspian GUI Terminal In both cases, a Firefox window opened. Both generated the same output as 'b' test above.
Thanks.
Mike
Hi Mike,
Thanks for testing it out.
Regarding the firefox process that was left behind, I think there might be a bug where nest-scrape is not cleaning up resources on those failures. I'll look into that.
As for the Marionette API version problem (the WebDriver:FindElement error) I think I might be able to solve it by getting the source code for an older version of the Go Marionette library and compiling against that for Raspbian. When I look in the git history for the library it seems like at one point it supported Firefox version 52. I'll try that when I get an opportunity, and provide you a version of the source code you can try.
Much appreciated!
Hi Mike,
I've made a new branch of the code to support Firefox version 52. Could you update your clone of the source code to that branch and try it out? I'm assuming since you're running this on Raspbian you compiled this yourself using Go. So to get this new version you should just have to:
cd src/github.com/jeffwilliams/nest-scrape
git pull
git checkout firefox-52
go get github.com/jeffwilliams/marionette_client
go install .
Let me know how it goes.
Also, I see what you mean about firefox still trying to open a window even with --headless specified. For some reason in version 52 it does that sometimes; in later versions it doesn't seem to do that.
Jeff,
nest-scrape
seems do be toiling a bit more... before it fails :frowning_face:
The results look pretty much the same from both MobaxTerm as well as the Desktop Terminal. Of course, from the Terminal, a Firefox window does pop up.
One thing from watching the Firefox window is that it did start to load the home.nest.com page... but it did seems as though it did not completely load when nest-scrape
exited. I could see messages in the browser status and the the page link in the URL bar... but the page had not rendered. Do you think that the fact that it's running on a very slow Pi ZeroW, and that the page takes a long time to load, that the nest-scrape
Firefox (marionette?) calls are timing out? If so, is there a way to extend the timeout for those calls, at least for testing purposes. It could be a command line option so if one has a slow system, you tell it to give it more time?
Also, in every case, nest-scrape
did clean up after itself nicely in that the Firefox process was gone a few seconds after nest-scrape
exited (when the "Failed downloading XML" message was output.
$ nest-scrape -v 2
browser: running browser using cmd '/usr/bin/firefox --marionette --headless --profile /home/pi/.nest-scrape/firefox-profile'
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
...
dial tcp 127.0.0.1:2828: getsockopt: connection refused
browser: retrying connection in 1 second
1548729152910 Marionette INFO Listening on port 2828
[0,1,"newSession",{"capabilities":null,"sessionId":""}]
2019/01/28 21:32:59 55:[0,1,"newSession",{"capabilities":null,"sessionId":""}]
2019/01/28 21:32:59 [1,1,null,{"sessionId":"97966f14-1a8a-4ad9-b53f-28501ed0bbc6","capabilities":{"browserName":"firefox","browserVersion":"52.9.0","platformName":"linux","platformVersion":"4.14.79+","pageLoadStrategy":"normal","acceptInsecureCerts":false,"timeouts":{"implicit":0,"page load":300000,"script":30000},"rotatable":false,"specificationLevel":0,"moz:processID":1067,"moz:profile":"/home/pi/.nest-scrape/firefox-profile","moz:accessibilityChecks":false}}]
Navigating to site
[0,2,"get",{"url":"https://home.nest.com"}]
2019/01/28 21:33:10 43:[0,2,"get",{"url":"https://home.nest.com"}]
2019/01/28 21:33:10 [1,2,null,{}]
Checking if login is required
Waiting for thermostat location to load (element css selector .puck-item > a)
[0,3,"findElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/28 21:33:13 74:[0,3,"findElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/28 21:33:13 [1,3,{"error":"no such element","message":"Unable to locate element: .puck-item > a","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nNoSuchElementError@chrome://marionette/content/error.js:388:5\nelement.find/</<@chrome://marionette/content/element.js:272:16\n"},null]
[0,4,"findElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/28 21:33:17 74:[0,4,"findElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/28 21:33:17 [1,4,{"error":"no such element","message":"Unable to locate element: .puck-item > a","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nNoSuchElementError@chrome://marionette/content/error.js:388:5\nelement.find/</<@chrome://marionette/content/element.js:272:16\n"},null]
[0,5,"findElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/28 21:33:21 74:[0,5,"findElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/28 21:33:21 [1,5,{"error":"no such element","message":"Unable to locate element: .puck-item > a","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nNoSuchElementError@chrome://marionette/content/error.js:388:5\nelement.find/</<@chrome://marionette/content/element.js:272:16\n"},null]
Finding thermostat location (element with id .puck-item > a)
[0,6,"findElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/28 21:33:24 74:[0,6,"findElement",{"using":"css selector","value":".puck-item \u003e a"}]
2019/01/28 21:33:24 [1,6,{"error":"no such element","message":"Unable to locate element: .puck-item > a","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nNoSuchElementError@chrome://marionette/content/error.js:388:5\nelement.find/</<@chrome://marionette/content/element.js:272:16\n"},null]
Waiting for login field to load (element id email)
[0,7,"findElement",{"using":"id","value":"email"}]
2019/01/28 21:33:38 50:[0,7,"findElement",{"using":"id","value":"email"}]
2019/01/28 21:33:38 [1,7,{"error":"no such element","message":"Unable to locate element: email","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nNoSuchElementError@chrome://marionette/content/error.js:388:5\nelement.find/</<@chrome://marionette/content/element.js:272:16\n"},null]
Finding login field (element with id email)
[0,8,"findElement",{"using":"id","value":"email"}]
2019/01/28 21:33:41 50:[0,8,"findElement",{"using":"id","value":"email"}]
2019/01/28 21:33:41 [1,8,{"error":"no such element","message":"Unable to locate element: email","stacktrace":"WebDriverError@chrome://marionette/content/error.js:200:5\nNoSuchElementError@chrome://marionette/content/error.js:388:5\nelement.find/</<@chrome://marionette/content/element.js:272:16\n"},null]
Error: When finding login field by id email, got an error: Unable to locate element: email
Login failed: When finding login field by id email, got an error: Unable to locate element: email
[0,9,"quitApplication",{"flags":"eForceQuit"}]
1548729221630 Marionette INFO New connections will no longer be accepted
2019/01/28 21:33:42 46:[0,9,"quitApplication",{"flags":"eForceQuit"}]
2019/01/28 21:33:42 [1,9,null,{}]
$
1548729101820 addons.productaddons WARN Failed downloading XML, status: 0, reason: timeout
Here is the log without the verbose output:
$ nest-scrape
Error: When finding login field by id email, got an error: Unable to locate element: email
Login failed: When finding login field by id email, got an error: Unable to locate element: email
Mike
Hi Mike,
Yes, there is a setting for making the timeout higher:
-t, --timeout int
Number of seconds to wait until each page element loads (default 8)
Maybe try something high like 60? I.e:
nest-scrape -v 2 -t 60
You can see all the options by running nest-scrape --help
.
Duh! I'd seen that but I misread/misinterpreted it.
Well, timeouts are definitely causing issues. After a bit more testing, it's obvious that a Pi ZeroW is not suitable for this.
I went ahead and powered down one of my Pi 3B. I can do without it for a couple of days. Much better!
As before, the browser window definitely wants to render... even when I'm using MobaXterm. I inadvertently had my VNC client up and invoked nest-scrape from MobaXterm... and to my surprise, the Firefox window popped up. Anyhow...
The process has gotten through different stages on different test runs. The Nest thermostat page renders The credentials dialog loads The credentials are filled in The credentials are authenticated The next page begins to load
After any of those, add "sometimes". And there does not seem to be a pattern as to when it will and when it won't. In fact, sometimes when the credentials authentication fails, I can type the password in manually and the process continues. Sometimes it tells me that the service is unavailable (I know I'm entering the right username/password). If I try again, it goes through fine. Sometimes when the "service is unavailable", I'll go to my PC and bring up Opera or IE and the nest thermostat page loads and lets me log in without issue.
Well, damn... it might have really been that the Nest service was unavailable - https://www.reddit.com/r/Nest/comments/al48qf/nest_down_500_and_504_error/?utm_source=reddit-android
I want to install this on one of my Windows machines just as a test. Is that possible? I installed Go and Firefox. nest-scrape --generate
and nest-scrape --help
do what is intended. I edited nest.yaml and specified the parameters with Windows in mind (e.g., double quotes around file paths that contain spaces).
When I try to execute nest-scrape -v 2
, I guess it doesn't like the permissions on my nest.yaml file
Loading config failed: The permissions on the config file must not allow group or other any access
Is trying to run it on Windows a bridge too far?
Mike
I tried just now (19:15 EST) and I can report that it was a success! Thanks! It looks like Nest is back online!
It still taxes an RPi 3 at times - CPU pegged at 100% for portions of the scrape.
Running it from an SSH session (i.e., MobaXterm) doesn't work. Even if my VNC client session is open and visible and the browser window can render. VNC is needed to allow the browser window to render. it's moot anyway. I've seen it error out, complete but not retrieve any actual measurements, just the JSON labels.
If I need VNC up and running, at that point. it kind of defeats the purpose of using MobaXterm since requiring VNC means I can just run nest-scrape
from a Desktop Terminal. This non-headless thing with version 52.9.0 is unfortunate. I did see that Raspberry was working on getting Firefox 60 as the supported version. But no real timeline on that!
Seems to take about 3 minutes start to finish. I thought that if I told it to not kill the browser session (-s
option) that it might save it some time by not having to open another session and window. It didn't affect it because running nest-scrape
again just opened a new browser window (not an additional tab in the existing browser window).
This much I can probably already say - it's going to require a dedicated platform to scrape the data. I now need to figure out how to couple this with a mechanism to publish it over MQTT so my home automation system can capture the information and make use of it in a more flexible way than what Nest allows!
Curious - what are you doing with the data?
Mike
Hi Mike,
I'm glad it's working somewhat better. Yes, that non-headless behavior is indeed a problem.
Are you still interested in a Windows version? I haven't tried it on Windows, but I think with a few tweaks it could probably run on it. That permissions error is just a flimsy attempt to make sure if you're running it on a multi-user system other users aren't able to read your password from that file.
Hmm, I'll see there's some way I can get it to push MQTT. If it helps, there is an option to change the output format to JSON.
For myself, I'm just running this on a Debian server from a cron job, and appending the output to a text file. We had an issue resolved under our home warranty related to a cold room and so I've been monitoring that room and a few other points to see how much of a temperature difference there is on cold days. Really nothing too fancy. I just ssh in and check the data to see how it's looking.
Yeah, I'm using the JSON option.
Yes, I'd like to try it on Windows to see if that runs fast enough where I can run it in the scheduler and have it pump the JSON over MQTT to my automation hub. If you have a means to integrate MQTT publish, then problem solved. Just a matter of adding some MQTT configuration information (broker, credentials, topic).
Mike
Ok, I've just pushed a commit that adds an option that disables nest-scrape from checking the permissions on the config file. So you can now add the option -p
or --no-check-perms
and it should work in Windows.
Thanks. I'll give it a try... after I figure out why Firefox on my laptop crashes every time :disappointed: Nothing to do with nest-scrape
- just won't run!
I still have not been able to resolve the Firefox installation issues so I can't try nest-scrape
on my Windows PC. In the interim, I purchased some Wi-Fi connected temperature sensors. I am now using those as my remote temperature inputs to the home automation routines I built. I wanted to re-use the Nest sensors as "unattached" sensors by scraping their readings. Unfortunately, Firefox on the Raspberry Pi is too resource "hungry" (as is Chrome, BTW). Without being able to successfully deploy Firefox on my PC and run nest-scrape
there, I probably will just scrap my Nest sensors altogether now that I have an alternative. I guess I'll be putting my Nest 3-pack on eBay ;-)
Mike
P.S. Even if I had gotten it going on my PC, I would have still had to get MQTT communications going. My new sensors are MQTT enabled out of the box. They are roughly $30 each and monitor both temperature and humidity.
Hi Mike,
Sorry it didn't work out. Ideally there would be a simpler way to get these readings from the Nest temperature sensors. I hope the new sensors work out well.
Jeff,
I KNEW someone would have devised this workaround to Nest's unresponsiveness in providing API access to the temperature sensor. Thanks!
I executed the command and it just sat there staring at me. So I ran it with verbosity:
The
connection refused
message is repeated until I interrupt it withCtrl-C
. Can you provide some guidance as to what the issue might be?Thank you.
Mike