Help Needed

[destorieswind]

Hello,

I'm relatively new to coding but I think what you have here to capture Nest data is seriously awesome. I've been trying for a week to get some type of data logger for Nest and I am realizing this is very difficult. I've used IFTTT with no luck and and trying to capture the nest data as part of an experiment I am going to run on the house. I am in USA, Arizona where summer temps can get to 120F and I'm trying to see the heat load calculations if I cover the roof in shade.

Where I am at with your code: I've downloaded the files and created the constants.js as well as the web/constants.js. They look something similar to below minus the username/password for the nest.

"use strict" module.exports = Object.freeze({ STORAGE_PATH : 'Downloads/DATAFILES', MAX_DAYS : 120, RECORD_MINS : 5, NEST_USERNAME : 'NULL@gmail.com', NEST_PASSWORD : 'PASSWORD', WUNDERGROUND_KEY : 'NOT USING THIS RIGHT NOW', SERVER_PORT : 8080 })

window.CONSTANTS = { DEVICE_ID : 'grand house' }

Once I have that set up, I'm not terribly sure how to actually run it. I've tried running "node index.js" and "npm install" out of a terminal window pointed to the "nest_history-master" folder and get "command not found". Am I trying to run this properly? Do I need to be using Windows cmd? Your help would be greatly appreciated!

[Thomas101]

Hey,

It sounds like you're on the right path, but maybe don't have something setup on your machine. From the terminal if you run node -v does it print something out or complain about not being able to find the command again? What os are you using? OSX or a flavour of linux?

[destorieswind]

I was using OSX but I think this may lead to more troubles if I continue? What do you think? I am also going to try this on a WIN10 PC right now.

[destorieswind]

Tried again with a windows 10 computer and still no luck. Ran "node -v" in cmd and terminal and both say no command found.

[Thomas101]

I would stick with osx. That's what I wrote it on! Not 100% sure if it will work on windows to be honest. All you need to do is download something called Node.js. It's a programming language/framework that lets you run JavaScript on your mac. You can grab it from (their website)[https://nodejs.org/en/].

Once you've installed that, open up the terminal and type node -v again. This time you should see a version number. You should be set to go from here. You can do...

npm install
node index.js

If you want to leave it running in the background you can always run

nohup node index.js &

[destorieswind]

Painfully made it through getting the directory pointed to usr/local/bin even though when I did echo $PATH it showed the path is there. Ultimately I used cd .. to keep going up until I could use cd usr/local/bin. This is a display of my lack of knowledge with terminal but besides that point..

Next, I had to unlock the folder since the permissions were denied using sudo chown -R $(whoami) /usr/local/bin

Now that it was unlocked, I was able to use npm install and node index.js

The message was shown below, I'm thinking I need to have the files you provided for the logger in this directory? Correct me if I'm wrong, I already did "show hidden files" so I could drag your folder here if that is the case.

192:bin jasonD$ npm install npm WARN enoent ENOENT: no such file or directory, open '/usr/local/bin/package.json' npm WARN bin No description npm WARN bin No repository field. npm WARN bin No README data npm WARN bin No license field.

192:bin jasonD$ node index.js module.js:440 throw err; ^ Error: Cannot find module '/usr/local/bin/index.js' at Function.Module._resolveFilename (module.js:438:15) at Function.Module._load (module.js:386:25) at Function.Module.runMain (module.js:575:10) at startup (node.js:160:18) at node.js:445:3

[destorieswind]

Went out on a limb and put your files into the "bin" folder. The commands ran and gave this:

192:bin jasonD$ npm install nest_history@1.0.3 /usr/local/bin ├── colors@1.1.2 ├─┬ express@4.13.4 │ ├─┬ accepts@1.2.13 │

[Thomas101]

Okay, I think you've got a little bit lost :-/. When you ran the node installer it should have setup your paths etc correctly. It does look like they're set-up as running node something is at least giving an error other than not knowing what node is. You don't really need to be placing anything under /usr/local/bin as this is where frameworks and libraries that are available to the whole system normally get placed.

I'm guessing at some point you hit the download button from github and downloaded the repository. At this point, you'd have found the nest_history folder in your downloads folder. You could use cd to get there by using ~/Downloads/nest_history-master. You can keep it in Downloads, or move it Documents, Desktop etc... doesn't really matter. I wouldn't normally recommend putting it somewhere like /usr/local/bin unless you really know what you're doing.

When you open terminal type cd ~/Downloads/nest_history-master which should take you into the right folder. (Note ~ is a shortcut for /Users/<your username>/) (Also note when you're trying to navigate using cd you can use tab to autocomplete. e.g. type cd ~/Downloads/nes<tab> will autocomplete as much as it can, so will probably go to cd ~/Downloads/nest_history-master).

Once you're in this directory type npm install. It will take a little bit of time whilst it fetches the required dependencies, but once finished you should be able to type node index.js to run.

Ask for more help if you need it! I was where you were once, it's all about just knowing what you're doing :)

[destorieswind]

Thanks for reining me in and taking the time to help me learn this. This stuff has always interested me but there is no easy entry into it so I am just starting the dive.

I made the usr/local/bin folder back the way it normally was. I've now set the directory to the Downloads/nest_history-master and ran npm install. It took a little bit of time to fetch the data and when I ran node index.js is pulled the data shown below. One of the things I don't think I have right is the "Nest Device ID" which is found in the "Web/constants.js". Where do I get this ID? I don't think it will fetch the Nest data without it?

192:~ jasonD$ cd Downloads/nest_history-master

192:nest_history-master jasonD$ ls LICENSE capture2.png package.json README.md constants.js recorder.js Server.js db.js web capture1.png index.js web:constants.js

192:nest_history-master jasonD$ npm install nest_history@1.0.3 /Users/jasonD/Downloads/nest_history-master ├── colors@1.1.2

[destorieswind]

Ignoring the Nest 'Device_ID" for a moment, I was reading the previous post comments and saw that you said to run 'node index.js record' to see if it would create a file, mine is not. This is the error I'm getting below using the following constant path: '/Mac/Users/jasonD/Downloads/nest_history-master/db/',

Error: ENOENT: no such file or directory, open '/Mac/Users/jasonD/Downloads/nest_history-master/db/2016_4_7.db' { Error: ENOENT: no such file or directory, open '/Mac/Users/jasonD/Downloads/nest_history-master/db/2016_4_7.db' at Error (native) errno: -2, code: 'ENOENT', syscall: 'open', path: '/Mac/Users/jasonD/Downloads/2016_4_7.db' } 192:nest_history-master jasonD$

[Thomas101]

Okay, I've put some patches in to the code to try and be a bit more clever about what it's doing. If you've only got one nest device, the code will now figure that out and use just that one, so don't worry about putting your device id into the constants file. The error you're seeing is basically because it can't find the database folder and isn't creating it automatically, so the update fixes that too.

What you'll need to do is re-download the code from github. Once it's downloaded copy your two constants files into the latest download folder. (You may want to delete the old one, or stash it somewhere else). Using terminal cd into the folder, then do npm install again. Wait for this complete, then have another go at running npm index.js.

As for running the npm index.js record command, you can do that but you won't be able to go to localhost:8080 in your web browser to see the results. That just takes a single recording then quits from there.

[destorieswind]

Ran it all again using the new patches and everything looks good until having to use npm index. It looks like it "index" isn't one of the commands available for "npm". The version of "node -v" is v6.1.0.

192:nest_history-master_v2 jasonD$ npm index.js

Usage: npm

where is one of: access, adduser, bin, bugs, c, cache, completion, config, ddp, dedupe, deprecate, dist-tag, docs, edit, explore, get, help, help-search, i, init, install, install-test, it, link, list, ln, logout, ls, outdated, owner, pack, ping, prefix, prune, publish, rb, rebuild, repo, restart, root, run-script, s, se, search, set, shrinkwrap, star, stars, start, stop, t, tag, team, test, tst, un, uninstall, unpublish, unstar, up, update, v, version, view, whoami

npm -h quick help on npm -l display full usage info npm help search for help on npm help npm involved overview

Specify configs in the ini-formatted file: /Users/jasonD/.npmrc or on the command line via: npm --key value Config info can be viewed via: npm help config

npm@3.8.6 /usr/local/lib/node_modules/npm 192:nest_history-master_v2 jasonD$

[Thomas101]

Sorry, that was a typo my end. Use node index.js instead of npm index.js.

[destorieswind]

Face palm, I should have at least caught that. After running node index.js, is stays on "Serving on port 8080" command and nothing happens. I let it do that for about 5 minutes before killing it with ctrl + c and tried running the "node index.js record" to see if it would throw any errors and this was the result:

192:nest_history-master_v2 jasonD$ node index.js record Error: ENOENT: no such file or directory, open '/Mac/Users/jasonD/Downloads/nest_history-master_v2/db/2016_4_8.db' { Error: ENOENT: no such file or directory, open '/Mac/Users/jasonD/Downloads/nest_history-master_v2/db/2016_4_8.db' at Error (native) errno: -2, code: 'ENOENT', syscall: 'open', path: '/Mac/Users/jasonD/Downloads/nest_history-master_v2/db/2016_4_8.db' }

[destorieswind]

Oye! The file error was my mistake, just learned how to call a directory. I used /mac/Users/jasonD/Downloads/nest_history-master_v2/db instead of mac/Users/jasonD/Downloads/nest_history-master_v2/db

It now yields a .db file when running the "node index.js record". This is the info that populated in the .db file:

{"time":"2016-05-08T16:26:07.571Z","exec_time_ms":1819,"devices":{"02AA01AC331308D2":{"id":"02AA01AC331308D2","humidity":38,"outside_humidity":51,"temperature":24.71,"temperature_scale":"f","target_temperature":27.778,"postal_code":"85201","outside_temperature":66.4}}}

[destorieswind]

I'm now running the "node index.js" which is pinging the Nest and gathering data in the db file every 5 minutes or so. The next step is to make this data useable which you've already taken steps to do with the web interface. However, I'm not sure how to use it. I open the HTML in the web folder and get a blank screen. The DB file does have data and is constantly logging it from the Nest.

So far, you are a genius. I'm digging around in the "record.js" file to figure out the Nest API commands to see how I could have it grab the temperature in F instead of C.

Also, is there a command to see if the Nest has the A/C on or off?

[destorieswind]

Been working with the record.js file for a while now. I'm also trying to get the "hvac_state" command to run. However, this is a string ENUM command and is not displaying in the db file. I've tried a few different ways to do this in terms of calling this as a string but no luck.

This works because the "where_id" is a string. hvac_status: nestData.device[id].where_id

This does NOT work because the "hvac_state" is a string ENUM. hvac_status: nestData.device[id].hvac_state

A few hours later and we get these results which return a TRUE/FALSE boolean. This should accomplish what I am after. Funny because these commands aren't listed on the Nest API.

      COOLING_status: nestData.shared[id].hvac_ac_state,
      HEATER_status: nestData.shared[id].hvac_heater_state

I'm still looking for advice on getting this db file to display in your web interface.

[Thomas101]

Once you run node index.js and get Serving on port 8080 that's it basically running. You can point your web browser at localhost:8080 and you'll get to the UI. It will take a recording every 5 minutes (or whatever you set in your constants file) so after a day you should have a days worth of data graphed for you

[destorieswind]

No luck with the UI after going to localhost:8080 on the latest Google Chrome version. Overview and Dayview stay blank. Would this be affected if I added additional data to the recorder.js? One of my goals was to collect the total time the A/C was running using the (sampling rate* hvac_ac_state(true)).

  COOLING_status: nestData.shared[id].hvac_ac_state,
  HEATER_status: nestData.shared[id].hvac_heater_state

An example of my db file is:

{"time":"2016-05-09T03:53:03.087Z","exec_time_ms":1459,"devices":{"02AA01AC331308D2":{"id":"02AA01AC331308D2","humidity":33,"outside_humidity":37,"temperature":78.476,"temperature_scale":"f","target_temperature":76.001,"postal_code":"85201","outside_temperature":71.2,"COOLING_status":false,"HEATER_status":false}}}

[destorieswind]

Brainstormed the end goal of this project, see the attached picture. I'm not familiar with the way you are generating the UI webpage. Do you think it would be worth trying to create this UI using GoogleSheets? That way one could access the data anywhere and it would be updated as long as the DB file was written to a GoogleDrive.

[Thomas101]

You would need to leave it running for a while before any data is generated for it to show. How many nest devices do you have? If there's more than one device you'll need to add the device id into settings.

You could pull the data into google sheets, but you'd need to convert the data that's logged into your sheet. The data as it is at the moment just a JSON record on each line, so it should be pretty easy to do. You could take a look at google-spreadsheet which might let you do what you need

[destorieswind]

There is only 1 device but I changed the type of data it grabs. I've left the UI running all day and it still isn't populating with data. I'll see what I can do with spreadsheets.

Also, could you explain how your time format works? It does not align with my computer time or any time I can think would be relevant: "time":"2016-05-09T22:33:03.564Z"

{"time":"2016-05-09T22:33:03.564Z","exec_time_ms":1388,"devices":{"02AA01AC331308D2":{"id":"02AA01AC331308D2","humidity":35,"outside_humidity":21,"temperature":78.566,"temperature_scale":"f","target_temperature":74.7176,"postal_code":"85201","outside_temperature":89.8,"COOLING_status":true,"HEATER_status":false}}}

[destorieswind]

One last comment, the dB file timestamp does not correlate to the actual time. It looks more like a counter from when the program started that resets after 24 hours.

[Thomas101]

You could check to see if there is an error in the client. Try opening the page in chrome, then going to View>Developer>JavaScript Console. You should see the terminal in there that will log any errors.

The time is in the ISO_8601 format. There are plenty of libraries that will parse that into a more manageable format. For example in JavaScript you can convert backwards and forwards doing....

var iso8601 = new Date().toJSON()
new Date(iso8601)
Thu May 12 2016 19:46:13 GMT+0100 (BST)

Hope this helps!

[destorieswind]

The web java console showed the error that it couldn't find the constants.js file. I dumped the web/constants.js into the web folder and renamed it constants.js. After doing that, the first error went away but now this error shows:

jquery-2.1.4.min.js:4 XMLHttpRequest cannot load file:///api/history.json?_=1463149373770. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.

The issue with the time is not the format but the fact that it is is more of a time for the hours/minutes part. After digging around, I found this command which I thought would have been causing the issue. However, after taking out the subtracted startTime, it did not change anything in the time log. Is there any way to change the time log so that it grabs the actual computer time?

const record = this._createDatabaseRecord(now.getTime() - startTime.getTime(), nestData, weatherData)

[Thomas101]

By the sounds of it you maybe double clicked on index.html. By doing so you loaded the webpage but there was no server available to respond to your requests and security restrictions will prevent you loading the api requests as the file is not being served from a web server. To access the web screens you'll need to be running node index.js in a terminal window, then go to localhost:8080 in your web browser.

As regards the time, because the timestamp is serialized to a string so it can be written to a file it has to attach a timezone to it. The default instance for most programming languages it to go to GMT. In JavaScript it indicates this by placing a Z on the end of the timestamp. When you de-serialize it, normally it re-resolves it to be in your current timezone. For example if you open node by typing node into a terminal window you can play about creating times. Type the following...

new Date().toJSON()

For me, right now it prints out 2016-05-13T18:18:50.256Z which is an hour behind my time (it's actually 19:18). However if I copy that string and paste the following in...

new Date("2016-05-13T18:18:50.256Z")

... it now logs out Fri May 13 2016 19:18:50 GMT+0100 (BST) which is the correct time. If I was to assign the date to a variable I could query it and log more information out. For example...

var d = new Date("2016-05-13T18:18:50.256Z")
console.log('Year is ' + d.getFullYear())
console.log('Month is (zero indexed) ' + d.getMonth())
console.log('Month date is ' + d.getDate())
console.log('Day of week is ' + d.getDay())
console.log('Hours is ' + d.getHours())
console.log('Minutes is ' + d.getMinutes())
console.log('Seconds is ' + d.getSeconds())

which would give me...

Year is 2016
Month is (zero indexed) 4
Month date is 13
Day of week is 5
Hours is 19
Minutes is 18
Seconds is 50

There's some more information about JavaScript works with dates here

[destorieswind]

Ok, so I've been spinning my wheels for the last 3 days and trying not to bother you on a question that may seem so simple to those who work with code. I've narrowed down your db recorded time call out to the recorder.js section. Here, you call out "new Date()". This date gives the GMT time which is 7 hours ahead of me. I've tried a hundred different ways to simply subtract this 7 hours, subtract the TimeZoneOffset, or even call out local time. Am I wrong in thinking there should just be a function that gives Local_Time along with the date?

_createDatabaseRecord (executionTime, nestData, weatherData) { return { time: new Date(),

[Thomas101]

If you want to string it in a custom format you could for example do....

const now = new Date()
const nowStr = now.getHours() + ':' + now.getMinutes() + ':' + now.getSeconds() + ' ' + now.getDate() + '/' + (now.getMonth() + 1) + '/' + now.getFullYear()
return {
    time: nowStr,
...

The get hours/minutes etc will all return the date in your timezone

[destorieswind]

Thomas101!! What you've created here is incredibly elegant IMHO. Your advice on the time by creating a string allowed me to learn how to even create a string. I was trying a whole bunch of methods to change the time but unfortunately this type of code (java?) is not intuitive at all.

Updates: I got a Raspberry Pi, got Node installed, and running this program (slightly modified for the data I was after). So far everything is working like a charm after collecting data for 2.5 days. Now, type to begin energy saving modifications and I will have data to confirm any results! What a journey!

THANK YOU!!!

[Thomas101]

Glad you got it up and running :)

[JamesTX10]

@destorieswind Are you still using this code with the changes you made? I just got a nest tstat and am wanting to do the same thing like you and track the usage to outside temp.

[destorieswind]

Hi James. Yes I do but I am currently on vacation. I will send you the code that I have once I get back on Sunday. On Jul 14, 2016 2:21 PM, "JamesTX10" notifications@github.com wrote:

@destorieswind https://github.com/destorieswind Are you still using this code with the changes you made? I just got a nest tstat and am wanting to do the same thing like you and track the usage to outside temp.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Thomas101/nest_history/issues/3#issuecomment-232765632, or mute the thread https://github.com/notifications/unsubscribe/ASWS6njvqmeM3LKjeNsZ0ZKlGO_Ym12Aks5qVowigaJpZM4IZbrU .

[lanerussell]

@destorieswind I would also be curious to know how you pulled Fahrenheit through the API instead of Celsius. Could you share how you accomplished this?

[destorieswind]

I used the mathematical formula to convert it in the code, nests API for Fahrenheit did not work. Let me know if that works for you, if not I can send you what I did when I get home from work.

On Aug 12, 2016 10:45 AM, "artillerydude" notifications@github.com wrote:

@destorieswind https://github.com/destorieswind I would also be curious to know how you pulled Fahrenheit through the API instead of Celsius. Could you share how you accomplished this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Thomas101/nest_history/issues/3#issuecomment-239513057, or mute the thread https://github.com/notifications/unsubscribe-auth/ASWS6q99mIrwsQuZ6ggEZNhYkzvBSjQIks5qfLEkgaJpZM4IZbrU .

[lanerussell]

Thanks @destorieswind, I updated my recorder.js as follows. It's too bad about the api though; it'd be nice to have the extra granularity provided by F.

  _createDatabaseRecord (executionTime, nestData, weatherData) {
    return {
      time: new Date(),
      exec_time_ms: executionTime,
      devices: Object.keys(nestData.device).reduce((acc, id) => {
        const tempScale = nestData.device[id].temperature_scale.toLowerCase()
        const postalCode = nestData.device[id].postal_code
        const weather = ((weatherData[postalCode] || {}).current_observation || {})
        acc[id] = {
          id: id,
          humidity: nestData.device[id].current_humidity,
          outside_humidity: parseFloat((weather.relative_humidity || '').replace('%', '')),
          temperature: ((nestData.shared[id].current_temperature * 1.8) + 32),
          temperature_scale: tempScale,
          target_temperature: ((nestData.shared[id].target_temperature * 1.8) + 32),
          postal_code: postalCode,
          outside_temperature: weather['temp_' + tempScale]
        }
        return acc
      }, {})
    }
  }

*EDIT: Ha, finally got the markdown right

[hcmveerman]

@destorieswind How is your code working out for you? I would love to be able to generate those awesome graphs. Is it possible to publish the code you used? Thank you in advance!

[destorieswind]

Hey guys,

I updated my code to display different units and gather all the info in a single text file. That way you can use Excel or R to compile your graphs. Let me know if you guys get this.

On Fri, Nov 11, 2016 at 1:54 PM, hcmveerman notifications@github.com wrote:

@destorieswind https://github.com/destorieswind How is your code working out for you? I would love to be able to generate those awesome graphs. Is it possible to publish the code you used? Thank you in advance!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Thomas101/nest_history/issues/3#issuecomment-260053245, or mute the thread https://github.com/notifications/unsubscribe-auth/ASWS6pJZ4D78CoiCR7nG1eaXfgh3j-oiks5q9NYGgaJpZM4IZbrU .

Cheers, Jason DeStories