Blank Temperatures entry in Daily Table and Daily Chart

[unlstorm]

I had another anomaly show up today. In my Daily Table and Daily Chart for today I had a blank entry show up for inside temperature and outside temperature. When I looked at the temperature table in the database I noticed that the date row for that entry was showing "2012-10-15 12:31:00" when it should have been "2012-10-15 12:30:00".

I think this is because php$today = date( 'Y-m-d' ); is set after the call to get external weather. Perhaps it should be changed to php$today = date( 'Y-m-d', $_SERVER['REQUEST_TIME'] ); or moved to the top of the thermo_update_temps.php script?

[devzzm]

I've moved it to the top in my branch but it will need to be merged. When I was rewriting the scripts I had intended for it to store the actual date/time in the database and then have the graphs average it out instead of relying on the time index. But I got bogged down in that so I am not sure how ThermoMan is handling it.

Also ThermoMan I notice in the script outdoortemp defaults to -999 and outdoorhumidity defaults to null. Can the temp default to null as well?

[ThermoMan]

Hmm, in the code I get rid of the seconds (so that I store HH:MM:00 regardless of the value of SS ) because sometimes cron is inconsistent. The charting package that I am using is sort of unforgiving in that regard. It doesn't know that these are times that are being charted, they are just points along an X-axis with labels that just happen to be HH:MM so it will not properly adjust the spacing for a data point that is :31 instead of :30

This bears some noodling I think.

As for the -999 vs null as default outdoor temp there is no reason that it can't change. I had originally put in the -999 so that I could see bad data, but a null works just as well.

This might be time to fully split out the locations so that there would be a third cron task and separate tables for the location temperature record and the thermostat temperature record. The new cron would loop through each location and grab the temperature for it.

[ThermoMan]

I see a couple of items here.

1. Change default temperature to null (from -999)
2. Normalize times so that they save as XX:00 and XX:30 even if cron causes them to delay a minute or so. (I don't see cron ever firing early)

1 is easy. For #2 how about forcing the unit digit in the time stamp to zero. So that the result will look like this (this is not proper syntax) HH:M0:00. That way the task might be delayed by as much as 9 minutes and still get the temperature attributed to the intended timestamp. It's not brain surgery, so approximation like this shouldn't harm anything.

Instead of forcing all timestamps to this format, how about making note when the task kick-off is delayed by writing to the log file. That way the data will all be OK but there will be a history of "issues" that if they happen too often can be addressed. That'll be a server issue at the time it happens rather than a code issue anyway.

[devzzm]

My idea was to have it store the actual run time in the database...that way you could adjust the accuracy of the graph by simply running it more or less often. The problem of course is the limitations of the graphing libraries

On 10/17/2012 11:56 AM, ThermoMan wrote:

I see a couple of items here.

1. Change default temperature to null (from -999)
2. Normalize times so that they save as XX:00 and XX:30 even if cron causes them to delay a minute or so. (I don't see cron ever firing early)

1 https://github.com/ThermoMan/3M-50-Thermostat-Tracking/issues/1

is easy. For #2 https://github.com/ThermoMan/3M-50-Thermostat-Tracking/issues/2 how about forcing the unit digit in the time stamp to zero. So that the result will look like this (this is not proper syntax) HH:M0:00. That way the task might be delayed by as much as 9 minutes and still get the temperature attributed to the intended timestamp. It's not brain surgery, so approximation like this shouldn't harm anything.

Instead of forcing all timestamps to this format, how about making note when the task kick-off is delayed by writing to the log file. That way the data will all be OK but there will be a history of "issues" that if they happen too often can be addressed. That'll be a server issue at the time it happens rather than a code issue anyway.

— Reply to this email directly or view it on GitHub https://github.com/ThermoMan/3M-50-Thermostat-Tracking/issues/3#issuecomment-9532692.

[ThermoMan]

When you say "run time", do you mean the event logging time? I confuse "run time" with how long the HVAC unit runs.

The table that contains the half-hour time stamps was how I worked around having missing data in my table. I was on a Windows server and between Windows Scheduler just skipping an event and the PHP .dll crashing, I had a lot of missing data early on. That missing data exposed a weakness in this charting package - so I needed some way to force the X-Axis labels to have all the EXPECTED data points and then placed a VOID in the temperature where one was missing.

I'm fixated on the 30 minute interval just because of previous experience with collecting temperature data.

I guess the best thing would be to store the actual timestamp with the data and then as you say handle the rest on the display side. I would still like to have an indicator of when I expected a data point so that a missing one would stand out. If the charting package showed that by a gap (as pChart does) or some other method (just connect the dots that you do have, but show in alternate color?)

Combining the list of actual times and the list of expected times might prove interesting. Assuming we're still separating the data functions and the display functions, how do you classify that? One way would be to have another cron task that does data cleansing?

Perhaps a half hour after each data point is expected to be collected a routine could come along as use some fuzzy logic to see if there are expected points (so that 12:31 would count as the 12:30 collection) and if it found one truly missing, that it would insert a timestamp and a NULL directly into the database. The run time for such a cleanup task would have to be at least "expected interval + 1" so that after the anticipated 13:00 data collection you'd clean up for a missed 12:30 data point. That way a 12:45 (or even, shudder, at 12:59) point would count.

The X-Axis labels should still be on a nice tidy and regular interval, but the actual x position of the data point should be appropriate to the actual time stamp.

That almost sounds like a whole new batch of code for charting and is not something I'm prepared to take on. But if you find a chart pacakge like that (or add a chart type to pChart so that it knows that it's a time stamp and not just a generic label) I'd absolutely be interested.

[devzzm]

On 10/17/2012 01:20 PM, ThermoMan wrote:

When you say "run time", do you mean the event logging time? I confuse "run time" with how long the HVAC unit runs.

Yes sorry I meant when the script ran = "run time"...i see the confusion there :)

The table that contains the half-hour time stamps was how I worked around having missing data in my table. I was on a Windows server and between Windows Scheduler just skipping an event and the PHP .dll crashing, I had a lot of missing data early on. That missing data exposed a weakness in this charting package - so I needed some way to force the X-Axis labels to have all the EXPECTED data points and then placed a VOID in the temperature where one was missing.

I'm fixated on the 30 minute interval just because of previous experience with collecting temperature data.

I guess the best thing would be to store the actual timestamp with the data and then as you say handle the rest on the display side. I would still like to have an indicator of when I expected a data point so that a missing one would stand out. If the charting package showed that by a gap (as pChart does) or some other method (just connect the dots that you do have, but show in alternate color?)

Right and this is where I got bogged down. I stored actual query times in the database which could be close together, have gaps, etc. And pChart just would not handle this... documentation for pchart is lacking and I couldn't figure it out. It would be nice if we could find a charting package that is "time aware" and could take any times with points and plot them appropriately on a 24-hour graph, etc. Unfortunately my experience with charting packages is slim so I have no idea if such a thing exists.

I tried with pchart to force a "grid" of sorts - one per minute so the times could fall wherever and be graphed..but it turned out to be too much for pchart (having 1440 points many of which could be null)

Combining the list of actual times and the list of expected times might prove interesting. Assuming we're still separating the data functions and the display functions, how do you classify that? One way would be to have another cron task that does data cleansing?

Perhaps a half hour after each data point is expected to be collected a routine could come along as use some fuzzy logic to see if there are expected points (so that 12:31 would count as the 12:30 collection) and if it found one truly missing, that it would insert a timestamp and a NULL directly into the database. The run time for such a cleanup task would have to be at least "expected interval + 1" so that after the anticipated 13:00 data collection you'd clean up for a missed 12:30 data point. That way a 12:45 (or even, shudder, at 12:59) point would count.

The X-Axis labels should still be on a nice tidy and regular interval, but the actual x position of the data point should be appropriate to the actual time stamp.

That almost sounds like a whole new batch of code for charting and is not something I'm prepared to take on. But if you find a chart pacakge like that (or add a chart type to pChart so that it knows that it's a time stamp and not just a generic label) I'd absolutely be interested.

— Reply to this email directly or view it on GitHub https://github.com/ThermoMan/3M-50-Thermostat-Tracking/issues/3#issuecomment-9535829.

[ThermoMan]

GitHub itself may have coughed up one answer to this. https://github.com/oesmith/morris.js I saw that link on the home page and followed its' links to http://www.howmanyleft.co.uk/vehicle/volkswagen_golf_driver_auto and to http://www.oesmith.co.uk/morris.js/

Not the prettiest chart I've ever seen, but it might satisfy the design requirements?

[unlstorm]

Those examples are kind of what I'm talking about with the image map stuff as well:

https://github.com/ThermoMan/3M-50-Thermostat-Tracking/issues/10

When you mouse over the chart it brings up information about the points on the chart.

[ThermoMan]

Check out the screenshots of my other project https://github.com/ThermoMan/TED-5000-Electricity-Tracking/wiki/Screenshots

The round dot on the second and fifth images shows where the mouse is hovering on the chart and the text box in the upper right shows the value at that point - both x and y values.

[unlstorm]

Yeah, that's what I'm talking about. Wouldn't mind that on the bar charts.... On Oct 18, 2012 5:23 PM, "ThermoMan" notifications@github.com wrote:

Check out the screenshots of my other project https://github.com/ThermoMan/TED-5000-Electricity-Tracking/wiki/Screenshots

The round dot on the second and fifth images shows where the mouse is hovering on the chart and the text box in the upper right shows the value at that point - both x and y values.

— Reply to this email directly or view it on GitHubhttps://github.com/ThermoMan/3M-50-Thermostat-Tracking/issues/3#issuecomment-9583594.

[ThermoMan]

In case were still tracking. I just got a -999 on outdoor temp. It's the first one in many months. Months ago I corrected the data by manually interpolating the two temperatures around it. I'll try this time with a NULL/VOID and see what that does to the chart.

Here is the data before the manual tweak.

date indoor_temp outdoor_temp set_point indoor_humidity outdoor_humidity 2012-10-29 09:00:00 64.50 -999.00 73.00 -1.00 NULL

and after 2012-10-29 09:00:00 64.50 NULL 73.00 -1.00 NULL

[ThermoMan]

Putting a NULL in worked. Blank spot in the chart, but NULL is a better default than -999

The previous and next outdoor temperatures had a large gap of 8.5 degrees in an hour. Previous 40.9 Next 49.4 (part of an upward trend)

The relative humidity had a similar change from 72 down to 56.

If the code had a built in retry on failure, that would reduce the number of these types on entries, but what would make a better process for dealing with it? A NULL - showing that we missed data, or, if the previous and subsequent data is present an interpolated number? It would be 45.15 in this case.

[unlstorm]

NULL at least shows there was a problem. Interpolating hides the problem. With NULL's you may be able to see the frequency of occurrence. I have more issues with dropouts both on the wifi and outdoor weather. Although, my outdoor weather significantly improved when I switched to a specific weather station. I would like to see a retry happen if the first attempt produces a NULL. After that a NULL is probably acceptable for the long haul.

On Mon, Oct 29, 2012 at 4:28 PM, ThermoMan notifications@github.com wrote:

Putting a NULL in worked. Blank spot in the chart, but NULL is a better default than -999

The previous and next outdoor temperatures had a large gap of 8.5 degrees in an hour. Previous 40.9 Next 49.4 (part of an upward trend)

The relative humidity had a similar change from 72 down to 56.

If the code had a built in retry on failure, that would reduce the number of these types on entries, but what would make a better process for dealing with it? A NULL - showing that we missed data, or, if the previous and subsequent data is present an interpolated number? It would be 45.15 in this case.

— Reply to this email directly or view it on GitHubhttps://github.com/ThermoMan/3M-50-Thermostat-Tracking/issues/3#issuecomment-9886148.

[devzzm]

I would say retry and if it fails again, then null

ThermoMan notifications@github.com wrote:

Putting a NULL in worked. Blank spot in the chart, but NULL is a better default than -999

The previous and next outdoor temperatures had a large gap of 8.5 degrees in an hour. Previous 40.9 Next 49.4 (part of an upward trend)

The relative humidity had a similar change from 72 down to 56.

If the code had a built in retry on failure, that would reduce the number of these types on entries, but what would make a better process for dealing with it? A NULL - showing that we missed data, or, if the previous and subsequent data is present an interpolated number? It would be 45.15 in this case.

---

Reply to this email directly or view it on GitHub: https://github.com/ThermoMan/3M-50-Thermostat-Tracking/issues/3#issuecomment-9886148

[unlstorm]

Ok, I think I have resolved my transient issues. I was getting 5-6 transients per day. Not sure if this is an artifact of me running on my local LAN with web server or what. Anyway, I tweaked t_lib.php to add a checkTransients function and do a retry if I detect one. I also tweaked updating rows into the temperatures table to make the minutes field of the time always 0. This should take care of my times that were off a minute or two. Finally, I also changed the default outdoor temperature to NULL. When it was -999.0 I would get occasional hiccups with that as an outdoor temperature. This really messes up the charts and I just changed it to NULL for now.

[unlstorm]

So, I've been monitoring and logging my transient issues for a while now. I'm getting 9-12 transients per cron hour. This is with the "getStatData ('/tstat');" call where any of the json fields has a -1 value. I think I'm going to further tweak the t_lib.php code to query in a loop of up to 5 times until all fields come back without transients. Perhaps I have a flaky wifi connection or something else driving these transients. Either way, it's putting erroneous data in my database and I'd like that to be resolved and made robust.

[ThermoMan]

If possible, log to a text file the raw output, then you can see if the json is even formatted correctly. That in itself might be the trigger for a retry.

[unlstorm]

I haven't done the raw output but my check for transients and retry up to five times has now been running for a few days and it has completely stabilized my issue. I parsed through my log files to see how many retries are occurring before success. Most of the time 1 retry is enough but I have an occasional 2 or 3 retries that have showed up. Fortunately, it has never gotten to the point of 4 or 5 retries yet. This is giving me much better records in my database for temperatures and hvac cycles. Basically, I look for any transients (-1) in the /tstat query. If found it repeats the query until all json fields have no transients. Makes me much happier now. I'm just baffled why others don't see that as much as me. These changes should be showing up in my current branch.

[ThermoMan]

I think this is resolved by the code changes that closed the other issue too. Will close this when I get to posting my latest code tomorrow or so.

[ThermoMan]

Is this still a thing? I've not had that particular timing issue since.

[unlstorm]

I haven't seen any more hiccups in my database so I assume this is all good and resolved. I probably should re-sync with the latest master code again one of these days.

[ThermoMan]

When I committed the last round I was using the new GitHub GUI and ended up committing some code that I had not intended to. I think I fixed the errors form the half-commit of the feature. If you resync I'd be interested in knowing. The DB structure was not affected.