Linear Timescales not showing gaps

[michaelundwd]

I noticed recently when I had some missing data (reconfiguring my weather station wiring) that the resulting data display did not show missing data as a gap as the standard weewx report. Instead, the timescale was made non-linear and you needed to look carefully to spot it. Would it be possible to use a different graph layout? I see Highcharts have one entitled "Time data with irregular intervals".

[poblabs]

With the typical caveat that I work on this project in my spare time, I can't guarantee any major changes to the charts. If you want to submit a PR to make this an option, I invite that.

Can you link what you've found? Maybe some screenshots of what you're seeing?

[michaelundwd]

Here's a link to what I was l looking at on Highcharts https://www.highcharts.com/demo/spline-irregular-time - screenshot attached below. It does not show gaps, but indicates the irregular nature of the data. Given that for operational weather stations, there will probably be very few occurrences of missing data, it's probably not going to be worth the effort.

Look forward to seeing the upcoming chart changes anyway.

[poblabs]

I might still be a little lost on the problem. If your station goes offline there's a gap in the data rather than a connecting line? Is that the concept?

[michaelundwd]

Sorry, misunderstood what you were asking for. The problem arose on the graphs on the home page. Instead of the horizontal timescale being every 2 hours, it jumped according to the missing points. The result is the data point are linearly represented. I have seen the effect in Excel charts where you have to be careful to se a scatter plot if the measurements are not taken at regular intervals. I will kill weewx for a couple of hours over the weekend, and replicate the problem for you.

[michaelundwd]

You asked "If your station goes offline there's a gap in the data rather than a connecting line? Is that the concept?"

My point is that there is neither a gap in the data or a connecting line, because Highcharts only displays the data-points it receives and hence the horizontal time-scale is no longer linear if there are no data points for the missing times. I stopped WeewX for an hour (15:30 to 16:30hrs gmt), and the non-linear time-scale occurred on today's graphs on both the home screen and graphs/today. My screenshots are for temperature, although it showed on other graphs as well. The Belchertown shots show a step drop in temperature around 16:00 hrs and both show missing values on the time-axis.

Home/Today

Graphs/Today

The standard WeewX report shows a gap in the data around 16:00 hrs,

Finally, the UK Met Office WOW site interpolates between15:00 and 17:00 with no reading at 16:00.

I hope that clarifies the issue I was raising - I guess the question I should be asking is whether this behaviour is what you intended. I appreciate the work that goes into graphing data, which is why I was so pleased to discover you had done all the hard work.

Michael

[poblabs]

The majority of the graphs and how they work are original work from Gary (https://github.com/gjr80/weewx-highcharts) and I have permission to include his work with my skin. So the charts are almost a "subskin" if you will with some styles changed and a little bit of an extension on them to include additional observations.

Sometimes I get a little "green" on the charts since they're pretty powerful.

The screenshots help. Looks like highcharts is attempting to fill in the gaps with just an arbitrary line - as opposed to weewx which shows the gap. I stopped my weewx for 10-15 mins and I see the arbitrary line connecting it as well. Which confuses me a little because I saw a gap in the line a few days ago when I was working on some updates.

So I think the answer to your question is yes, this is expected behavior (right now), but what is "right"? Should there be a gap in the graph which is a more realistic representation of the data?

[michaelundwd]

Thanks for the reply. Clearly showing gaps would be more realistic. However, I expect most uses of the charts are for operational stations, which will be expected to run 24/7. It was only when I took mine down for rewiring (to include a UPS!) that I happened to see the glitch. So does it really matter?

As I see it, there are only 2 solutions; change the data that goes to the graphs (no gaps in the time data) or use different Highchart graphs. I hadn't appreciated the origin of the graphs - maybe Gary has a view.

[poblabs]

I found how highcharts handles gaps and I made a quick test change to the temperature chart. This might more accurately reflect the data?

My thought is: If no data, show a gap instead of trying to "make up data" with the arbitrary line.

[michaelundwd]

I agree, much better to show a gap and not distort the timescale which is what currently happens.

[poblabs]

So maybe the fix is to apply this gap treatment on all line charts?

Here's what it looks like with weewx stopped for an hour.

Which matches the weewx built in images. Though harder to see since you can't click to zoom in on those 😃

[michaelundwd]

Yes, agreed. Go for it!

[poblabs]

@michaelundwd I just pushed the change. When you get some time give the change a test.

1. Stop weewx
2. Download this GitHub repo as a zip file from the front page here, select Clone or Download > Download Zip 2a. Or use whatever method you're used to for getting file updates.
3. Copy the updated skins/Belchertown/js files to your skins/Belchertown/js folder.
4. Restart weewx and wait for an archive interval.

If you have missing data the gap should show - and hopefully doesn't show any other time. That's why I'm looking for a test.

Let me know

[michaelundwd]

Ran a test today with a 2 hour stoppage, and got the following Running WeewX 3.8.2 and Belchertown 0.9rc1 with the latest js files. Same effect with all Today's graphs

[poblabs]

Which looks like the expected output to me?

It looks weird with the current zoom level of 00:00 to current. But you can click your mouse to drag an area which will zoom in.

I'm having a problem with my station when it is in the negative degrees, so here's my 10 hour gap from today.

Here's how Weather Underground treated it that same timeframe:

[michaelundwd]

Afraid it's not right there - see this for more detail What has happened to 12:00, 13:00, 14:00 hrs?

[poblabs]

Good question.

I'm assuming the 12:00 is your planned stoppage?

The 13:00 and 14:00 looks like it could be from a gapsize calculation.

This could be an issue with the highcharts gapsize measurement. I set it at 5 minutes in milliseconds. I'm not sure what it's supposed to be.

If you're feeling up to it, try editing the skins/Belchertown/js/highcharts-dayplots.js.tmpl file. At line 110 you'll see the gapSize for the temperature plot.

Try changing that to 10 minutes in milliseconds which is 600000 and restart weewx. Force a skin update with sudo wee_reports then see what the graphs show for 13:00 and 14:00?

I think this is the setting that needs to be fine tuned.

[poblabs]

For reference, here's the documentation on gapSize and gapUnit. Think we need to find the right combination of these two values to reduce false positives.

https://api.highcharts.com/highstock/plotOptions.area.gapSize

[michaelundwd]

Stoppage was from ~11:50 to 14:45; I wanted it to show up! Tried the edit, made no difference Commented out gapsize and gapunit to use the default settings of a relative gap just to see - no joy there either

[poblabs]

I'm still testing and trying to understand the gapsize. So far the settings I have in the commit above seem to be working for me. I don't know why they aren't working for you during the gap you created.

I woke up this morning and saw gaps in my charts, so I dug into the database further and saw that the gaps are valid gaps.

No idea why weewx didn't record those observations, but the gaps are validated.

I'll try more testing with intentional gaps.

If/when you try again, can you verify the gaps are in the database?

[michaelundwd]

Apologies for the delay in replying.

I checked the database for the day when I turned off WeewX. There was a gap of 32 readings corresponding to 2hrs 40mins.

[poblabs]

So... did those gaps in the database match up to gaps in the chart? If so then the setting is a good one.

The skin doesn't cause gaps. So if you're seeing database gaps, you have something else going on with your system.

[michaelundwd]

My attachment of 21 Jan showed there was no gap in the way that you are showing, just a distortion of the timescale that I have had all along. No 12:00, 13:00 or 14:00 hrs on the chart. And yes, the missing data corresponds to the time that the timescale distortion happens. I actually stopped WeewX for that length of time, so the distortion is a genuine result of the data.

Any suggestions as to what I should do as you are not getting the problem

[poblabs]

No, I have no suggestions right now. I was hoping to be able to replicate the data so I can try to get on the same page. Doesn't seem to be happening. So I'm confused as to where to go next.

You could play around with those gapSize and gapUnit values until something looks valid?

[michaelundwd]

Now I know what to change, I'll do some tests and let you know how I get on. It may be a few days before I will have anything definitive though.

[poblabs]

I'm struggling with this and have reverted the change to remove it. I'm planning on releasing skin 0.9 before the weewx upgrade to 3.9.0 later this week so the skin is ready for those changes. The gapSize and gaps in the charts will not be in skin v0.9 since it's not ready yet.

How has your testing gone?

[michaelundwd]

Afraid my testing has drawn a blank. Latest results as follows with a confirmed 2 hr gap today between 08:40 and 10:40. Temperature: missing x-axis hrs 09:00 and 10:00, no gap in the graph, just a temperature jump as before Windspeed: missing x-axis hrs 09:00 and 10:00, but there is a gap in the graph Rainfall: missing x-axis hrs 09:00 and 10:00, but there is a gap in the graph Wind Direction: 09:00 and 10:00 values are present on the x-axis, gaps in the points - ok Barometer: missing x-axis hrs 09:00 and 10:00, but there is a gap in the graph Solar Radiation: missing x-axis hrs 09:00 and 10:00, but there is a gap in the graph

I will install the new WeewX and charts and test again with a clean installation of both to rule out any funnies in my system/configuration. I guess that means leaving this issue open for a while. Is that ok?

[poblabs]

Looks like your gaps line up on everything except Temperature. That's bizarre.

Let's plan on upgrading to the new weewx release later this week / early next. Then upgrade to the skin 0.9 when I release it around the same time frame. That way we're back to basics. Then we can continue to look at this.

[michaelundwd]

Not quite. The only one that behaves as expected is the wind direction - points rather than a line All the others have missing hours, and where there is a gap it is a vertical gap not a horizontal one. Temperature shows no gap at all.

Lets go back to basics

[michaelundwd]

I had one more go and think I have cracked it.

I found a typo where I had changed the gapsize but the real change requires an extra property called ordinal in the xAxis object for temperature in skins/Belchertown/js/highcharts-dayplots.js.tmpl

ordinal: false

See https://api.highcharts.com/highstock/xAxis.ordinal for details. That explains the problems I have been having.

Hope that helps.

[poblabs]

I had the ordinal: false setup in my test last night, but it made the rain plot go sideways. It was not spanning the full width of the chart.

Do you think the gap size should apply to all charts?

If so, apply it to your rain chart, restart weewx and let me know if it does the same thing to you.

[michaelundwd]

Yes, got the same results as you. Solved it by restoring the commented value for minrange in var options rain (~line 726) from 86400000 (24 hrs) to 900000 (15 mins - the same as all the other graphs). The 24hr setting meant that the x-axis was for the last 24 hrs, but it only had data for the current day. With that done, I think all the var option objects need ordinal setting to false. It makes no difference to wind direction that is a scatter plot that behaved anyway.

HTH

[poblabs]

Sounds good. Let me see if I can replicate some of that today, and will post some screenshots for verification

[michaelundwd]

Feeling brave, I also took a look at the week, month and year charts tmpl files. They all have gapsize and gapunit settings, but none of them have the ordinal setting.

[poblabs]

I'll take a look and post my results. Instead of stopping my station for a full day, i'll just delete a day out of my test database - which will replicate the same behavior.

[poblabs]

Ok, I think this might be looking good now...

For reference, a gap should show if the following time range is missing:

• Day: 5 minute gap
• Week: 1 hour gap
• Month: 24 hour gap
• Year: 24 hour gap

Here's some screenshots of those gap ranges.

Day:

Week:

Month:

Year:

I'm going to commit the changes and close this one out. Thanks for the help!

[michaelundwd]

My pleasure - I enjoyed the challenge! Thanks for sorting it.

M