Add Optional Filter for Erroneous Data

[leoherzog]

@rubengte Sure, it should be possible. What datasource are you using, and what are the upper limits that you'd like to ignore? If the number that the datasource returns is higher than your predefined ceiling, what do you want to happen? Clamp it to the ceiling, or set it to 0, or something else?

That would be amazing. I'm getting the data from Wunderground to windy and to Windguru. Not sure about setting a specific amount, i would prefer (if possible) to set a compare to previous, i attached an image to better illustrate what i mean.

If this is not possible we could use 100km/h for the wind and maybe 50mm for the rain however, this site can (although rarely) reach 100km/h wind.

What we could do is ignore the reading and wait for the next or repeat the previous until it's back to "normal", something like this, what do you think?

These "aberration" readings happen every week, at least once, and it messes all the graph readings, so it would be really super to filter them out.

P.s. i would like to contribute also to the project, you're doing a fantastic work and I would like to send a contribution, however can't find a way. Do you have revolut?

Thanks again Leo

Originally posted by @rubengte in https://github.com/leoherzog/WundergroundStationForwarder/issues/31#issuecomment-2412362725

[leoherzog]

This sounds like hardware failure. I had this happen once with one of my stations (a WeatherFlow Tempest) and it turned out that a few droplets of rain were wiggling in the gentle breeze, tricking the ultrasonic wind sensor into thinking I was intermittently in a hurricane :sweat_smile:. A quick dry with the leaf blower + a reboot fixed the issue for me.

That being said, if you wanted to do this on the Apps Script side... I kind of like that, in principle, this code only takes readings in and forwards them on, without having any opinions about averaging, changing, etc those readings. I'll need to think about whether or not this should be built-in as a first-class feature.

In the meantime, if you take v2.8.1 and add something like this within the refreshFromIBM_() function at Line 203:

let lastrun = CacheService.getScriptCache().get('conditions');
if (lastrun) {
  lastrun = JSON.parse(lastrun);
  if (conditions.windSpeed.kph - lastrun?.windSpeed?.kph > 100) conditions.windSpeed = lastrun.windSpeed;// change greater than 100kph since last time
  if (conditions.windGust.kph - lastrun?.windGust?.kph > 100) conditions.windGust = lastrun.windGust;
  if (conditions.precipRate.mm - lastrun?.precipRate?.mm > 50) conditions.precipRate = lastrun.precipRate;
}

This says that, if the difference between last time and this time is a jump of greater than x kph or x mm, then set it to last time's readings instead.

[rubengte]

Brilliant. You're a genius Leo 🤩 I'll give that one a go, although just by the look of the code it should definitely work.

For some reason that "simple solution code" you came up with reminded me of that story of the engineer charging 1001€ for a machine repair that was just turning a switch. He said it was 1€ to turn the switch and 1000 for knowing which one 😅

This specific weather station is from Vevor, and it's brand new, however, i have seen this exact same issue with a "top branded" Davis stations also, on Wunderground. There's one station in a neighbor island that this happens almost every week.

Thanks again and will implement that now.

Best,

Ruben

[leoherzog]

If your neighbor also experiences these readings, then maybe you just have intermittent 100+km/h winds!

[rubengte]

Its's not 100 km/h Leo, mine registers sometimes 200 and theirs 400km/h

[rubengte]

You can see it's a communication flaw because it goes from 20 to 100/400 in a minute then back to 20

[rubengte]

Here's a print to better illustrate

[rubengte]

I do Paragliding and I've seen this discrepancies in top Garmin GPS also, sometimes when we downloaded the track record to the computer we could see readings of airplane kinds of speed from our paragliding flights. Not entirely sure what causes this surges of misplaced readings.

[leoherzog]

My silly joke didn't translate well into text. I know that it's a mistake in the readings. :)

Let me know if that fix helps!

[rubengte]

Heheh, irony sometimes passes by unnoticed 😋 I will for sure, already implemented it and watching the "game" live 😁 Thanks again for being there 🙌

[rubengte]

Hey @leoherzog Leo, just to let you know that your code worked perfectly. Last night there was a spike in the readings to Wunderground and the code filtered it brilliantly through @vhornik Windguru and windy 💖🙏

Thank you again.

Just wish that some day soon Wunderground will implement some a.i. filters to avoid this from the "source", until then, this is the best solution.

I was thinking that this is somehow ironic, because the secondary platforms (Windguru and windy) in this case get more reliable weather history data because of your script and custom code than the source (Wunderground) 😋 Hehe

Have a great day 🙌 Best,

Ruben