Enhancement - check for excessive change in temp_out

[borpin]

In the same way as rain is checked, it would be very useful to check for excessive changes in temp_out that will likely indicate corrupt data. A check is already made for some aspects of inconsistent temp_out figures that would indicate corrupt data and this would enhance it. I have seen values vastly different from previous values, that were corrupt data.

https://github.com/jim-easterbrook/pywws/blob/33e10ef279705d3c0c37d97b1019c2bf962fcfd0/src/pywws/process.py#L278

[jim-easterbrook]

This sort of thing is very station dependent - I don't get them on my current station but have had them in the past. See the remove_temperature_spikes.py example for how you could do this in a user calibration module.

[borpin]

Hi Jim, agreed, but a jump in temp is almost bound to be corruption and extremely unlikely to be genuine (I'm talking 100's of degrees).

I have only had this since I had to change stations due to the old one failing (slightly different model).

Where do I find the installation directory and the examples?

Cheers

[edit]

It is happening a couple of times a month and will be a real pest to run this on 10 years of data all the time 😭.

[borpin]

This sort of thing is very station dependent

How about an option to enable it with a user configurable threshold? I know more complexity 🙂

[jim-easterbrook]

Most people do not need temperature spike removal so it would add unwanted complexity to include it. I provide an example user calibration module for the few people who need it.

The example modules are with the example templates etc. (e.g. /usr/local/lib/python3.5/dist-packages/pywws-20.10.0-py3.5.egg/pywws/examples.) The user calibration is run when data is processed, i.e. if you reprocess all your data it will all pass through your user calibration module. In normal use it's just new data that's calibrated as it arrives.

You might find the documentation of the pywws.calib module useful. https://pywws.readthedocs.io/en/latest/api/pywws.calib.html

PS You can apply the despiking algorithm to a limited date range (e.g. only since you changed station) in your user calibration module. Some of the other example modules show how to do this.