Packets processed in not correct order

[GSzabados]

Hi,

I am using an Eurochorn EFWS2900 (Fine Offset clone) with interceptor in wu-client mode. The whole setup is running on a RPi Zero W.

I have faced an issue where a lot of wraparound event has been detected when it was raining outside. The reason for the wraparound was that the packets were not coming in order. Some packets were repeating itself later on. It caused the rain value to wraparound back to previous state and add the rain again and again.

Would be possible to put a block to the beginning of the parsing to check the dateutc and drop the complete packet if it is from an earlier time than the previously processed packet.

It would fix the rain issue, and it would fix any incorrect order data to be recorded into the database.

Thanks,

Gábor

[matthewwall]

the root cause may be more complicated. if the station is reporting to multiple destinations, then that is probably the source of the duplicate information.

as mentioned in issue #58 i think i need to restructure the interceptor so that it will only capture one kind of packet, not every kind of packet. for example, if you tell it to capture wu packets, then it would ignore any ecowitt or ambient packets. if you tell it to capture ecowitt packets, then it would ignore any wu packets.

if you could capture the detailed logs (set debug=1 then restart weewx) that show the duplicate packets happening, that would help me ensure that the interceptor avoids the duplicates.

[GSzabados]

Hi,

I have done some modifications on my side of the interceptor code to avoid the wrong order by checking time stamps.

The weather station is a FineOffset clone, Eurochron, sold by Conrad in Europe.

The issue might comes from my RPi Zero W. I use the RPi as a Wifi repeater. It creates a dedicated hotspot for the weather station and passes all packets to the another wifi network which is connected to the internet. Interceptor runs in sniffing mode.

The weather station sends only to WU, not to any other sites. (I would love if the FW would allow to set a local IP to send to, but suddenly it doesn't.)

The RPi Zero W might be too weak for the task and mixes up the order of packets.

I can post a log if you need it.

Best Regards,

Gábor Szabados

[matthewwall]

gábor, please post a log. always better to have real data on which to test! you could also post a diff with your modifications - since you have a working implementation then we might be able to incorporate it into the driver.

[GSzabados]

Hi,

This was my little tweak:

            # get the rain this period from total
            if 'rain_total' in pkt:
                newtot = pkt['rain_total']
                datetime_now = pkt['dateTime']
                if datetime_now < self._last_datetime:
                    newtot = self._last_rain
                    loginf("Date order issue %s is earlier than the last time %s" % (datetime_now, self._last_datetime))
                    datetime_now = self._last_datetime
                pkt['rain'] = self._delta_rain(newtot, self._last_rain)
                self._last_datetime = datetime_now
                self._last_rain = newtot

Looking at your code, is it possible that it would build data from the packets wrongly? Mixing up the packets I mean.

[GSzabados]

I have just looked at the log option, but during the past few days the duplication happened only 6 times total and absolutely randomly during the day.

Jan 2 08:01:31 WeeWX weewx[2913]: interceptor: MainThread: Date order issue 1577948461 is earlier than the last time 1577948477 Jan 2 08:06:00 WeeWX weewx[2913]: interceptor: MainThread: Date order issue 1577948685 is earlier than the last time 1577948717 Jan 4 02:55:53 WeeWX weewx[2913]: interceptor: MainThread: Date order issue 1578102894 is earlier than the last time 1578102910 Jan 4 09:05:13 WeeWX weewx[2913]: interceptor: MainThread: Date order issue 1578125054 is earlier than the last time 1578125078 Jan 4 09:18:00 WeeWX weewx[2913]: interceptor: MainThread: Date order issue 1578125838 is earlier than the last time 1578125854 Jan 4 20:48:56 WeeWX weewx[2913]: interceptor: MainThread: Date order issue 1578167166 is earlier than the last time 1578167294

[matthewwall]

please try interceptor v0.51 (as of commit 9f8c0cc)

i added a generic test for out-of-order packets, not just for rain