Unreliable "start" filter

[BastienDurel]

Hello,

I'm trying to load aranet data points in a database, and I tried to use the "start" filter to not get the already recorded points, but it's not working correctly :

I start by getting the last mesurement :

cur.execute('select max(time) from data')
last = cur.fetchone()[0]
if last != None:
    filter["start"] = last

And then I call aranet4.client.get_all_records(device_mac, entry_filter=filter)

I often get all recorded datapoints in response.

I reproduced this with aranetctl :

(env) bastien@sandworm:~/projects/aracollect$ aranetctl -s 2022-10-07T16:33:51 -r E5:47:86:60:6A:4D
{'device_mac': 'E5:47:86:60:6A:4D', 'scan': False, 'url': None, 'records': True, 'start': datetime.datetime(2022, 10, 7, 16, 33, 51), 'end': None, 'output': None, 'wait': False, 'last': None, 'temp': True, 'humi': True, 'pres': True, 'co2': True}
Next data point will be logged in 91 seconds
-------------------------------------------------------------
Device Name    :        Aranet4 1CCCF
Device Version :              v0.4.14
-------------------------------------------------------------
 id  |        date         |  co2   | temp | hum | pressure |
-------------------------------------------------------------
1637 | 2022-10-07T16:35:51 |    624 | 20.1 |  51 |   1012.5 |
1638 | 2022-10-07T16:37:51 |    639 | 20.0 |  51 |   1012.5 |
-------------------------------------------------------------
(env) bastien@sandworm:~/projects/aracollect$ aranetctl -s 2022-10-07T16:35:51 -r E5:47:86:60:6A:4D
{'device_mac': 'E5:47:86:60:6A:4D', 'scan': False, 'url': None, 'records': True, 'start': datetime.datetime(2022, 10, 7, 16, 35, 51), 'end': None, 'output': None, 'wait': False, 'last': None, 'temp': True, 'humi': True, 'pres': True, 'co2': True}
Next data point will be logged in 75 seconds
-------------------------------------------------------------
Device Name    :        Aranet4 1CCCF
Device Version :              v0.4.14
-------------------------------------------------------------
 id  |        date         |  co2   | temp | hum | pressure |
-------------------------------------------------------------
   1 | 2022-10-05T10:03:50 |   1349 | 20.8 |  58 |   1010.5 |
   2 | 2022-10-05T10:05:50 |   1676 | 21.8 |  54 |   1010.6 |
   3 | 2022-10-05T10:07:50 |   1676 | 21.8 |  54 |   1010.6 |
   4 | 2022-10-05T10:09:50 |   1676 | 21.8 |  54 |   1010.6 |
[...]
1637 | 2022-10-07T16:35:50 |    624 | 20.1 |  51 |   1012.5 |
1638 | 2022-10-07T16:37:50 |    639 | 20.0 |  51 |   1012.5 |
-------------------------------------------------------------

It seems if the result set would contains less than 2 rows, all datapoints are returned

[BastienDurel]

In client.py, the _calc_start_end(datapoint_times: int, entry_filter) function should allow start == end to let getting 1-value resultset :

        if 0 < time_start <= end:
            start = time_start

and

        if start <= time_end < end:
            end = time_end

You may also setting start at end + 1 if no value is past requested start, and handle this in _all_records by returning an empty set:

    if begin > end:
        return Record(dev_name, dev_version, log_size, rec_filter)

It's something I would expect with a filter

[Anrijs]

Try v2.1.1 It should return empty result set, if dates are out of range.

[BastienDurel]

I upgraded to 2.1.1, but it crashes

{'start': datetime.datetime(2022, 10, 7, 17, 9, 50)}
Next data point will be logged in 101 seconds
Traceback (most recent call last):
  File "/home/bastien/projects/aracollect/./collect.py", line 23, in <module>
    history = aranet4.client.get_all_records(device_mac, entry_filter=filter)
  File "/home/bastien/projects/aracollect/env/lib/python3.10/site-packages/aranet4/client.py", line 677, in get_all_records
    return asyncio.run(_all_records(mac_address, entry_filter))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/bastien/projects/aracollect/env/lib/python3.10/site-packages/aranet4/client.py", line 618, in _all_records
    begin, end = _calc_start_end(log_points, entry_filter)
  File "/home/bastien/projects/aracollect/env/lib/python3.10/site-packages/aranet4/client.py", line 482, in _calc_start_end
    if filter_start <= timestamp:
TypeError: can't compare offset-naive and offset-aware datetimes

[Anrijs]

datetime objects now needs to have timezone info. Some examples might be outdated. For now, you could add current timezone to start/end objects like this:

now = datetime.datetime.now().astimezone()
start = start.replace(tzinfo=now.tzinfo)

I might update library later to do this automatically.

[BastienDurel]

I went for

    if last.tzinfo is None:
        last = pytz.utc.localize(last)

and it does not crashes anymore (my fault, I don't use python), but I get a lot of out-of range datapoints :/ (I've put timezone in DB too)

{'start': datetime.datetime(2022, 10, 8, 17, 53, 24, tzinfo=datetime.timezone(datetime.timedelta(seconds=7200)))}
Next data point will be logged in 73 seconds
Ignored 2336 values
Saving 1 values

NB: full code here : https://git.geekwu.org/aranet/collect/-/blob/master/collect.py

[Anrijs]

I took a closer look. aranet4.client.get_all_records will always return all records, but only records in time range will be read. Try now version v2.1.2 and add remove_empty=Ture to get_all_records function:

history = aranet4.client.get_all_records(device_mac, entry_filter=filter, get_all_records=True)

If remove_empty is set to true, it will remove all records out of specified time range.

[BastienDurel]

It works, thanks :)