Error parsing a file timestamp in getfiledate()

[peterk]

Thank you for your great project. I followed the guide to add tag blocks to a simple AIS string from the https://github.com/GlobalFishingWatch/ais-tools project.

$ echo '!AIVDM,1,1,,A,15NTES0P00J>tC4@@FOhMgvD0D0M,0*49' | ais-tools add-tagblock -s my-station > datesample.nm4

The resulting file contains the single line:

\c:1722621763484,s:my-station,T:2024-08-02 18.02.43*26\!AIVDM,1,1,,A,15NTES0P00J>tC4@@FOhMgvD0D0M,0*49

I tried the following script using aisdb to load the file into an sqlite db:

import aisdb
filepaths=['./datesample.nm4']

with aisdb.DBConn(dbpath = './datesample.db') as dbconn:
    aisdb.decode_msgs(filepaths=filepaths, dbconn=dbconn, source='TESTING')

This gives the following error:

generating file checksums...
checking file dates...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/.../aisdb/database/decoder.py", line 249, in decode_msgs
    filedates = [getfiledate(f) for f in raw_files]
  File "/.../aisdb/database/decoder.py", line 249, in <listcomp>
    filedates = [getfiledate(f) for f in raw_files]
  File "/.../aisdb/proc_util.py", line 275, in getfiledate
    fdate = datetime.fromtimestamp(epoch).date()
ValueError: year 56557 is out of range

I expected the file to be parsed and loaded into the db.

Running Python 3.10.12 on macOS.

[tsuzzy]

Hi @peterk, sorry for our late reply. Thank you for targeting this issue. We have fixed the long timestamps processing, and now AISdb supports the extraction of seconds (10-digit), milliseconds (13-digit) and microseconds (16-digit) timestamps in raw AIS messages.

In your example, after adding tagblocks, please run the following code as before:

with aisdb.DBConn(dbpath = './datesample.db') as dbconn:
    aisdb.decode_msgs(filepaths=['./datesample.nm4'], dbconn=dbconn, source='TESTING', vacuum=True)

The decoded messages will be stored in a table named by year and month in the SQLite database.

We thank you for your help with finding and reporting this issue. Please let us know if you have any questions!