AISViz / AISdb

AISdb Python package for smart AIS data storage and interaction.
https://aisviz.cs.dal.ca
GNU Affero General Public License v3.0
12 stars 0 forks source link

Error processing .nm4 and .nmea files #93

Open tsuzzy opened 1 week ago

tsuzzy commented 1 week ago

When using the function aisdb.database.decode_msgs() to decode and process .nm4 and .nmea files, the following message is prompted:

error processing path/to/filename.nm4, skipping checksum...

This results in an empty table being created in the SQLite database.

The issue can be reproduced with the following code,:

import aisdb

filepath = ['./sample_data.nm4']

with aisdb.DBConn(dbpath = 'sample.db') as dbconn:
    aisdb.decode_msgs(filepaths=filepath, dbconn=dbconn, source='TESTING')

Content in sample_data.nm4 is:

\s:41925,c:1635731906,t:1635731966*63\!AIVDM,1,1,,,11mg=5@000<Oe=Cbv:E@08L`00Ra,0*41
\s:41925,c:1635731906,t:1635731966*63\!AIVDM,1,1,,,15Utl40P0:<2TeiUEfmbk?vd08=T,0*78

Expected the file should be processed and data should be populated in the database.

Actual output:

generating file checksums...
checking file dates...
creating tables and dropping table indexes...
Memory: 21.92GB remaining.  CPUs: 12.  Average file size: 0.00MB  Spawning 4 workers
processing ./sample_data.nm4
saving checksums...
error processing ./sample_data.nm4, skipping checksum...
cleaning temporary data...

As observed, the message indicates a failure in processing the file, leading to the checksum being skipped and an empty table in the database. Running the test script test_002_decode.py produces a similar output. However, the test passes because this failure does not raise any errors.

Environment:

gabrielspadon commented 4 days ago

Related to #78 , please address this issue as well.

gabrielspadon commented 3 days ago

Also related to #85 , please address this issue as well.