search

aerosense-ai / data-gateway

Data influx for Aerosense.
https://www.aerosense.ai/
Other
3 stars 1 forks source link

Data loss while writing to file #19

Closed thclark closed 2 years ago

thclark commented 2 years ago

Recreated by @thclark after migration from gitlab - originally filed by Rafael Fischer Jun 17, 2021 12:52pm GMT+0100

Bug report

What is the current behavior?

Once, the data of the newly added constat "sensor" got not written to the file completely. It doesn't happen regularly. The produced .json file can be found here

I think that the issue is the writing to the file because the file stops abruptly (\n79.64799072265616,-30.104429244995117,-29,-9,28600,\n79.69299072265616,-30.070194244384766,-29,"}) without finishing the current sample. Note that the other sensor data are complete up to timestamp 125s.

Curiously, also the end of the console output is strange (see below) with the "Lost set of Constat packets: -125733.03588867171 ms gap" message after the "stopping gateway" message. The gap duration of -125s would happen when the current_timestamp variable is set to zero somewhere.

I think what happened is, that the constat packet arrived in some critical moment of shutdown for the python application.

Your environment

Console log:

(venv) PS C:\Users\Raphael Fischer\OneDrive\PBL\AeroSense\data-gateway\data_gateway> gateway start --interactive

[data_gateway.cli | INFO | 2021-06-17 12:05:28,950 | logging_handlers | 9496 | 15860] Using local logger.

[data_gateway.cli | INFO | 2021-06-17 12:05:28,950 | cli | 9496 | 15860] Using default configuration.

[data_gateway.cli | INFO | 2021-06-17 12:05:28,966 | cli | 9496 | 15860] Starting gateway in interactive mode - files will not be uploaded to cloud storage but will instead be saved to disk at '.\data_gateway' at intervals of 600.0 seconds.

Successfully updated handles.

startMics

Successfully updated handles.

Received first set of Constat packets

Lost set of Constat packets: 30.70434570312841 ms gap

startMics

Received first set of Mics packets

startBaros

Received first set of Baros_P packets

Received first set of Baros_T packets

startIMU

sReceived first Acc packet

Received first Gyro packet

tartAnalog

Received first Mag packet

Received first set of Analog Vbat packets

Lost set of Mics packets: 538.2341308577452 ms gap

Lost set of Baros_P packets: 515.991210937429 ms gap

Lost set of Baros_T packets: 515.991210937429 ms gap

Lost set of Mics packets: 539.6538085929024 ms gap

Successfully updated handles.

stop

[data_gateway.cli | INFO | 2021-06-17 12:07:46,315 | cli | 9496 | 15860] Stopping gateway.

Lost set of Constat packets: -125733.03588867171 ms gap

time-trader commented 2 years ago

Fixed with #30