search

fermi-ad / datalogger-to-ml

Scripts for requesting AD Controls data logger data and transforming those to the desired ML output format and destination.
3 stars 0 forks source link

Store data in memory before writing to disk #24

Closed beauremus closed 3 years ago

beauremus commented 3 years ago

In order to limit the number of writes to disk, we should store data in memory before writing to disk.

This is ok because we know the size of the requests. This will become memory-hungry as the request size grows.

This change is affected by a bug in the Python DPM data logger DAQ. It has been found that the penultimate block of data is sometimes returned after the ultimate empty data message to indicate that there are no more data.

beauremus commented 3 years ago

This was solved as part of the refactor. 8b90a0ff3fe306f37f36e1aeeba2b75baa73a90f