NDCLab / lab-devOps

NDCLab mgmt and operations
GNU Affero General Public License v3.0
0 stars 0 forks source link

Ensure that timestamps are shared between pending_files and pending_errors CSVs #270

Closed nolanwelch closed 1 month ago

nolanwelch commented 1 month ago

These files should have the names pending-files-[timestamp].csv and pending-errors-[timestamp].csv respectively, where timestamp is the same between both. The function that creates these files takes a timestamp as a parameter to reflect this. However, in the current implementation, the two files are not created at the same time.

This suggests that it would be a good idea to instantiate the timestamp variable outside of both checked and raw data processing, and then pass it as a parameter of the functions that handle each part of the pipeline.

nolanwelch commented 1 month ago

Resolved by 2d34a07b31a0f48365e97ef39ea2d925c29004c5.