bcgov / nr-rfc-grib-copy

Demo of using github actions to copy / process raw data
Apache License 2.0
2 stars 2 forks source link

Track Events Emitted #56

Open franTarkenton opened 1 year ago

franTarkenton commented 1 year ago

Currently script will collect messages, then when all the messages are collected an event is emitted for the idem_key (datestring). Getting a 200 status code response from the emitted event, then the database will delete all the records for the idem_key.

After this happens more events will continue to stream into the listener. These events get logged, and subsequent queries to the api will tell us that all the files for the current idem_key are missing.

Need another table to track the idem_keys that have been emitted. If the idem_key is in the emitted table then any incomming should not get cached in the database.

Queries to the missing messages api end point, should first check to see if the idem_key exists in the database table for already emitted events. If this is the case the query should just return an empty set as no data is missing as the event has already been emitted.

Add end point that reports on the already emitted events.

Emitted events table should track the following columns: