bcgov / nr-rfc-grib-copy

Demo of using github actions to copy / process raw data
Apache License 2.0
2 stars 2 forks source link

Get idempotency key from message #20

Closed franTarkenton closed 1 year ago

franTarkenton commented 1 year ago

Purpose of the idempotency key is to enable the logic to continue working correctly if it happens to receive multiple messages that describe the same event.

Currently we are using datestamps that get calculated by the code as the idempotency keys. This should be modified to come from the actual messages emitted by the message queue.

the body of messages emitted has the following format:

20230404225731.353 https://hpfx.collab.science.gc.ca /20230404/WXO-DD/bulletins/alphanumeric/20230404/SR/KWAL/22/SRMN70_KWAL_042256___4633

The first element in the message is a datetime stamp that should be parsed to extract the Ymd and used as the idempotency key for the cmc data.

This ticket will implement the following:

franTarkenton commented 1 year ago

This was completed in other ticket! this line: https://github.com/bcgov/nr-rfc-grib-copy/blob/6f740667149fc5d97ba7ca2d0534c2779581c2b2/backend/messaging/cmc_grib_callbacks.py#L57 shows where the idem_key is retrieved from the message. There is more to this, however that line demonstrates that the key is comming from the message