Feature/issue 212 - Update track ingest table with granule status

[nikki-t]

Github Issue: #212

Description

Update track ingest table with granule status after determining what is in CMR but not in Hydrocron and locating all granules in the track ingest table with a status of "to_ingest".

Overview of work done

• Added update_track_ingest method to update track ingest table with granules that have been ingested and need to be ingested. Uses load_data function.

Overview of verification done

• Added new test to test track ingest update for a granule that has a status of "to_ingest".
• Added new test to test track status update for a granule that has been ingested and no longer has a status of "to_ingest".

New and existing unit tests pass.

Overview of integration done

Deployed to SIT environment and executed tests on CMR granules with a revision_date range of : 2024-09-05 23:00:00+00:00 to 2024-09-05 23:59:59+00:00 with 1 granule that has a "to_ingest" status and 2 granules that are not in the Hydrocron table.

[INFO] 2024-09-09T19:27:41.765Z Collection shortname: SWOT_L2_HR_RiverSP_reach_2.0
[INFO] 2024-09-09T19:27:41.765Z Hydrocron table: hydrocron-swot-reach-table
[INFO] 2024-09-09T19:27:41.765Z Hydrocron track ingest table: hydrocron-swot-reach-track-ingest-table
[INFO] 2024-09-09T19:27:41.765Z Temporal indicator for revision dates: True
[INFO] 2024-09-09T19:27:41.765Z Temporal start date: 2024-09-05 23:00:00+00:00
[INFO] 2024-09-09T19:27:41.765Z Temporal end date: 2024-09-05 23:59:59+00:00
[INFO] 2024-09-09T19:27:41.765Z Querying CMR temporal range: 2024-09-05 23:00:00+00:00 to 2024-09-05 23:59:59+00:00.
[INFO] 2024-09-09T19:27:42.302Z Located 2 granules in CMR.
[INFO] 2024-09-09T19:27:42.347Z Located 2 granules NOT in Hydrocron.
[INFO] 2024-09-09T19:27:42.360Z Located 1 granules with 'to_ingest' status.
[INFO] 2024-09-09T19:27:42.406Z Located 2 granules that require ingestion.
[INFO] 2024-09-09T19:27:42.406Z Located 1 granules that are already ingested.
[INFO] 2024-09-09T19:27:42.445Z Set up dynamo table connection
[INFO] 2024-09-09T19:27:42.488Z Adding track ingest reaches items to table individually
[INFO] 2024-09-09T19:27:42.488Z Item granuleUR: SWOT_L2_HR_RiverSP_Reach_010_177_NA_20240131T074748_20240131T074759_PIC0_01.zip
[INFO] 2024-09-09T19:27:42.495Z Item granuleUR: SWOT_L2_HR_RiverSP_Reach_020_457_NA_20240905T233134_20240905T233135_PIC0_01.zip
[INFO] 2024-09-09T19:27:42.506Z Item granuleUR: SWOT_L2_HR_RiverSP_Reach_020_457_AR_20240905T233224_20240905T233235_PIC0_01.zip
[INFO] 2024-09-09T19:27:42.520Z Updated hydrocron-swot-reach-track-ingest-table with 3 items.
[INFO] 2024-09-09T19:27:42.521Z To Ingest: [{'granuleUR': 'SWOT_L2_HR_RiverSP_Reach_020_457_NA_20240905T233134_20240905T233135_PIC0_01.zip', 'revision_date': '2024-09-09T02:57:10.684Z', 'checksum': '6ce27e868bd90055252de186f554759f', 'expected_feature_count': -1, 'actual_feature_count': 0, 'status': 'to_ingest'}, {'granuleUR': 'SWOT_L2_HR_RiverSP_Reach_020_457_AR_20240905T233224_20240905T233235_PIC0_01.zip', 'revision_date': '2024-09-09T02:57:13.806Z', 'checksum': 'd9fdbea55becb22dacea13c4055aa5ad', 'expected_feature_count': -1, 'actual_feature_count': 0, 'status': 'to_ingest'}]
[INFO] 2024-09-09T19:27:42.522Z Ingested: [{'granuleUR': 'SWOT_L2_HR_RiverSP_Reach_010_177_NA_20240131T074748_20240131T074759_PIC0_01.zip', 'revision_date': '2024-07-30T03:13:39.720Z', 'checksum': '936a4fcb3e28eb7e246eba532d36a434', 'expected_feature_count': 438, 'actual_feature_count': 438}]

PR checklist:

• [X] Linted
• [X] Updated unit tests
• [X] Updated changelog
• [X] Integration testing

See Pull Request Review Checklist for pointers on reviewing this pull request

[nikki-t]

Please see #238 for the latest testing in SIT.