search

unity-sds / unity-data-services

Apache License 2.0
0 stars 3 forks source link

feat: Daac sender + receiver logic #402

Closed wphyojpl closed 1 month ago

wphyojpl commented 2 months ago

Closes #380 Closes #378 Closes #377

wphyojpl commented 2 months ago

archiving_types is not really working correctly at this moment.. Need more tests.

wphyojpl commented 2 months ago

archiving_types is not really working correctly at this moment.. This is fixed.

But Archive lambda is being triggered before ES indexing lambda is executed. As a result, it's always missing / no need to archive.

wphyojpl commented 2 months ago

DAAC SNS is published twice. Need to check.

wphyojpl commented 2 months ago

cnm response only has collection name w/o version. Need to use only collection name.

wphyojpl commented 1 month ago

Testing failure:

            "archive_status": "cnm_r_failed",
            "archive_error_message": "[{\"uri\": \"https://uds-distribution-placeholder/uds-sbx-cumulus-staging/URN:NASA:UNITY:UDS_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2407291400/URN:NASA:UNITY:UDS_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2407291400:abcd.1234.efgh.test_file05/abcd.1234.efgh.test_file05.data.stac.json\", \"error\": \"unknown error: An error occurred (403) when calling the HeadObject operation: Forbidden\"}, {\"uri\": \"https://uds-distribution-placeholder/uds-sbx-cumulus-staging/URN:NASA:UNITY:UDS_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2407291400/URN:NASA:UNITY:UDS_LOCAL_ARCHIVE_TEST:DEV:UDS_UNIT_COLLECTION___2407291400:abcd.1234.efgh.test_file05/abcd.1234.efgh.test_file05.cmr.xml\", \"error\": \"unknown error: An error occurred (403) when calling the HeadObject operation: Forbidden\"}]",
            "archive_error_code": "VALIDATION_ERROR"

Who will take care of S3 bucket's IAM role updates for the real DAAC to pull data?

wphyojpl commented 1 month ago

Testing with Unity-Venue-Dev customer created bucket. Looks like need to add arn:aws:s3:::*unity* to lambda-processing-role.

Probably need to do similar thing on mock-daac lambda as well to that custom bucket

wphyojpl commented 1 month ago

Should we store the success / failure history of DAAC? Currently, when a status is updated, the old failure messages will be deleted. Should they be stored?

wphyojpl commented 1 month ago

Keep Granules stac compliant. Move archival to "properties"