Implement queuing service for streaming data into Nucleus

NASA-PDS / nucleus

Nucleus is a software platform used to create workflows for the Planetary Data (PDS).

https://nasa-pds.github.io/nucleus

Apache License 2.0

0 stars 0 forks source link

Implement queuing service for streaming data into Nucleus #63

Closed ramesh-maddegoda closed 10 months ago

ramesh-maddegoda commented 10 months ago

Implement queuing service for streaming data into Nucleus based on the following design related inputs:

https://github.com/NASA-PDS/nucleus/issues/60
https://github.com/NASA-PDS/nucleus/issues/60#issuecomment-1683199443

ramesh-maddegoda commented 10 months ago

Created lambda to check product completion status. Created terraform to setup DynamoDB tables and SQS queue. The related pull request is available at https://github.com/NASA-PDS/nucleus/pull/64

It was identified that race conditions can happen while multiple S3 event notifications invoke lambda to update DynamoDB tables. To avoid that, a locking mechanism will be implemented.