search

NASA-PDS / nucleus

Nucleus is a software platform used to create workflows for the Planetary Data (PDS).
https://nasa-pds.github.io/nucleus
Apache License 2.0
0 stars 0 forks source link

Integrate the latest version 4.0.1 of harvest in Nucleus workflow #121

Open ramesh-maddegoda opened 3 weeks ago

ramesh-maddegoda commented 3 weeks ago

💡 Description

Recently there were lots if changes happened in harvest related with the Cognito authentication and accessing Open Search. This task is to integrate the latest version 4.0.1 of harvest in Nucleus workflow

⚔️ Parent Epic / Related Tickets

No response

ramesh-maddegoda commented 3 weeks ago

Expecting new changes to be implemented in harvest, when using it in a VPC inside MCP Dev.

tloubrieu-jpl commented 3 weeks ago

blocked by ticket https://github.com/NASA-PDS/registry-common/issues/83

jordanpadams commented 1 week ago

Standup status: Testing this out now .

tloubrieu-jpl commented 12 hours ago

I have fixed the registry-loader docker image that @ramesh-maddegoda was working with to use a version of harvest which supports to get the AWS credentials from the AWS assumed role. However I have not been able to make that work on EC2 following manual https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html. An alternative could be to deploy on ECS where this method has already been successfully tested for the registry API.

Another option is to wait for the SA's to solve the ticket DSIO-6578 in which we ask to be able to access the API gateway from EC2 or ECS.

We should keep working on both options in parallel.

tloubrieu-jpl commented 10 hours ago

@ramesh-maddegoda will work on the ECS deployment, using the URL http://169.254.170.2/AWS_CONTAINER_CREDENTIALS_RELATIVE_URI in the connection configuration, as documented in https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html