search

awslabs / autonomous-driving-data-framework

ADDF is a collection of modules, deployed using the SeedFarmer orchestration tool. ADDF modules enable users to quickly bootstrap environments for the process and analysis of autonomous driving data.
Apache License 2.0
109 stars 45 forks source link

[BUG] demo-core-eks deployment failure #575

Closed serge-dolgavin-dxc closed 2 months ago

serge-dolgavin-dxc commented 2 months ago

Describe the bug I tried to deploy ADDF (https://github.com/awslabs/autonomous-driving-data-framework/blob/main/docs/deployment_guide.md) and got the following error: $ seedfarmer apply ./manifests/demo/deployment.yaml --debug ... Build Info Debug Data demo-core-eks status: ERROR ...

Modules list: Modules deployed that are up to date (will not be changed): demo
┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ ┃ Account ┃ Region ┃ Deployment ┃ Group ┃ Module ┃ ┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ primary │ eu-west-1 │ demo │ optionals │ networking │ │ primary │ eu-west-1 │ demo │ optionals │ datalake-buckets │ │ primary │ eu-west-1 │ demo │ replication │ replication │ │ primary │ eu-west-1 │ demo │ core │ mwaa │ │ primary │ eu-west-1 │ demo │ core │ metadata-storage │ │ primary │ eu-west-1 │ demo │ core │ opensearch │ │ primary │ eu-west-1 │ demo │ core │ batch-compute │ │ primary │ eu-west-1 │ demo │ core │ fsx-lustre │ │ primary │ eu-west-1 │ demo │ core │ efs │ └─────────┴───────────┴────────────┴─────────────┴──────────────────┘ Modules scheduled to be deployed (created or updated): demo
┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Account ┃ Region ┃ Deployment ┃ Group ┃ Module ┃ Path ┃ ┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ primary │ eu-west-1 │ demo │ core │ eks │ git::https://github.com/awslabs/idf-modules.git//modules/compute/eks?ref=release/1.10.0 │ │ primary │ eu-west-1 │ demo │ core │ neptune │ git::https://github.com/awslabs/idf-modules.git//modules/database/neptune?ref=release/1.1.0&depth=1 │ │ primary │ eu-west-1 │ demo │ prereqs │ tf-prereqs │ modules/examples/example-tf-prereqs/ │ │ primary │ eu-west-1 │ demo │ examples │ example-dags │ modules/examples/example-dags/ │ │ primary │ eu-west-1 │ demo │ examples │ docker-build │ git::https://github.com/awslabs/idf-modules.git//modules/storage/ecr?ref=release/1.3.0&depth=1 │ │ primary │ eu-west-1 │ demo │ examples │ tf-pattern │ modules/examples/example-tf/ │ │ primary │ eu-west-1 │ demo │ rosbag │ rosbag-webviz │ modules/demo-only/rosbag-webviz │ │ primary │ eu-west-1 │ demo │ simulations │ k8s-managed-simulations │ modules/simulations/k8s-managed │ │ primary │ eu-west-1 │ demo │ simulations │ batch-managed │ modules/simulations/batch-managed │ │ primary │ eu-west-1 │ demo │ integration │ eks-os │ modules/integration/eks-to-opensearch/ │ │ primary │ eu-west-1 │ demo │ integration │ opensearch-proxy │ modules/demo-only/opensearch-proxy/ │ │ primary │ eu-west-1 │ demo │ integration │ rosbag-ddb-to-os │ modules/integration/ddb-to-opensearch/ │ │ primary │ eu-west-1 │ demo │ integration │ emrlogs-to-os │ modules/integration/emr-to-opensearch/ │ └─────────┴───────────┴────────────┴─────────────┴─────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┘

To Reproduce To reproduce the behavior use the following documentation: https://github.com/awslabs/autonomous-driving-data-framework/blob/main/docs/deployment_guide.md

  1. git clone --origin upstream --branch main https://github.com/awslabs/autonomous-driving-data-framework

  2. cd autonomous-driving-data-framework

  3. python3 -m venv .env && source .env/bin/activate

  4. pip install -r ./requirements.txt

  5. pip install -r ./requirements-dev.txt

  6. export AWS_DEFAULT_REGION="<>" export AWS_REGION="<>" export AWS_ACCESS_KEY_ID="<>" export AWS_SECRET_ACCESS_KEY="<>" export AWS_SESSION_TOKEN="<>"

  7. cdk bootstrap aws://...

  8. seedfarmer bootstrap toolchain --project addf --trusted-principal ...

  9. cp -R manifests/example-dev manifests/demo

  10. sed -i "s/example-dev/demo/g" manifests/demo/deployment.yaml

  11. Update manifests/demo/deployment.yaml

  12. source scripts/setup-secrets-example.sh

  13. ./scripts/setup-secrets-dockerhub.sh

  14. seedfarmer apply ./manifests/demo/deployment.yaml

Expected behavior demo-core-eks module deployed without issues.

Screenshots na

Additional context

Codeseeder-addf build log:

647 | Traceback (most recent call last): 648 | File "/codebuild/output/src2902401516/src/bundle/module/app.py", line 57, in 649 | stack = Eks( 650 | ^^^^ 651 | File "/root/.venv/lib/python3.11/site-packages/jsii/_runtime.py", line 118, in call 652 | inst = super(JSIIMeta, cast(JSIIMeta, cls)).call(*args, **kwargs) 653 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 654 | File "/codebuild/output/src2902401516/src/bundle/module/stack.py", line 149, in init 655 | secrets_key=secrets_key, 656 | ^^^^^^^^^^^ 657 | UnboundLocalError: cannot access local variable 'secrets_key' where it is not associated with a value 658 |   659 | Subprocess exited with error 1

malachi-constant commented 2 months ago

Could be an issue with the EKS module, but first things first, we need to update the demo manifests to use the latest modules from IDF. I have opened a PR and am testing that now. Will keep this issue updated. Thanks!

malachi-constant commented 2 months ago

PR opened to fix bug in EKS module.

malachi-constant commented 2 months ago

Ok think the EKS module should be fixed if you want to try the below.

name: eks
path: git::https://github.com/awslabs/idf-modules.git//modules/compute/eks?ref=main
....

We'll continue cleanup here with the demo as well.

malachi-constant commented 2 months ago

Please reopen if you encounter the same or additional issues.

serge-dolgavin-dxc commented 2 months ago

Thank you!

eks works well with

name: eks path: git::https://github.com/awslabs/idf-modules.git//modules/compute/eks?ref=release/1.11.0 ...

malachi-constant commented 2 months ago

Perfect @serge-dolgavin-dxc thanks for getting back to me, everything in the ADDF repo should be updated now. Just doing full testing on all our manifest to confirm. But if you encounter anything else let me know!

serge-dolgavin-dxc commented 2 months ago

Hi @malachi-constant,

I hope you have a good weekend.

It might be a related issue with IDF-ADDF updates.

I failed with 

name: opensearch
path: git::https://github.com/awslabs/idf-modules.git//modules/storage/opensearch?ref=release/1.11.0&depth=1

but 

name: opensearch
path: git::https://github.com/awslabs/idf-modules.git//modules/storage/opensearch?ref=release/1.10.0&depth=1

works well for me.