aws-samples / aws-genai-llm-chatbot

A modular and comprehensive solution to deploy a Multi-LLM and Multi-RAG powered chatbot (Amazon Bedrock, Anthropic, HuggingFace, OpenAI, Meta, AI21, Cohere, Mistral) using AWS CDK on AWS
https://aws-samples.github.io/aws-genai-llm-chatbot/
MIT No Attribution
980 stars 287 forks source link

'npx cdk destroy' failed because ENI was in use #350

Open alexeyshishkin01 opened 5 months ago

alexeyshishkin01 commented 5 months ago

Hello, everyone !

My command 'npx cdk destroy' failed because ENI was in use during the deletion.

So I had to manually delete ENI, then subnet, then VPC, then Cloud Formation stack.

Could you pls. have a look at the code to fix the issue ?

best regards Alexey Shishkin

scottschreckengaust commented 5 months ago

Does the force option help?

npx cdk destroy --all --force

alexeyshishkin01 commented 5 months ago

I don't know. To test this option I have to launch the solution again and try to destroy it.

Is it possible to update the documentation https://aws-samples.github.io/aws-genai-llm-chatbot/guide/deploy.html#clean-up with this option ?

QuinnGT commented 5 months ago

Just did a full deployment with each model and RAG. When I went to destroy, two resources were marked as deleted but in fact not actually delete.

  1. Sagemaker Endpoint (IDEFICS)
  2. VPC Endpoint (shared vpc)

No solution in cdk yet, just stating I'm seeing the same thing. I had to go and find through through resources in cloudformation stack, delete them manually, then go back to cloudformation to complete the deletion.

alexeyshishkin01 commented 3 months ago

Does the force option help?

npx cdk destroy --all --force

It looks like it does not help as well:

3:36:49 PM | DELETE_FAILED | AWS::EC2::Subnet | SharedVPCprivateSubnet1Subnet5A4C2616 Resource handler returned message: "The subnet 'subnet-07241ef8cc7f8a234' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400, Request ID: c1672d1b-1cd3-4d86-bb4e-09c53864da90)" (RequestToken: c0ecb171-cf72-ff24-eb38-317bdd2dddfd, HandlerErrorCode: InvalidRequest)

❌ genaiGenAIChatBotStack: destroy failed Error: The stack named genaiGenAIChatBotStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [SharedVPCprivateSubnet1Subnet5A4C2616]. ): Resource handler returned message: "The subnet 'subnet-07241ef8cc7f8a234' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400, Request ID: c1672d1b-1cd3-4d86-bb4e-09c53864da90)" (RequestToken: c0ecb171-cf72-ff24-eb38-317bdd2dddfd, HandlerErrorCode: InvalidRequest) at destroyStack (/home/ubuntu/environment/aws-genai-llm-chatbot/node_modules/aws-cdk/lib/index.js:428:2204) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async CdkToolkit.destroy (/home/ubuntu/environment/aws-genai-llm-chatbot/node_modules/aws-cdk/lib/index.js:431:204632) at async exec4 (/home/ubuntu/environment/aws-genai-llm-chatbot/node_modules/aws-cdk/lib/index.js:486:54250)

The stack named genaiGenAIChatBotStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [SharedVPCprivateSubnet1Subnet5A4C2616]. ): Resource handler returned message: "The subnet 'subnet-07241ef8cc7f8a234' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400, Request ID: c1672d1b-1cd3-4d86-bb4e-09c53864da90)" (RequestToken: c0ecb171-cf72-ff24-eb38-317bdd2dddfd, HandlerErrorCode: InvalidRequest)

The subnet has an ENI which is not deleted. The ENI Description is 'SageMaker managed ENI for endpoint amazon-FalconLite instance i-03b16c72c5b18266d'

alexeyshishkin01 commented 3 months ago

these commands can be helpful for the troubleshooting:

aws ec2 delete-network-interface --network-interface-id $(aws ec2 describe-network-interfaces --filters "Name=description,Values=SageMaker managed ENI for endpoint amazon-FalconLite" --query 'NetworkInterfaces[].NetworkInterfaceId' --output text)

aws ec2 delete-subnet --subnet-id $(aws ec2 describe-network-interfaces --filters "Name=description,Values=SageMaker managed ENI for endpoint amazon-FalconLite" --query 'NetworkInterfaces[].SubnetId' --output text)

alexeyshishkin01 commented 3 months ago

as a workaround one can run the following in the script:

npx cdk destroy --all --force aws ec2 delete-network-interface --network-interface-id $(aws ec2 describe-network-interfaces --filters "Name=description,Values=SageMaker managed ENI for endpoint amazon-FalconLite" --query 'NetworkInterfaces[].NetworkInterfaceId' --output text) aws ec2 delete-subnet --subnet-id $(aws ec2 describe-network-interfaces --filters "Name=description,Values=SageMaker managed ENI for endpoint amazon-FalconLite" --query 'NetworkInterfaces[].SubnetId' --output text) npx cdk destroy --all --force

alexeyshishkin01 commented 2 months ago

it looks like it's an old issue in AWS - https://stackoverflow.com/questions/52416454/sagemaker-model-cloudformation-stack-deletion

github-actions[bot] commented 20 hours ago

This issue is stale because it has been open for 60 days with no activity.