moelasmar
commented
2 years ago Thanks @traceon for raising this error. Could you please share the exact steps to reproduce this issue.
Closed traceon closed 1 year ago
moelasmar
commented
2 years ago Thanks @traceon for raising this error. Could you please share the exact steps to reproduce this issue.
traceon
commented
2 years ago @moelasmar here you go:
sam init
You can preselect a particular runtime or package type when using the `sam init` experience.
Call `sam init --help` to learn more.
Which template source would you like to use?
1 - AWS Quick Start Templates
2 - Custom Template Location
Choice: 1
Choose an AWS Quick Start application template
1 - Hello World Example
2 - Multi-step workflow
3 - Serverless API
4 - Scheduled task
5 - Standalone function
6 - Data processing
7 - Infrastructure event management
8 - Machine Learning
Template: 1
Use the most popular runtime and package type? (Python and zip) [y/N]: n
Which runtime would you like to use?
1 - dotnet6
2 - dotnet5.0
3 - dotnetcore3.1
4 - go1.x
5 - graalvm.java11 (provided.al2)
6 - graalvm.java17 (provided.al2)
7 - java11
8 - java8.al2
9 - java8
10 - nodejs16.x
11 - nodejs14.x
12 - nodejs12.x
13 - python3.9
14 - python3.8
15 - python3.7
16 - python3.6
17 - ruby2.7
18 - rust (provided.al2)
Runtime: 10
What package type would you like to use?
1 - Zip
2 - Image
Package type: 2
Based on your selections, the only dependency manager available is npm.
We will proceed copying the template using npm.
Would you like to enable X-Ray tracing on the function(s) in your application? [y/N]: y
X-Ray will incur an additional cost. View https://aws.amazon.com/xray/pricing/ for more details
Project name [sam-app]:
Cloning from https://github.com/aws/aws-sam-cli-app-templates (process may take a moment)
-----------------------
Generating application:
-----------------------
Name: sam-app
Base Image: amazon/nodejs16.x-base
Architectures: x86_64
Dependency Manager: npm
Output Directory: .
Next steps can be found in the README file at ./sam-app/README.md
Commands you can use next
=========================
[*] Create pipeline: cd sam-app && sam pipeline init --bootstrap
[*] Validate SAM template: sam validate
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watchgit init sam-app
Initialized empty Git repository in /private/tmp/sam-app/.git/cd sam-app && sam pipeline init --bootstrap
sam pipeline init generates a pipeline configuration file that your CI/CD system
can use to deploy serverless applications using AWS SAM.
We will guide you through the process to bootstrap resources for each stage,
then walk through the details necessary for creating the pipeline config file.
Please ensure you are in the root folder of your SAM application before you begin.
Select a pipeline template to get started:
1 - AWS Quick Start Pipeline Templates
2 - Custom Pipeline Template Location
Choice: 1
Cloning from https://github.com/aws/aws-sam-cli-pipeline-init-templates.git (process may take a moment)
Select CI/CD system
1 - Jenkins
2 - GitLab CI/CD
3 - GitHub Actions
4 - Bitbucket Pipelines
5 - AWS CodePipeline
Choice: 3
You are using the 2-stage pipeline template.
_________ _________
| | | |
| Stage 1 |->| Stage 2 |
|_________| |_________|
Checking for existing stages...
[!] None detected in this account.
Do you want to go through stage setup process now? If you choose no, you can still reference other bootstrapped resources. [y/N]: y
For each stage, we will ask for [1] stage definition, [2] account details, and [3]
reference application build resources in order to bootstrap these pipeline
resources.
We recommend using an individual AWS account profiles for each stage in your
pipeline. You can set these profiles up using aws configure or ~/.aws/credentials. See
[https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-set-up-credentials.html].
Stage 1 Setup
[1] Stage definition
Enter a configuration name for this stage. This will be referenced later when you use the sam pipeline init command:
Stage configuration name: stage1
[2] Account details
The following AWS credential sources are available to use.
To know more about configuration AWS credentials, visit the link below:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
1 - Environment variables (not available)
2 - default (named profile)
q - Quit and configure AWS credentials
Select a credential source to associate with this stage: 2
Associated account <aws-account-id> with configuration stage1.
Enter the region in which you want these resources to be created [us-east-1]:
Enter the pipeline IAM user ARN if you have previously created one, or we will create one for you []:
[3] Reference application build resources
Enter the pipeline execution role ARN if you have previously created one, or we will create one for you []:
Enter the CloudFormation execution role ARN if you have previously created one, or we will create one for you []:
Please enter the artifact bucket ARN for your Lambda function. If you do not have a bucket, we will create one for you []:
Does your application contain any IMAGE type Lambda functions? [y/N]: y
Please enter the ECR image repository ARN(s) for your Image type function(s).If you do not yet have a repository, we will create one for you []:
[4] Summary
Below is the summary of the answers:
1 - Account: <aws-account-id>
2 - Stage configuration name: stage1
3 - Region: us-east-1
4 - Pipeline user: [to be created]
5 - Pipeline execution role: [to be created]
6 - CloudFormation execution role: [to be created]
7 - Artifacts bucket: [to be created]
8 - ECR image repository: [to be created]
Press enter to confirm the values above, or select an item to edit the value:
This will create the following required resources for the 'stage1' configuration:
- Pipeline IAM user
- Pipeline execution role
- CloudFormation execution role
- Artifact bucket
- ECR image repository
Should we proceed with the creation? [y/N]: y
Creating the required resources...
Successfully created!
The following resources were created in your account:
- Pipeline IAM user
- Pipeline execution role
- CloudFormation execution role
- Artifact bucket
- ECR image repository
Pipeline IAM user credential:
AWS_ACCESS_KEY_ID: AAAAAAAAAAAAAAAAAAAA
AWS_SECRET_ACCESS_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
View the definition in .aws-sam/pipeline/pipelineconfig.toml,
run sam pipeline bootstrap to generate another set of resources, or proceed to
sam pipeline init to create your pipeline configuration file.
Before running sam pipeline init, we recommend first setting up AWS credentials
in your CI/CD account. Read more about how to do so with your provider in
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-generating-example-ci-cd-others.html.
Checking for existing stages...
Only 1 stage(s) were detected, fewer than what the template requires: 2.
Do you want to go through stage setup process now? If you choose no, you can still reference other bootstrapped resources. [y/N]: y
For each stage, we will ask for [1] stage definition, [2] account details, and [3]
reference application build resources in order to bootstrap these pipeline
resources.
We recommend using an individual AWS account profiles for each stage in your
pipeline. You can set these profiles up using aws configure or ~/.aws/credentials. See
[https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-set-up-credentials.html].
Stage 2 Setup
[1] Stage definition
Enter a configuration name for this stage. This will be referenced later when you use the sam pipeline init command:
Stage configuration name: stage2
[2] Account details
The following AWS credential sources are available to use.
To know more about configuration AWS credentials, visit the link below:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
1 - Environment variables (not available)
2 - default (named profile)
q - Quit and configure AWS credentials
Select a credential source to associate with this stage: 2
Associated account <aws-account-id> with configuration stage2.
Enter the region in which you want these resources to be created [us-east-1]:
Pipeline IAM user ARN: arn:aws:iam::<aws-account-id>:user/aws-sam-cli-managed-stage1-pipeline-r-PipelineUser-TMTY69WHSII0
[3] Reference application build resources
Enter the pipeline execution role ARN if you have previously created one, or we will create one for you []:
Enter the CloudFormation execution role ARN if you have previously created one, or we will create one for you []:
Please enter the artifact bucket ARN for your Lambda function. If you do not have a bucket, we will create one for you []:
Does your application contain any IMAGE type Lambda functions? [y/N]: y
Please enter the ECR image repository ARN(s) for your Image type function(s).If you do not yet have a repository, we will create one for you []:
[4] Summary
Below is the summary of the answers:
1 - Account: <aws-account-id>
2 - Stage configuration name: stage2
3 - Region: us-east-1
4 - Pipeline user ARN: arn:aws:iam::<aws-account-id>:user/aws-sam-cli-managed-stage1-pipeline-r-PipelineUser-TMTY69WHSII0
5 - Pipeline execution role: [to be created]
6 - CloudFormation execution role: [to be created]
7 - Artifacts bucket: [to be created]
8 - ECR image repository: [to be created]
Press enter to confirm the values above, or select an item to edit the value:
This will create the following required resources for the 'stage2' configuration:
- Pipeline execution role
- CloudFormation execution role
- Artifact bucket
- ECR image repository
Should we proceed with the creation? [y/N]: y
Creating the required resources...
Successfully created!
The following resources were created in your account:
- Pipeline execution role
- CloudFormation execution role
- Artifact bucket
- ECR image repository
View the definition in .aws-sam/pipeline/pipelineconfig.toml,
run sam pipeline bootstrap to generate another set of resources, or proceed to
sam pipeline init to create your pipeline configuration file.
Checking for existing stages...
This template configures a pipeline that deploys a serverless application to a testing and a production stage.
What is the GitHub secret name for pipeline user account access key ID? [AWS_ACCESS_KEY_ID]:
What is the GitHub Secret name for pipeline user account access key secret? [AWS_SECRET_ACCESS_KEY]:
What is the git branch used for production deployments? [main]: master
What is the template file path? [template.yaml]:
We use the stage configuration name to automatically retrieve the bootstrapped resources created when you ran `sam pipeline bootstrap`.
Here are the stage configuration names detected in .aws-sam/pipeline/pipelineconfig.toml:
1 - stage1
2 - stage2
Select an index or enter the stage 1's configuration name (as provided during the bootstrapping): 1
What is the sam application stack name for stage 1? [sam-app]: aws-sam-cli-managed-stage1-pipeline-resources
Stage 1 configured successfully, configuring stage 2.
Here are the stage configuration names detected in .aws-sam/pipeline/pipelineconfig.toml:
1 - stage1
2 - stage2
Select an index or enter the stage 2's configuration name (as provided during the bootstrapping): 2
What is the sam application stack name for stage 2? [sam-app]: aws-sam-cli-managed-stage2-pipeline-resources
Stage 2 configured successfully.
SUMMARY
We will generate a pipeline config file based on the following information:
What is the GitHub secret name for pipeline user account access key ID?: AWS_ACCESS_KEY_ID
What is the GitHub Secret name for pipeline user account access key secret?: AWS_SECRET_ACCESS_KEY
What is the git branch used for production deployments?: master
What is the template file path?: template.yaml
Select an index or enter the stage 1's configuration name (as provided during the bootstrapping): 1
What is the sam application stack name for stage 1?: aws-sam-cli-managed-stage1-pipeline-resources
What is the pipeline execution role ARN for stage 1?: arn:aws:iam::<aws-account-id>:role/aws-sam-cli-managed-stage1-p-PipelineExecutionRole-D658PJ0HKDWL
What is the CloudFormation execution role ARN for stage 1?: arn:aws:iam::<aws-account-id>:role/aws-sam-cli-managed-stage-CloudFormationExecutionR-CL6NW0P7SYZU
What is the S3 bucket name for artifacts for stage 1?: aws-sam-cli-managed-stage1-pipeli-artifactsbucket-13zxizl2xe60a
What is the ECR repository URI for stage 1?: <aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/aws-sam-cli-managed-stage1-pipeline-resources-imagerepository-rqgeabycsle8
What is the AWS region for stage 1?: us-east-1
Select an index or enter the stage 2's configuration name (as provided during the bootstrapping): 2
What is the sam application stack name for stage 2?: aws-sam-cli-managed-stage2-pipeline-resources
What is the pipeline execution role ARN for stage 2?: arn:aws:iam::<aws-account-id>:role/aws-sam-cli-managed-stage2-p-PipelineExecutionRole-BFW01F445EXH
What is the CloudFormation execution role ARN for stage 2?: arn:aws:iam::<aws-account-id>:role/aws-sam-cli-managed-stage-CloudFormationExecutionR-ZTFYO4US12WB
What is the S3 bucket name for artifacts for stage 2?: aws-sam-cli-managed-stage2-pipeli-artifactsbucket-ovsle77asffw
What is the ECR repository URI for stage 2?: <aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/aws-sam-cli-managed-stage2-pipeline-resources-imagerepository-axf9pdegav6w
What is the AWS region for stage 2?: us-east-1
Successfully created the pipeline configuration file(s):
- .github/workflows/pipeline.yamlVerify the user existence: aws iam get-user --user-name aws-sam-cli-managed-stage1-pipeline-r-PipelineUser-TMTY69WHSII0
{
"User": {
"Path": "/",
"UserName": "aws-sam-cli-managed-stage1-pipeline-r-PipelineUser-TMTY69WHSII0",
"UserId": "AAAAAAAAAAAAAAAAAAAAA",
"Arn": "arn:aws:iam::<aws-account-id>:user/aws-sam-cli-managed-stage1-pipeline-r-PipelineUser-TMTY69WHSII0",
"CreateDate": "2022-06-08T01:19:41+00:00",
"Tags": [
{
"Key": "ManagedStackSource",
"Value": "AwsSamCli"
}
]
}
}gh repo create sam-app --private
✓ Created repository <github-username>/sam-app on GitHubI set access keys for the user arn:aws:iam::<aws-account-id>:user/aws-sam-cli-managed-stage1-pipeline-r-PipelineUser-TMTY69WHSII0 as GitHub Actions secrets at https://github.com/<github-username>/sam-app/settings/secrets/actions under AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY accordingly.
git add --all
git commit -m "Initial commit"
[master (root-commit) 4099ab7] Initial commit
10 files changed, 755 insertions(+)
create mode 100644 .github/workflows/pipeline.yaml
create mode 100644 .gitignore
create mode 100644 README.md
create mode 100644 events/event.json
create mode 100644 hello-world/.npmignore
create mode 100644 hello-world/Dockerfile
create mode 100644 hello-world/app.js
create mode 100644 hello-world/package.json
create mode 100644 hello-world/tests/unit/test-handler.js
create mode 100644 template.yamlgit remote add origin git@github.com:<github-username>/sam-app.git
git branch -M master
git push -u origin master
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Delta compression using up to 8 threads
Compressing objects: 100% (12/12), done.
Writing objects: 100% (18/18), 9.50 KiB | 1.05 MiB/s, done.
Total 18 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:<github-username>/sam-app.git
* [new branch] master -> master
branch 'master' set up to track 'origin/master'.Now I go to https://github.com/<github-username>/sam-app/actions, select the running action for this commit, select the deploy-testing job, and wait until it fails at Deploy to testing account step with error:
...
DELETE_FAILED AWS::ECR::Repository ImageRepository Resource handler
returned message: "The
repository with name
'aws-sam-cli-managed-
stage1-pipeline-
resources-
imagerepository-
rqgeabycsle8' in
registry with id
'***' cannot
be deleted because it
still contains images
(Service: Ecr, Status
Code: 400, Request ID:
215b77e2-11c2-4163-
a9e3-498dc8c8cc36,
Extended Request ID:
null)" (RequestToken:
92bc380d-2541-e377-
2df4-cd24335200ce,
HandlerErrorCode: Gene
ralServiceException)
Describing stack events for aws-sam-cli-managed-stage1-pipeline-resources failed: An error occurred (InvalidClientTokenId) when calling the DescribeStacks operation: The security token included in the request is invalid.
-------------------------------------------------------------------------------------------------
Error: Failed to create/update the stack: aws-sam-cli-managed-stage1-pipeline-resources, Waiter StackUpdateComplete failed: An error occurred (InvalidClientTokenId): The security token included in the request is invalid.
Error: Process completed with exit code 1.Verify that the user doesn't exist anymore: aws iam get-user --user-name aws-sam-cli-managed-stage1-pipeline-r-PipelineUser-TMTY69WHSII0
An error occurred (NoSuchEntity) when calling the GetUser operation: The user with name aws-sam-cli-managed-stage1-pipeline-r-PipelineUser-TMTY69WHSII0 cannot be found.Moreover, now if I try to delete the stacks I get this annoying errors:
sam delete --stack-name aws-sam-cli-managed-stage1-pipeline-resources
Are you sure you want to delete the stack aws-sam-cli-managed-stage1-pipeline-resources in the region us-east-1 ? [y/N]: y
- Deleting ECR image helloworldfunction-25b593277c76-nodejs16.x-v1 in repository aws-sam-cli-managed-stage1-pipeline-resources-imagerepository-rqgeabycsle8
- Deleting Cloudformation stack aws-sam-cli-managed-stage1-pipeline-resources
Failed to delete stack : An error occurred (ValidationError) when calling the DeleteStack operation: Stack [arn:aws:cloudformation:us-east-1:<aws-account-id>:stack/aws-sam-cli-managed-stage1-pipeline-resources/04dbee60-e6c9-11ec-a3c2-1203f3b6522d] cannot be deleted while in status UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
Error: Failed to delete the stack: aws-sam-cli-managed-stage1-pipeline-resources, An error occurred (ValidationError) when calling the DeleteStack operation: Stack [arn:aws:cloudformation:us-east-1:<aws-account-id>:stack/aws-sam-cli-managed-stage1-pipeline-resources/04dbee60-e6c9-11ec-a3c2-1203f3b6522d] cannot be deleted while in status UPDATE_COMPLETE_CLEANUP_IN_PROGRESSsam delete --stack-name aws-sam-cli-managed-stage2-pipeline-resources
Are you sure you want to delete the stack aws-sam-cli-managed-stage2-pipeline-resources in the region us-east-1 ? [y/N]: y
ECR repository None may not be empty. Do you want to delete the repository and all the images in it ? [y/N]: y
- Deleting Cloudformation stack aws-sam-cli-managed-stage2-pipeline-resources
Error: Stack could not be deleted as it encountered DELETE_FAILED status: aws-sam-cli-managed-stage2-pipeline-resources, ex: Waiter StackDeleteComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "DELETE_FAILED" at least once
bdowney
commented
2 years ago ^ i am having this issue as well, almost identical setup process. stacks never disappear, nor do s3 buckets, but IAM roles for test + prod are (seemingly) randomly deleted
traceon
commented
2 years ago @moelasmar do you need anything else to reproduce this?
ethanphelps
commented
1 year ago This is happening to me too when I select the GitHub Actions CI/CD system. I used the sam pipeline init --bootstrap and was following this AWS SAM workshop. The dev deploy stage deletes the pipeline user that's needed in the prod deploy stage. It also deletes the pipeline execution role and the CloudFormation execution role, which is extremely annoying because I can't delete the pipeline CloudFormation stacks with sam delete or via the AWS console because I get the following error:
aws-sam-cli-managed-dev-p-CloudFormationExecutionR-1WYXMYU4J6U03 is invalid or cannot be assumed
I tried locating the logic that causes these IAM roles and users to be deleted but could not find anything.
karimdaghari
commented
1 year ago Can confirm that this is still an issue!
hnnasit
commented
1 year ago Hi, thanks @traceon for providing the detailed steps. I did a deep dive and noticed that in Step 3 for the prompt What is the sam application stack name for stage 1/2, you are using the same stack-names as the stacks created by sam pipeline for the resources of each stage. This overwrites the stage stack CFN template with the template for hello-world when the deploy job is run and deletes all the stage related resources including the Pipeline user. Could you try assigning stack-names for those prompts that are different from the stacks created by sam pipeline?
As for deleting this overwritten stack, sam delete fails because the CFN execution role is deleted. As a workaround, you can also follow this doc for deleting this stack.
Please let us know if you have more questions or concerns.
hnnasit
commented
1 year ago Closing this issue as there are no further questions or concerns. Please feel free to open a new issue if you have any other questions or issues.
github-actions[bot]
commented
1 year ago Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
In one of the steps in the generated (using
sam pipeline init --bootstrap) GitHub Actions pipeline, whensam deployis executed, I am seeing the error below.Looks like the changeset includes deletion of the IAM user. It this correct, or am I missing something?