dlpzx
commented
1 year ago Hi @idpdevops, thanks for opening an issue. Even if it is "outside the intended use" we will try to help you on fixing your deployment. From the logs I can distinguish 2 different issues:
- Issue 1 --> ECS glue sync task failure
- Issue 2 --> Migration CodeBuild stage failure
Issue 1 --> ECS glue sync task failure
Root cause of the issue: access denied for ecs-task-role to AssumeRole arn:aws:iam::ZZZZZZZZZZZZ:role/dataallPivotRole-cdk
This type of error (Access Denied for AssumeRole) has 2 possible causes.
- the
ecs-task-rolelacks AssumeRole permissions - the
arn:aws:iam::ZZZZZZZZZZZZ:role/dataallPivotRole-cdklacks permissions for theecs-task-rolein its trust policy
In v1.6 we focused on security hardening features, including the hardening of the trust policies on the pivot role, so that is the first thing that I would verify. We moved the external ID used in the trusted accounts to SSM, so its value has been updated.
In case you don't know what the external ID is, here is some documentation of why cross-account role assumption should use external IDs
- Has the environment for account
ZZZZZZZZZZZZbeen updated after the upgrade? - If the environment was not updated, then the trust policy contains an external ID that is outdated. The ECS task is trying to assume the role with a new external ID, but the trust policy only allows assumptions verified with an old external ID.
- To verify this hypothesis, you need to compare the external ID defined in the trust policy in the environment account with the SSM parameter in the central account (look for a parameter called
externalIdor something similar) and see if they match
Issue 2 --> Migration CodeBuild stage failure
There are better logs to debug this issue. Migrations are the way data.all has to update the RDS tables schema when new features are introduced. We want to include the update of RDS as part of our CICD pipeline (tooling account), but our RDS database is deployed in the central deployment account. To be able to modify the RDS database we deploy a CodeBuild project in the central deployment account, something like prefix-env-dbmigration, this is the "real migration codebuild stage" where we run the alembic commands. In the tooling account the migration stage (the "false migration stage" is just calling this real CodeBuild stage in the other account. Your logs are the fetching of the status, but as you probably noticed, they do not provide much info about the actual error.
What we need is to go to the central account > Codebuild> Projects > search for prefix-env-dbmigration and check the "real migration" logs
I hope this long test was helpful, please reach out with any new findings, logs or questions that might arise :)
idpdevops
Upgrade from V1.5.6 to V1.6.2 fails if using baseline_codebuild_role.without_policy_updates
This is technically not a bug as the as the data.all code was used outside it's intended purpose.
The fix https://github.com/awslabs/aws-dataall/pull/774 was implemented in the main branch so didn't help with our upgrade from V1.5.6 to V1.6.2. So I merged the pipeline.py changes into the V1.6.2 code (code as per "How to reproduce") and ran the pipeline. This worked well initially and it made it past the quality gate stage, the ecr-stage and also the dev-backend-stage. However, the DB migration stage failed (see logs in "Additional context").
The command that failed (error 255 hints at a permissions issue) is
aws codeartifact login --tool pip --domain -domain-master --domain-owner YYYYYYYYYYYY --repository -pypi-store
Interestingly, this command executed perfectly fine in the QualityGate ValidateDBMigrations stage (see logs).
I tried to execute this manually, assuming the relevant role that is used in the pipeline and that also worked fine!?
Since the pipeline ran, I have also received tons of emails with data.all alarms for various accounts relating to the ecr-stage (I think), although they stopped after about 2 days.
You are receiving this email because your DATAALL platdev environment in the eu-west-2 region has entered the ALARM state, because it failed to synchronize Dataset AAAAAAAAAAAA-risk-and-control tables from AWS Glue to the Search Catalog.
Alarm Details:
Glue Database: lbx_member_db
The emails stopped after 2 days, data.all may have given up on whatever it was trying to do.
Here are the Cloudtrail logs:
{ "eventVersion": "1.08", "userIdentity": { "type": "AssumedRole", "principalId": "*****:149dfc83a8ff464abfd0ffd63d62deaf", "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/*-platdev-ecs-tasks-role/149dfc83a8ff464abfd0ffd63d62deaf", "accountId": "XXXXXXXXXXXX", "accessKeyId": "****", "sessionContext": { "sessionIssuer": { "type": "Role", "principalId": "****", "arn": "arn:aws:iam::XXXXXXXXXXXX:role/-platdev-ecs-tasks-role", "accountId": "XXXXXXXXXXXX", "userName": "-platdev-ecs-tasks-role" }, "webIdFederationData": {}, "attributes": { "creationDate": "2023-10-13T14:09:31Z", "mfaAuthenticated": "false" } } }, "eventTime": "2023-10-13T14:10:01Z", "eventSource": "sts.amazonaws.com", "eventName": "AssumeRole", "awsRegion": "eu-west-2", "sourceIPAddress": "10.0.32.173", "userAgent": "Boto3/1.24.85 Python/3.8.16 Linux/5.10.192-183.736.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.27.85 data.all/0.5.0", "errorCode": "AccessDenied", "errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/***-platdev-ecs-tasks-role/149dfc83a8ff464abfd0ffd63d62deaf is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::WWWWWWWWWWWW:role/dataallPivotRole-cdk", "requestParameters": null, "responseElements": null, … } }
So some aspects are a bit of a mystery, maybe you can figure out what exactly has gone wrong.
DB Migration log:
… [Container] 2023/10/16 14:52:25 Entering phase BUILD [Container] 2023/10/16 14:52:25 Running command mkdir ~/.aws/ && touch ~/.aws/config
[Container] 2023/10/16 14:52:25 Running command echo "[profile buildprofile]" > ~/.aws/config
[Container] 2023/10/16 14:52:25 Running command echo "role_arn = arn:aws:iam::XXXXXXXXXXXX:role/***-platdev-cb-dbmigration-role" >> ~/.aws/config
[Container] 2023/10/16 14:52:25 Running command echo "credential_source = EcsContainer" >> ~/.aws/config
[Container] 2023/10/16 14:52:25 Running command aws sts get-caller-identity --profile buildprofile { "UserId": "****:botocore-session-1697467958", "Account": "XXXXXXXXXXXX", "Arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/***-platdev-cb-dbmigration-role/botocore-session-1697467958" }
[Container] 2023/10/16 14:52:39 Running command aws codebuild start-build --project-name ***-platdev-dbmigration --profile buildprofile --region eu-west-2 > codebuild-id.json
[Container] 2023/10/16 14:52:39 Running command aws codebuild batch-get-builds --ids $(jq -r .build.id codebuild-id.json) --profile buildprofile --region eu-west-2 > codebuild-output.json
[Container] 2023/10/16 14:52:40 Running command while [ "$(jq -r .builds[0].buildStatus codebuild-output.json)" != "SUCCEEDED" ] && [ "$(jq -r .builds[0].buildStatus codebuild-output.json)" != "FAILED" ]; do echo "running migration"; aws codebuild batch-get-builds --ids $(jq -r .build.id codebuild-id.json) --profile buildprofile --region eu-west-2 > codebuild-output.json; echo "$(jq -r .builds[0].buildStatus codebuild-output.json)"; sleep 5; done running migration IN_PROGRESS running migration …
IN_PROGRESS
running migration
FAILED
[Container] 2023/10/16 15:08:44 Running command if [ "$(jq -r .builds[0].buildStatus codebuild-output.json)" = "FAILED" ]; then echo "Failed"; cat codebuild-output.json; exit -1; fi Failed { "builds": [ { "id": "-platdev-dbmigration:556525ab-375d-481c-8eab-20358cfb3ec8", "arn": "arn:aws:codebuild:eu-west-2:XXXXXXXXXXXX:build/-platdev-dbmigration:556525ab-375d-481c-8eab-20358cfb3ec8", "buildNumber": 12, "startTime": 1697467959.665, "endTime": 1697468915.942, "currentPhase": "COMPLETED", "buildStatus": "FAILED", "projectName": "-platdev-dbmigration", "phases": [ … { "phaseType": "BUILD", "phaseStatus": "FAILED", "startTime": 1697467990.638, "endTime": 1697468915.594, "durationInSeconds": 924, "contexts": [ { "statusCode": "COMMAND_EXECUTION_ERROR", "message": "Error while executing command: aws codeartifact login --tool pip --domain -domain-master --domain-owner YYYYYYYYYYYY --repository -pypi-store. Reason: exit status 255" } ] }, { "phaseType": "POST_BUILD", "phaseStatus": "SUCCEEDED", "startTime": 1697468915.594, "endTime": 1697468915.63, "durationInSeconds": 0, "contexts": [ { "statusCode": "", "message": "" } ] }, { "phaseType": "UPLOAD_ARTIFACTS", "phaseStatus": "SUCCEEDED", "startTime": 1697468915.63, "endTime": 1697468915.708, "durationInSeconds": 0, "contexts": [ { "statusCode": "", "message": "" } ] }, { "phaseType": "FINALIZING", "phaseStatus": "SUCCEEDED", "startTime": 1697468915.708, "endTime": 1697468915.942, "durationInSeconds": 0, "contexts": [ { "statusCode": "", "message": "RequestError: send request failed\ncaused by: Post \"https://logs.eu-west-2.amazonaws.com/\": dial tcp 10.82.2.164:443: i/o timeout" } ] }, { "phaseType": "COMPLETED", "startTime": 1697468915.942 } ], "source": { "type": "NO_SOURCE", "buildspec": "{\n \"version\": \"0.2\",\n \"phases\": {\n \"build\": {\n \"commands\": [\n \"aws s3api get-object --bucket -master-code-YYYYYYYYYYYY-eu-west-2 --key source_build.zip source_build.zip\",\n \"unzip source_build.zip\",\n \"python -m venv env\",\n \". env/bin/activate\",\n \"aws codeartifact login --tool pip --domain -domain-master --domain-owner YYYYYYYYYYYY --repository -pypi-store\",\n \"pip install -r backend/requirements.txt\",\n \"pip install alembic\",\n \"export PYTHONPATH=backend\",\n \"export envname=platdev\",\n \"alembic -c backend/alembic.ini upgrade head\"\n ]\n }\n }\n}", "insecureSsl": false }, "secondarySources": [], "secondarySourceVersions": [], "artifacts": { "location": "" }, "cache": { "type": "NO_CACHE" }, "environment": { "type": "LINUX_CONTAINER", "image": "aws/codebuild/amazonlinux2-x86_64-standard:3.0", "computeType": "BUILD_GENERAL1_SMALL", "environmentVariables": [], "privilegedMode": false, "imagePullCredentialsType": "CODEBUILD" }, "serviceRole": "arn:aws:iam::XXXXXXXXXXXX:role/-platdev-cb-dbmigration-role", "logs": { "groupName": "/aws/codebuild/-platdev-dbmigration", "streamName": "556525ab-375d-481c-8eab-20358cfb3ec8", "deepLink": "https://console.aws.amazon.com/cloudwatch/home?region=eu-west-2#logsV2:log-groups/log-group/$252Faws$252Fcodebuild$252F***********-platdev-dbmigration/log-events/556525ab-375d-481c-8eab-20358cfb3ec8", "cloudWatchLogsArn": "arn:aws:logs:eu-west-2:XXXXXXXXXXXX:log-group:/aws/codebuild/-platdev-dbmigration:log-stream:556525ab-375d-481c-8eab-20358cfb3ec8" }, "timeoutInMinutes": 60, "queuedTimeoutInMinutes": 480, "buildComplete": true, "initiator": "-platdev-cb-dbmigration-role/botocore-session-1697467958", "vpcConfig": { "vpcId": "vpc-000382333791308c1", "subnets": [ "subnet-01baa5a50fc364b02", "subnet-0a98df0ce73be5447" ], "securityGroupIds": [ "sg-0e68460a73d0ac50d" ] }, "networkInterface": { "subnetId": "subnet-0a98df0ce73be5447", "networkInterfaceId": "eni-052532d0cd1bf9110" }, "encryptionKey": "arn:aws:kms:eu-west-2:XXXXXXXXXXXX:alias/aws/s3" } ], "buildsNotFound": [] }
[Container] 2023/10/16 15:08:44 Command did not exit successfully if [ "$(jq -r .builds[0].buildStatus codebuild-output.json)" = "FAILED" ]; then echo "Failed"; cat codebuild-output.json; exit -1; fi exit status 255 [Container] 2023/10/16 15:08:44 Phase complete: BUILD State: FAILED [Container] 2023/10/16 15:08:44 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: if [ "$(jq -r .builds[0].buildStatus codebuild-output.json)" = "FAILED" ]; then echo "Failed"; cat codebuild-output.json; exit -1; fi. Reason: exit status 255 [Container] 2023/10/16 15:08:44 Entering phase POST_BUILD [Container] 2023/10/16 15:08:44 Phase complete: POST_BUILD State: SUCCEEDED [Container] 2023/10/16 15:08:44 Phase context status code: Message:
Quality Gate DB Validation log:
… [Container] 2023/10/13 13:32:05 Entering phase BUILD [Container] 2023/10/13 13:32:05 Running command aws codeartifact login --tool pip --repository -pypi-store --domain -domain-master --domain-owner YYYYYYYYYYYY Successfully configured pip to use AWS CodeArtifact repository [https://***********-domain-master-YYYYYYYYYYYY.d.codeartifact.eu-west-2.amazonaws.com/pypi/***********-pypi-store/](https://***********-domain-master- YYYYYYYYYYYY.d.codeartifact.eu-west-2.amazonaws.com/pypi/***-pypi-store/) Login expires in 12 hours at 2023-10-14 01:32:18+00:00
…
[Container] 2023/10/13 13:33:41 Phase complete: BUILD State: SUCCEEDED [Container] 2023/10/13 13:33:41 Phase context status code: Message: [Container] 2023/10/13 13:33:41 Entering phase POST_BUILD [Container] 2023/10/13 13:33:41 Phase complete: POST_BUILD State: SUCCEEDED [Container] 2023/10/13 13:33:41 Phase context status code: Message: