Support User Management Tool updates: ECR publishing and pushing live to EKS

[paulpascal]

This ticket has two areas that need fixing after new updates to the tool which add a new image and have three services now instead of one:

EKS/helm

Right now the User Man Tool exists in three production URLS:

• users-chis-ke.app.medicmobile.org
• users-chis-tg.app.medicmobile.org
• users-chis-ug.app.medicmobile.org

We do production updates by following these steps. However, with the new updates we need to ensure that 3 services (main, worker & redis) will be pushed to prod instead of just one (main).

ECR Image publishing

To support the cloud base execution of Contact Move we added a job worker that should handle all the contact move job.

There is an existing job that build and publish the cht-user-management image and has been updated to also build the cht-user-management-worker image.

While cht-user-management image gets published correctly, the cht-user-management-worker image fails with error:

denied: User: arn:aws:iam::***:user/ci-ecr is not authorized to perform: ecr-public:InitiateLayerUpload on resource: arn:aws:ecr-public::***:repository/cht-user-management-worker because no identity-based policy allows the ecr-public:InitiateLayerUpload action

cc: @henokgetachew @Hareet @mrjones-plip

[paulpascal]

Dear @henokgetachew @Hareet, Is there please any update on this ?

[mrjones-plip]

I have tried:

• logged into AWS console
• created new cht-user-management-worker ECR repo: public and private view
• tried re-running the failed CI job but got same InitiateLayerUpload error. Note that cht-user-management image correctly gets published.
• didn't see any permissions settings on the ECR admin side of things, so assume that secrets.ECR_ACCESS_KEY_ID and secrets.ECR_SECRET_ACCESS_KEY work as expected. They obviously work at a high level with cht-user-management image
• not knowing how to get CI builds to work, I tried following the manual Push commands from the ECR dashboard. running aws ecr-public get-login-password --region us-east-1 I get the error not authorized to perform: ecr-public:GetAuthorizationToken. I see no way of uploading an image via the web UI :(

[mrjones-plip]

I'm stuck on this and can't proceed any further.

@1yuv - any thoughts on how to either upload an image manually (maybe you can try? happy to pair real quick!) or get the CI to do it by re-running the failed job after some fix?

Otherwise, back to SRE!

[paulpascal]

Hello @Hareet @henokgetachew , is there please any update on this ?

Thanks

[Hareet]

This is fixed.

created new cht-user-management-worker ECR repo: public and private view

Thanks! And then the original error Paul posted above, and likely the one you ran into " tried re-running the failed CI job but got same InitiateLayerUpload error. "

Original error from top post:

denied: User: arn:aws:iam::***:user/ci-ecr is not authorized to perform

Peaking into IAM in AWS, we see each ecr registry needs to be added to the ci-user to enable push access.

"Action": [
                "ecr-public:InitiateLayerUpload",
                "ecr-public:UploadLayerPart",
                "ecr-public:PutImage",
                "ecr-public:CompleteLayerUpload",
                "ecr-public:BatchCheckLayerAvailability"
            ],
 "Resource": [
                "arn:aws:ecr-public:::repository/couchdb-migration",
                "arn:aws:ecr:eu-west-2::repository/medic/cht-api",
                 "arn:aws:ecr-public:::repository/cht-user-management",
                "arn:aws:ecr-public:::repository/cht-user-management-worker"

I've added the registry name that @mrjones-plip created in the earlier step

[mrjones-plip]

Thanks @Hareet !

[paulpascal]

Thanks @Hareet. Really appreciate that. Just one question please, will the cht-user-management helm chart need an update as well, to be able to run now three services instead of one ?

cc: @mrjones-plip

[mrjones-plip]

@paulpascal - We should try as best we can to fix helm and if we can't, then open a separate issue to track updating that.

[paulpascal]

Of course @mrjones-plip - thanks