Error on AWS ECR

[Rowern]

Using the docker image: gcr.io/makisu-project/makisu-alpine:v0.1.9 With the folllowing configuration:

"525034297126.dkr.ecr.eu-west-3.amazonaws.com":
  .*:
    security:
      tls:
        client:
          disabled: false
      credsStore: ecr-login

I get certificate validation errors:

$ /makisu-internal/makisu build --modifyfs=true --registry-config='config.yaml' --build-arg PORT=$PORT -t api-gateway:$CI_COMMIT_SHA --push 525034297126.dkr.ecr.eu-west-3.amazonaws.com .
<truncated but can send it by email if needed>
time="2019-04-12T12:13:00Z" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: RequestError: send request failed\ncaused by: Post https://ecr.eu-west-3.amazonaws.com/: x509: certificate signed by unknown authority"
{"level":"info","ts":1555071255.1415355,"msg":"Stored cacheID mapping to KVStore: 85c88ea0 => MAKISU_CACHE_EMPTY"}
{"level":"info","ts":1555071255.1417222,"msg":"Stored cacheID mapping to KVStore: ff676eff => MAKISU_CACHE_EMPTY"}
{"level":"info","ts":1555071255.1419046,"msg":"Stored cacheID mapping to KVStore: 438ce7d5 => MAKISU_CACHE_EMPTY"}
{"level":"error","ts":1555071255.3646443,"msg":"Failed to push cache: push layer sha256:1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain; push layer sha256:981c099da328ffa21d5f38204830fd00c474758345137d11addd092eaab83264: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:981c099da328ffa21d5f38204830fd00c474758345137d11addd092eaab83264): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain; push layer sha256:80595cee04976b73593ebdda624ce272999a093e0197c3dec83b90e094810633: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:80595cee04976b73593ebdda624ce272999a093e0197c3dec83b90e094810633): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain; push layer sha256:eb16b04f4aacca5f3f82fcd255c489b4079ecb6f70f1e0e756b1ca21efb6ddf8: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:eb16b04f4aacca5f3f82fcd255c489b4079ecb6f70f1e0e756b1ca21efb6ddf8): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain; push layer sha256:e48ed15f47c0b9701d10fcaca563f2ed5711fe3ebb4dcfda6d3a3d4c7e277062: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:e48ed15f47c0b9701d10fcaca563f2ed5711fe3ebb4dcfda6d3a3d4c7e277062): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain; push layer sha256:e35e61acf59cecd1db0738f4cae62842218e074c1e3ac44f1dbb913317c48591: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:e35e61acf59cecd1db0738f4cae62842218e074c1e3ac44f1dbb913317c48591): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain; push layer sha256:d10454735bdb59ff68ce77a19817245de40bde98f9c7c0cf5b61cb88e1e9d80b: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:d10454735bdb59ff68ce77a19817245de40bde98f9c7c0cf5b61cb88e1e9d80b): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain; push layer sha256:e0178382573e9623cadadc07e9d4cc343fb308b101859e0f7591904f1ac4e8a2: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:e0178382573e9623cadadc07e9d4cc343fb308b101859e0f7591904f1ac4e8a2): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain; push layer sha256:7d8400440c5dc0beb52ca30deaeeb9fe666fc1c79237ae45aac67484f07e2d63: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:7d8400440c5dc0beb52ca30deaeeb9fe666fc1c79237ae45aac67484f07e2d63): get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain"}
{"level":"info","ts":1555071255.3652835,"msg":"Computed total image size 38038948","total_image_size":38038948}
{"level":"info","ts":1555071255.3653061,"msg":"Successfully built image api-gateway:b94322e73b7a17f0e4acc1199be56064cd073f96"}
{"level":"info","ts":1555071255.3653715,"msg":"* Started pushing image 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway:b94322e73b7a17f0e4acc1199be56064cd073f96"}
time="2019-04-12T12:14:15Z" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: RequestError: send request failed\ncaused by: Post https://ecr.eu-west-3.amazonaws.com/: x509: certificate signed by unknown authority"
{"level":"error","ts":1555071256.0205042,"msg":"failed to push image: failed to push image: check manifest exists for image 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway:b94322e73b7a17f0e4acc1199be56064cd073f96: get security opt: get credentials: get credentials from helper ecr-login: credentials not found in native keychain"}

It seems like the ecr credentials helper cannot find the cacert to validate the aws certificate.

Did I miss something ?

[evelynl94]

@Rowern we don't use aws registry in production but I guess you might need to set a few environment variables: https://github.com/awslabs/amazon-ecr-credential-helper#prerequisites

[Rowern]

Manually calling the /makisu-internal/docker-credential-ecr-login does work. (using the command: echo '525034297126.dkr.ecr.eu-west-3.amazonaws.com' | /makisu-internal/docker-credential-ecr-login get, I do get the output {"ServerURL":"525034297126.dkr.ecr.eu-west-3.amazonaws.com","Username":"AWS","Secret":"ey.."}).

What is weird is that when runned inside makisu, it does get a x509: certificate signed by unknown authority.

I tried installing ca-certificates inside the alpine, moving ca certs in the internal dir cp /etc/ssl/certs/ca-certificates.crt /makisu-internal/certs/ca-certs.pem or even specifying the CA file cp /etc/ssl/certs/ca-certificates.crt /makisu-internal/certs/ca-certs.pem in the config.yaml. Nothing seems to work...

As I'm able to generate valid credentials manually calling the docker-credential-ecr-login, how can I tweak the config.yaml to use them ?

I tried soemthing like this but it did not work:

"525034297126.dkr.ecr.eu-west-3.amazonaws.com":
  ".*":
    push_chunk: -1
    security:
      tls:
        client:
          disabled: false
      basic:
        username: AWS
        password: |-
            <base64 decoded value from the above `docker-credential-ecr-login get`["Secret"] output>

[Rowern]

Some more debug infos:

• I'm able to fix the certificate error by installing the ca-certificates package inside the alpine image
• After "fixing" the certificate errors I do get a {"level":"error","ts":1555402933.345393,"msg":"Failed to push cache: push layer sha256:bdf0201b3a056acc4d6062cc88cd8a4ad5979983bfb640f15a145e09ed985f92: check layer exists: 525034297126.dkr.ecr.eu-west-3.amazonaws.com/api-gateway (sha256:bdf0201b3a056acc4d6062cc88cd8a4ad5979983bfb640f15a145e09ed985f92): check manifest exists: HEAD https://525034297126.dkr.ecr.eu-west-3.amazonaws.com/v2/api-gateway/blobs/sha256:bdf0201b3a056acc4d6062cc88cd8a4ad5979983bfb640f15a145e09ed985f92 401"}

Weirdly enough, I do get an ouput when using curl (using credential from the docker ecr helper) inside the same docker:

$ curl -u AWS:ey... --head https://525034297126.dkr.ecr.eu-west-3.amazonaws.com/v2/api-gateway/blobs/sha256:bdf0201b3a056acc4d6062cc88cd8a4ad5979983bfb640f15a145e09ed985f92
HTTP/1.1 200 OK
...

So I think it might come from here: https://github.com/uber/makisu/blob/43f600617ec80b1984fbb0b7841fde8c0aa49ee6/lib/registry/client.go#L426-L450

[orf]

I'm also receiving this error. Is there any reason why ca-certificates shouldn't be installed inside the image?

[evelynl94]

Remember this happens in the build environment, which means packages will be installed only when it is stated in your Dockerfile. By default makisu does not install it, but it has a list of default certificates under /makisu-internal.

When I was testing the gcr credhelper I found that I needed to specify a few additional environment variables (for example, SSL_CERT_DIR) due to the assumptions in the gcr credhelper. Maybe this helps a little bit: https://github.com/uber/makisu/pull/125

[Rowern]

Doing a simple apk add -U ca-certificates fixes the issue with certificate. But then the next error is the 401.

Investigating further I found a fix (see https://github.com/uber/makisu/pull/208) that does fix the issue of authenticating to the AWS ECR.

[Rowern]

Should be fixed now that the PR is merged!

[yiranwang52]

Thanks for the fix!