search

Azure / AKS

Azure Kubernetes Service
https://azure.github.io/AKS/
1.96k stars 306 forks source link

newly spawned AKS cluster can't pull external-dns image #358

Closed jodok closed 6 years ago

jodok commented 6 years ago

Failed to pull image "zalan.azureedge.net/teapot/external-dns:v0.4.8": rpc error: code = Unknown desc = received unexpected HTTP status: 504 Gateway Timeout

39822472-7f0ff73c-535f-11e8-8f89-0b986bf21e6b

however docker run -it zalan.azureedge.net/teapot/external-dns:v0.5.0 --help on my machine works

usma0118 commented 6 years ago

facing same issue.

slack commented 6 years ago

@jodok @neutrongenious few questions!

  1. In what regions are your clusters running?
  2. What time (and timezone) did the 504 errors occur?
  3. Are those 504 errors still happening?
jodok commented 6 years ago

i'm in westeurope. yes, the error still occurs. when curl'ing the url (with https) inside a pod i could access it. it must be sth. that prevents the node accessing the url. i'm using advanced networking with my own vnet.

slack commented 6 years ago

@jodok can you successfully docker pull zalan.azureedge.net/teapot/external-dns:v0.4.8 from a node on your cluster?

From a Pod, can you run the following:

$ curl -v --location https://zalan.azureedge.net/v2/teapot/external-dns/blobs/sha256:d3e06df3437ee661a6923335e7c11fbf8581b0cb474592de21151cc9433d782b
jodok commented 6 years ago

inside a pod:

root@omsagent-qgmzl:/opt# curl -v --location https://zalan.azureedge.net/v2/teapot/external-dns/blobs/sha256:d3e06df3437ee661a6923335e7c11fbf8581b0cb474592de21151cc9433d782b
*   Trying 152.199.20.1...
* Connected to zalan.azureedge.net (152.199.20.1) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.vo.msecnd.net (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=*.vo.msecnd.net
*        start date: Fri, 30 Mar 2018 17:48:56 GMT
*        expire date: Mon, 30 Mar 2020 17:48:56 GMT
*        issuer: C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,OU=Microsoft IT,CN=Microsoft IT TLS CA 2
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /v2/teapot/external-dns/blobs/sha256:d3e06df3437ee661a6923335e7c11fbf8581b0cb474592de21151cc9433d782b HTTP/1.1
> Host: zalan.azureedge.net
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 307 Temporary Redirect
< Access-Control-Allow-Headers: 
< Access-Control-Allow-Methods: GET, POST, DELETE, PUT, PATCH, OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 3600
< Date: Thu, 10 May 2018 03:56:44 GMT
< Location: https://zalando-opensource-os-registry-eu-west-1.s3.eu-west-1.amazonaws.com/sha256%3Ad3e06df3437ee661a6923335e7c11fbf8581b0cb474592de21151cc9433d782b?X-Amz-Security-Token=FQoDYXdzEOX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDFhcpFtARYc5lu1v1yK3A9rqR7sc6IrpURZEvAcpB%2Ft7%2FDmhLHIUX00mdWDTgvxiOJi%2FbAsdFPWRYm9kTtBznlfFdXoWJM9u2bnxRi8eL%2B%2FH10sjI8pMYqmI8PVi96OmKw02mUPjWMvkj0X40Q4hAYK5tz5ABakzqznM2vlLPlVXkQPXMjoqFnlws5I8tMqsPaoju8diDTF9GnKBepXqveLuEJnjB5uTFZIchd2E0aNrnr%2F9GIhcTLCNzixEGDdd%2FIE8yl7Z%2FDIJ2x3ywijgvoVFT%2BYKmw3Xg%2BSAcioo63L5hl2bPmu1rzwXjnPRcGaTAJT43jdgDWtvTbMSAUGq%2BdDG%2BuZ4APr2QPgmUoc9HcYLfcfCrHKHwAm9PltWiTvkTvuttmvtNxO%2FLjPYr52WQsscHBdz%2Fp24lja%2Fgx2taba1SbTQt2bdNpNwpFK0DPA1YYvjXM0j6zjZYWJhYmU3BMtjpNbUgkaSWuUfI3LXKaaMxMv1bMZR3ZrF1yFi0H%2BAZEwDVDrBNaqqfVfOnjBBR7bcMWJfAyipQCcOrk5oPjaxpzrZ1jg9dX0fntmMq9BbaQNsWZ3%2FtEHingdUBdTUQfQIBSDS8fgo5fvO1wU%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180510T035644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=30&X-Amz-Credential=ASIAIGON2A2PQB3OWCQA%2F20180510%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=487d9e5a048875a7a43ec67913912ce621942c1d9e9d001d525d71430dcf3a7c
< Server: Jetty(9.2.z-SNAPSHOT)
< Strict-Transport-Security: max-age=10886400
< Content-Length: 0
< 
* Connection #0 to host zalan.azureedge.net left intact
* Issue another request to this URL: 'https://zalando-opensource-os-registry-eu-west-1.s3.eu-west-1.amazonaws.com/sha256%3Ad3e06df3437ee661a6923335e7c11fbf8581b0cb474592de21151cc9433d782b?X-Amz-Security-Token=FQoDYXdzEOX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDFhcpFtARYc5lu1v1yK3A9rqR7sc6IrpURZEvAcpB%2Ft7%2FDmhLHIUX00mdWDTgvxiOJi%2FbAsdFPWRYm9kTtBznlfFdXoWJM9u2bnxRi8eL%2B%2FH10sjI8pMYqmI8PVi96OmKw02mUPjWMvkj0X40Q4hAYK5tz5ABakzqznM2vlLPlVXkQPXMjoqFnlws5I8tMqsPaoju8diDTF9GnKBepXqveLuEJnjB5uTFZIchd2E0aNrnr%2F9GIhcTLCNzixEGDdd%2FIE8yl7Z%2FDIJ2x3ywijgvoVFT%2BYKmw3Xg%2BSAcioo63L5hl2bPmu1rzwXjnPRcGaTAJT43jdgDWtvTbMSAUGq%2BdDG%2BuZ4APr2QPgmUoc9HcYLfcfCrHKHwAm9PltWiTvkTvuttmvtNxO%2FLjPYr52WQsscHBdz%2Fp24lja%2Fgx2taba1SbTQt2bdNpNwpFK0DPA1YYvjXM0j6zjZYWJhYmU3BMtjpNbUgkaSWuUfI3LXKaaMxMv1bMZR3ZrF1yFi0H%2BAZEwDVDrBNaqqfVfOnjBBR7bcMWJfAyipQCcOrk5oPjaxpzrZ1jg9dX0fntmMq9BbaQNsWZ3%2FtEHingdUBdTUQfQIBSDS8fgo5fvO1wU%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180510T035644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=30&X-Amz-Credential=ASIAIGON2A2PQB3OWCQA%2F20180510%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=487d9e5a048875a7a43ec67913912ce621942c1d9e9d001d525d71430dcf3a7c'
*   Trying 52.218.49.84...
* Connected to zalando-opensource-os-registry-eu-west-1.s3.eu-west-1.amazonaws.com (52.218.49.84) port 443 (#1)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.s3-eu-west-1.amazonaws.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=Washington,L=Seattle,O=Amazon.com Inc.,CN=*.s3-eu-west-1.amazonaws.com
*        start date: Fri, 22 Sep 2017 00:00:00 GMT
*        expire date: Thu, 03 Jan 2019 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert Baltimore CA-2 G2
*        compression: NULL
* ALPN, server did not agree to a protocol
> GET /sha256%3Ad3e06df3437ee661a6923335e7c11fbf8581b0cb474592de21151cc9433d782b?X-Amz-Security-Token=FQoDYXdzEOX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDFhcpFtARYc5lu1v1yK3A9rqR7sc6IrpURZEvAcpB%2Ft7%2FDmhLHIUX00mdWDTgvxiOJi%2FbAsdFPWRYm9kTtBznlfFdXoWJM9u2bnxRi8eL%2B%2FH10sjI8pMYqmI8PVi96OmKw02mUPjWMvkj0X40Q4hAYK5tz5ABakzqznM2vlLPlVXkQPXMjoqFnlws5I8tMqsPaoju8diDTF9GnKBepXqveLuEJnjB5uTFZIchd2E0aNrnr%2F9GIhcTLCNzixEGDdd%2FIE8yl7Z%2FDIJ2x3ywijgvoVFT%2BYKmw3Xg%2BSAcioo63L5hl2bPmu1rzwXjnPRcGaTAJT43jdgDWtvTbMSAUGq%2BdDG%2BuZ4APr2QPgmUoc9HcYLfcfCrHKHwAm9PltWiTvkTvuttmvtNxO%2FLjPYr52WQsscHBdz%2Fp24lja%2Fgx2taba1SbTQt2bdNpNwpFK0DPA1YYvjXM0j6zjZYWJhYmU3BMtjpNbUgkaSWuUfI3LXKaaMxMv1bMZR3ZrF1yFi0H%2BAZEwDVDrBNaqqfVfOnjBBR7bcMWJfAyipQCcOrk5oPjaxpzrZ1jg9dX0fntmMq9BbaQNsWZ3%2FtEHingdUBdTUQfQIBSDS8fgo5fvO1wU%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180510T035644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=30&X-Amz-Credential=ASIAIGON2A2PQB3OWCQA%2F20180510%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=487d9e5a048875a7a43ec67913912ce621942c1d9e9d001d525d71430dcf3a7c HTTP/1.1
> Host: zalando-opensource-os-registry-eu-west-1.s3.eu-west-1.amazonaws.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-amz-id-2: sYWB5KmbHM2CMZrU5APLf6fj/wI2JtU7YcGnb8zZQaoOmOPk0jk9Hcdyp2I4IKpGb1j8SMQib6U=
< x-amz-request-id: 45E69267D1106CB8
< Date: Thu, 10 May 2018 03:56:45 GMT
< Last-Modified: Wed, 22 Nov 2017 18:43:53 GMT
< ETag: "efc966146b67f14cb21d8e570aa1996c"
< Accept-Ranges: bytes
< Content-Type: application/octet-stream
< Content-Length: 4699
< Server: AmazonS3
< 
{"architecture":"amd64","author":"Team Teapot @ Zalando SE \u003cteam-teapot@zalando.de\u003e","config":{"Hostname":"3fab32a108fc","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":null,"ArgsEscaped":true,"Image":"sha256:e41bb346a3be39080bcc7b89042f2f626d461a1e6cee84314960c1214a50d4c7","Volumes":null,"WorkingDir":"","Entrypoint":["/bin/external-dns"],"OnBuild":[],"Labels":{}},"container":"04c31459c2348de379a11968f90b2e63076baa0087bd70e7aea2a1980b1c8806","container_config":{"Hostname":"3fab32a108fc","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/sh","-c","#(nop) ","ENTRYPOINT [\"/bin/external-dns\"]"],"ArgsEscaped":true,"Image":"sha256:e41bb346a3be39080bcc7b89042f2f626d461a1e6cee84314960c1214a50d4c7","Volumes":null,"WorkingDir":"","Entrypoint":["/bin/external-dns"],"OnBuild":[],"Labels":{}},"created":"2017-11-22T18:43:48.496739189Z","docker_version":"17.09.0-ce","history":[{"created":"2017-05-10T16:38:16.632557712Z","created_by":"/bin/sh -c #(nop) ADD file:63f63606d6e289eb607c90e31de81802258906712727e473a2898f0f1ae55bb5 in / "},{"created":"2017-05-10T16:38:17.391552466Z","created_by":"/bin/sh -c #(nop)  CMD [\"/bin/sh\"]","empty_layer":true},{"created":"2017-05-23T09:34:00.675250786Z","author":"Zalando SE","created_by":"/bin/sh -c #(nop)  MAINTAINER Zalando SE","empty_layer":true},{"created":"2017-05-23T09:34:01.751109275Z","author":"Zalando SE","created_by":"/bin/sh -c apk --no-cache upgrade \u0026\u0026 apk --no-cache add ca-certificates"},{"created":"2017-05-23T09:34:02.00359598Z","author":"Zalando SE","created_by":"/bin/sh -c #(nop) ADD tarsum.v1+sha256:c5d0df989477cf5023a5478b04e7ab4bb41732760b696d84b90dbe8517a30c69 in /usr/local/share/ca-certificates/zalando-root.crt "},{"created":"2017-05-23T09:34:02.195479696Z","author":"Zalando SE","created_by":"/bin/sh -c #(nop) ADD tarsum.v1+sha256:1b1365fa1b33801fd231ef5709d9e7ad097edc22b33a2e0cd4edc98c67069c70 in /usr/local/share/ca-certificates/zalando-service.crt "},{"created":"2017-05-23T09:34:02.843494616Z","author":"Zalando SE","created_by":"/bin/sh -c #(nop) ADD tarsum.v1+sha256:9f67a2172b3039d13b9af49cf02eebc6d4edc5e50a950df12579e81f7a69ea4f in /tmp/rds-ca/aws-rds-ca-bundle.pem "},{"created":"2017-05-23T09:34:03.357446657Z","author":"Zalando SE","created_by":"/bin/sh -c cd /tmp/rds-ca \u0026\u0026 cat aws-rds-ca-bundle.pem|awk 'split_after==1{n++;split_after=0} /-----END CERTIFICATE-----/ {split_after=1} {print \u003e \"cert\" n \"\"}'     \u0026\u0026 for CERT in /tmp/rds-ca/cert*; do mv $CERT /usr/local/share/ca-certificates/aws-rds-ca-$(basename $CERT).crt; done     \u0026\u0026 rm -rf /tmp/rds-ca     \u0026\u0026 update-ca-certificates"},{"created":"2017-05-23T09:34:03.467282599Z","author":"Zalando SE","created_by":"/bin/sh -c #(nop)  CMD [\"/bin/sh\"]","empty_layer":true},{"created":"2017-11-22T18:43:46.723785308Z","author":"Team Teapot @ Zalando SE \u003cteam-teapot@zalando.de\u003e","created_by":"/bin/sh -c #(nop)  MAINTAINER Team Teapot @ Zalando SE \u003cteam-teapot@zalando.de\u003e","empty_layer":true},{"created":"2017-11-22T18:43:47.373334542Z","author":"Team Teapot @ Zalando SE \u003cteam-teapot@zalando.de\u003e","created_by":"/bin/sh -c #(nop) COPY file:9461d375c7da0e013b5b39a40506f121962d94ec8d832b7e85fbc9a2382e6fd1 in /bin/external-dns ","empty_layer":true},{"created":"2017-11-22T18:43:47.491551906Z","author":"Team Teapot @ Zalando SE \u003cteam-teapot@zalando.de\u003e","created_by":"/bin/sh -c #(nop)  ENTRYPOINT [\"/bin/external-dns\"]","empty_layer":true},{"created":"2017-11-22T18:43:48.496739189Z","comment":"merge sha256:e57bc5b495219fdb095276b7ef2e1bccb83e136401141b608c62c28791513a08 to sha256:3be2bfccaf548d06623a16b2a80aa51326ba2f1a4b859f780033c4262f2f8d9c"}],"os":"linux","parent":"sha256:e41bb346a3be39080bcc7* Connection #1 to host zalando-opensource-os-registry-eu-west-1.s3.eu-west-1.amazonaws.com left intact
b89042f2f626d461a1e6cee84314960c1214a50d4c7","rootfs":{"type":"layers","diff_ids":["sha256:e154057080f406372ebecadc0bfb5ff8a7982a0d13823bab1be5b86926c6f860","sha256:cd15158f3bfd21290947120aabf4c7581cab55d98453ee278cc4a91b7d00adf4","sha256:b044fabc2e41b1e8144dfbfc6ed1df38c65887a35be161b699d41ab07810ddf5","sha256:0997a3b18b81f452f2777598e64922666629c94811b9acbcc3d0b157f138fb40","sha256:7f3d9579fb8678a394644d3b3962d30023c91397219f60ab1a8d545ae1b6da98","sha256:5d0059763112f46d59f35ea0df8147c1cd0da700da2b1d80fe5b16054925f55d","sha256:8fa16ef94de2e2bb9c2fd0da356c3ab729d439e3584c0b119c792a65c6cbda51"]}}
root@omsagent-qgmzl:/opt#
jodok commented 6 years ago

after SSHing to the node it seems a layer can't be pulled:

azureuser@aks-agentpool-42886762-0:~$ docker pull zalan.azureedge.net/teapot/external-dns:v0.4.8
v0.4.8: Pulling from teapot/external-dns
d5d0715331fc: Pull complete
a11c6302aab6: Pull complete
9637a3af7b6a: Pull complete
6d5225acfc23: Pull complete
1d5f6d9cdece: Retrying in 7 seconds
08a3bfc4a41b: Download complete
0a1073fc4ca6: Download complete

and after the timeout:

azureuser@aks-agentpool-42886762-0:~$ docker pull zalan.azureedge.net/teapot/external-dns:v0.4.8
v0.4.8: Pulling from teapot/external-dns
d5d0715331fc: Pull complete
a11c6302aab6: Pull complete
9637a3af7b6a: Pull complete
6d5225acfc23: Pull complete
1d5f6d9cdece: Downloading
08a3bfc4a41b: Download complete
0a1073fc4ca6: Download complete
received unexpected HTTP status: 504 Gateway Timeout
slack commented 6 years ago

Ok, took some digging, how about fetching the blob for the specific layer:


$ curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  -vvv -k \
https://zalan.azureedge.net/v2/teapot/external-dns/blobs/sha256:1d5f6d9cdecebbd551e24c467fcc83def6611ddf20564759405392c00b275a93
``
jodok commented 6 years ago

thanks for your help (easy way for me to learn more about pulling layers via curl :)). here's the output:

azureuser@aks-agentpool-42886762-0:~$ curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -vvv -k https://zalan.azureedge.net/v2/teapot/external-dns/blobs/sha256:1d5f6d9cdecebbd551e24c467fcc83def6611ddf20564759405392c00b275a93
*   Trying 152.199.20.1...
* Connected to zalan.azureedge.net (152.199.20.1) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*    server certificate verification SKIPPED
*    server certificate status verification SKIPPED
*    common name: *.vo.msecnd.net (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: CN=*.vo.msecnd.net
*    start date: Fri, 30 Mar 2018 17:48:56 GMT
*    expire date: Mon, 30 Mar 2020 17:48:56 GMT
*    issuer: C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,OU=Microsoft IT,CN=Microsoft IT TLS CA 2
*    compression: NULL
* ALPN, server accepted to use http/1.1
> GET /v2/teapot/external-dns/blobs/sha256:1d5f6d9cdecebbd551e24c467fcc83def6611ddf20564759405392c00b275a93 HTTP/1.1
> Host: zalan.azureedge.net
> User-Agent: curl/7.47.0
> Accept: application/vnd.docker.distribution.manifest.v2+json
>
< HTTP/1.1 504 Gateway Timeout
< Content-Type: application/vnd.docker.distribution.manifest.v2+json
< Date: Thu, 10 May 2018 05:02:44 GMT
< Server: ECAcc (lhb/636B)
< Content-Length: 357
<
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
        <title>504 - Gateway Timeout</title>
    </head>
    <body>
        <h1>504 - Gateway Timeout</h1>
    </body>
</html>
* Connection #0 to host zalan.azureedge.net left intact
joakimhellum commented 6 years ago

Any update on this? Still experiencing same issue; not able to pull this image from a node in Azure (or anywhere in Azure), however it works fine locally.

lorsabyan commented 6 years ago

I have the same issue and solved it by copying the image from my local machine to Kubernetes Nodes.

Steps are the following:

on local machine:

  1. docker pull zalan.azureedge.net/teapot/external-dns:v0.4.8
  2. docker save -o ~/external-dns.tar zalan.azureedge.net/teapot/external-dns:v0.4.8

then SSH into Azure Kubernetes Service (AKS) cluster nodes and then

  1. copy external-dns.tar from local machine via scp to aks-ssh container
  2. then from aks-ssh container copy external-dns.tar to target Nodes
  3. on target Node run docker load -i external-dns.tar
  4. delete failed Pod from Kubernetes Pods under kube-system namespace and Kubernetes will do the remaining work :smile:

I hope this will be helpful for you.

slack commented 6 years ago

Thanks @jodok, I'm sending this information over to the CDN team to look at.

joachimaumann commented 6 years ago

same issue here:

Warning Failed 41m (x740 over 2d) kubelet, aks-agentpool-75369199-0 Error: ErrImagePull Warning Failed 11m (x537 over 2d) kubelet, aks-agentpool-75369199-0 Failed to pull image "zalan.azureedge.net/teapot/external-dns:v0.4.8": rpc error: code = Unknown desc = received unexpected HTTP status: 504 Gateway Timeout Warning Failed 6m (x16361 over 2d) kubelet, aks-agentpool-75369199-0 Error: ImagePullBackOff Normal BackOff 1m (x16378 over 2d) kubelet, aks-agentpool-75369199-0 Back-off pulling image "zalan.azureedge.net/teapot/external-dns:v0.4.8"

usma0118 commented 6 years ago

same here:

 kubectl describe pods addon-http-application-routing-external-dns-7464fdb989-f8f7g -n kube-system
Name:           addon-http-application-routing-external-dns-7464fdb989-f8f7g
Namespace:      kube-system
Node:           aks-agentpool-38494388-0/10.35.104.6
Start Time:     Fri, 11 May 2018 11:58:39 +0200
Labels:         app=addon-http-application-routing-external-dns
                pod-template-hash=3020986545
Annotations:    <none>
Status:         Pending
IP:             10.35.104.25
Controlled By:  ReplicaSet/addon-http-application-routing-external-dns-7464fdb989
Containers:
  addon-http-application-routing-external-dns:
    Container ID:
    Image:         zalan.azureedge.net/teapot/external-dns:v0.4.8
    Image ID:
    Port:          <none>
    Args:
      --source=service
      --source=ingress
      --provider=azure
      --azure-resource-group=MC_akscluster-we_akscluster-we_westeurope
      --domain-filter=f1e70880-7550-4af8-857d-1a23d06e91bd.westeurope.aksapp.io
      --annotation-filter=kubernetes.io/ingress.class=addon-http-application-routing
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/kubernetes/azure.json from azure-config-file (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from addon-http-application-routing-external-dns-token-hzkrc (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  azure-config-file:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:
  addon-http-application-routing-external-dns-token-hzkrc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  addon-http-application-routing-external-dns-token-hzkrc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  **Type     Reason   Age                  From                               Message
  ----     ------   ----                 ----                               -------
  Warning  Failed   56m (x87 over 9h)    kubelet, aks-agentpool-38494388-0  Error: ErrImagePull
  Normal   BackOff  51m (x1840 over 9h)  kubelet, aks-agentpool-38494388-0  Back-off pulling image "zalan.azureedge.net/teapot/external-dns:v0.4.8"
  Normal   Pulling  26m (x92 over 9h)    kubelet, aks-agentpool-38494388-0  pulling image "zalan.azureedge.net/teapot/external-dns:v0.4.8"
  Warning  Failed   1m (x2017 over 9h)   kubelet, aks-agentpool-38494388-0  Error: ImagePullBackOff**
slack commented 6 years ago

Still working with the CDN team to find the Edge node which is failing to return the layer.

slack commented 6 years ago

@neutrongenious @joachimaumann I believe this is fixed on the CDN. Can you confirm?

jodok commented 6 years ago

@slack image can be pulled now. thanks for your help!

dknoodle commented 6 years ago

We are still seeing this same issue pulling from an Azure container registry. Our AKS cluster is running in Central US but our container registry is deployed to South Central US. Surely the registry data center really shouldn't matter?

dknoodle commented 6 years ago

I deployed a new registry in the same data center (Central US) and same resource group but we are still getting the blob error. Does anyone have any insights? Should I submit a support ticket to Azure support for this?