search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.27k stars 243 forks source link

[BUG] podman copy to internal crc registry fails connection refused #3246

Open ayates83 opened 2 years ago

ayates83 commented 2 years ago

General information

CRC version

CRC version: 2.4.1+6b954d40
OpenShift version: 4.10.14
Podman version: 4.0.2

CRC status

 crc status --log-level debug
DEBU CRC version: 2.4.1+6b954d40                  
DEBU OpenShift version: 4.10.14                   
DEBU Podman version: 4.0.2                        
DEBU Running 'crc status'                         
DEBU Checking file: /Users/userid/.crc/machines/crc/.crc-exist 
DEBU Checking file: /Users/userid/.crc/machines/crc/.crc-exist 
DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1 
DEBU Using ssh private keys: [/Users/userid/.crc/machines/crc/id_ecdsa /Users/userid/.crc/cache/crc_vfkit_4.10.14_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: <nil>, output: 32737570816 18435547136 /sysroot 
DEBU Unexpected operator status for etcd: RecentBackup 
CRC VM:          Running
OpenShift:       Running (v4.10.14)
Podman:          
Disk Usage:      18.44GB of 32.74GB (Inside the CRC VM)
Cache Usage:     37GB
Cache Directory: /Users/userid/.crc/cache

CRC config

- consent-telemetry                     : no

Host Operating System

ProductName:    macOS
ProductVersion: 12.4
BuildVersion:   21F79

Steps to reproduce

  1. oc login -u kubeadmin -p $pw https://https://api.crc.testing:6443
  2. REG=$(oc get route -n openshift-image-registry | awk '{print $2}' | grep -v HOST)
  3. podman login -u kubeadmin -p $(oc whoami -t)
  4. podman tag localhost/image ${REG}/project/image:latest
  5. podman push ${REG}/project/image:latest

Expected

expecting the copy to succeed to the internal registry

Actual

  1. podman push ${REG}/project/image:latest Error: error pushing image "default-route-openshift-image-registry.apps-crc.testing/project/image:latest": trying to reuse blob sha256:5d256cdcff666d660337d13f80caec5485cb8997642ce5e97cf4191cb5a06585 at destination: pinging container registry default-route-openshift-image-registry.apps-crc.testing: Get "https://default-route-openshift-image-registry.apps-crc.testing/v2/": dial tcp 127.0.0.1:443: connect: connection refused

Logs

See attached debug log from crc start --log-level debug


[crc-debug.txt](https://github.com/code-ready/crc/files/9043737/crc-debug.txt)
ayates83 commented 2 years ago

crc-debug.txt

praveenkumar commented 2 years ago

@ayates83 As per debug logs I can see following message which means cluster was not even running because of cert error

evel=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 1, output: "
level=debug msg="Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-07-01T19:53:29Z is after 2022-07-01T03:52:10Z\n"

Can you download the latest version of crc 2.5.1 and try again?

ayates83 commented 2 years ago

CRC version: 2.5.1+5b02b826 OpenShift version: 4.10.18 Podman version: 4.1.0

updated to crc 2.5.1 ; here's the debug log output 2crc-debug.txt

login successful:

podman login -u kubeadmin -p $(oc whoami -t) $REG --tls-verify=false
Login Succeeded!

push fails:

podman push ${REG}/project/image:latest
Error: error pushing image "default-route-openshift-image-registry.apps-crc.testing/project/image:latest": trying to reuse blob sha256:5d256cdcff666d660337d13f80caec5485cb8997642ce5e97cf4191cb5a06585 at destination: pinging container registry default-route-openshift-image-registry.apps-crc.testing: Get "https://default-route-openshift-image-registry.apps-crc.testing/v2/": dial tcp 127.0.0.1:443: connect: connection refused

same error occurs when trying --tls-verify=false as well.

praveenkumar commented 2 years ago

@ayates83 Can you provide the output of following?

$ oc get co
$ oc projects | grep <project_where_you_want_to_push>  // Make sure that project exist
$ oc get pods -n openshift-image-registry
NAME                                               READY   STATUS      RESTARTS   AGE
cluster-image-registry-operator-55fb9dcd9b-5qpcp   1/1     Running     0          22h
image-pruner-27620640-cw79d                        0/1     Completed   0          3h12m
image-registry-5bd79db9d6-szsn5                    1/1     Running     0          20h
node-ca-dtwc5                                      1/1     Running     0          47h
$ oc logs <image_registry_pod> -n openshift-image-registry | tail
ayates83 commented 2 years ago 

2022.07.07 22:15:24 [userid@host ~/git/osecli/ocp_resources] $ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.18   True        False         False      11h     
config-operator                            4.10.18   True        False         False      16d     
console                                    4.10.18   True        False         False      11h     
dns                                        4.10.18   True        False         False      11h     
etcd                                       4.10.18   True        False         False      16d     
image-registry                             4.10.18   True        False         False      11h     
ingress                                    4.10.18   True        False         False      16d     
kube-apiserver                             4.10.18   True        False         False      16d     
kube-controller-manager                    4.10.18   True        False         False      16d     
kube-scheduler                             4.10.18   True        False         False      16d     
machine-api                                4.10.18   True        False         False      16d     
machine-approver                           4.10.18   True        False         False      16d     
machine-config                             4.10.18   True        False         False      16d     
marketplace                                4.10.18   True        False         False      16d     
network                                    4.10.18   True        False         False      16d     
node-tuning                                4.10.18   True        False         False      11h     
openshift-apiserver                        4.10.18   True        False         False      11h     
openshift-controller-manager               4.10.18   True        False         False      11h     
openshift-samples                          4.10.18   True        False         False      16d     
operator-lifecycle-manager                 4.10.18   True        False         False      16d     
operator-lifecycle-manager-catalog         4.10.18   True        False         False      16d     
operator-lifecycle-manager-packageserver   4.10.18   True        False         False      11h     
service-ca                                 4.10.18   True        False         False      16d     
2022.07.07 22:15:49 [userid@host ~/git/osecli/ocp_resources] $ oc project | grep backup
Using project "backup" on server "https://api.crc.testing:6443".
2022.07.07 22:15:57 [userid@host ~/git/osecli/ocp_resources] $ oc get pods -n openshift-image-registry
NAME                                              READY   STATUS      RESTARTS   AGE
cluster-image-registry-operator-5959746b9-4tq76   1/1     Running     0          15d
image-pruner-27620640-d76p8                       0/1     Completed   0          3h16m
image-registry-7dc8f48898-8jztf                   1/1     Running     0          11h
node-ca-kkrfz                                     1/1     Running     0          16d
2022.07.07 22:16:12 [userid@host ~/git/osecli/ocp_resources] $ oc logs -n openshift-image-registry image-registry-7dc8f48898-8jztf | tail
time="2022-07-08T03:16:07.345738576Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=173c9211-da64-402b-8521-c8576d233530 http.request.method=GET http.request.remoteaddr="10.217.0.1:54492" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="23.673µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:07.345763881Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=85519578-d484-4de0-ac84-e8daf9087b33 http.request.method=GET http.request.remoteaddr="10.217.0.1:54490" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="16.355µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:17.347388073Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=976636b3-c65b-4157-810a-02630f616c73 http.request.method=GET http.request.remoteaddr="10.217.0.1:54676" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="25.159µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:17.347683133Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=2b754824-c390-443f-b821-f8ee15db4f89 http.request.method=GET http.request.remoteaddr="10.217.0.1:54674" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="13.388µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:27.345786456Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=44b3bc60-1f84-4153-9aaf-9684a3b9c95e http.request.method=GET http.request.remoteaddr="10.217.0.1:54878" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="28.977µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:27.346075778Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=187b76db-4be2-4ba9-97e0-4af9fc8a443f http.request.method=GET http.request.remoteaddr="10.217.0.1:54880" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="16.331µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:37.347688518Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=7d45817d-05f9-4f5e-98d4-468457bfeca7 http.request.method=GET http.request.remoteaddr="10.217.0.1:55092" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="25.419µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:37.348086428Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=fe2f5a8c-408b-4a76-8bdc-979d5eb91838 http.request.method=GET http.request.remoteaddr="10.217.0.1:55094" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="18.906µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:47.348250833Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=b11ec538-5704-4007-ad6b-b044e55a9822 http.request.method=GET http.request.remoteaddr="10.217.0.1:55284" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="50.495µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:47.348512699Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=14cd2483-80f1-47be-952c-f7488cf9190a http.request.method=GET http.request.remoteaddr="10.217.0.1:55282" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="16.311µs" http.response.status=200 http.response.written=0

I'm wondering if it has something to do with net namespaces? Something similar to the docker on osx issue interacting via namespaces?

ayates83 commented 2 years ago

Something similar to this issue? https://github.com/code-ready/crc/issues/2354

ayates83 commented 2 years ago

Fails with same messages on the latest release as well:

 $ crc version

CRC version: 2.6.0+e7251680
OpenShift version: 4.10.22
Podman version: 4.1.0

$ podman push localhost/ocpbackup $REG/backup/ocpbackup

Error: error pushing image "default-route-openshift-image-registry.apps-crc.testing/backup/ocpbackup": trying to reuse blob sha256:5bc03dec623975972bc748cb1da06ce8b4cf854d38c854a9e5f5bcad48caf6cc at destination: pinging container registry default-route-openshift-image-registry.apps-crc.testing: Get "https://default-route-openshift-image-registry.apps-crc.testing/v2/": dial tcp 127.0.0.1:443: connect: connection refused

I can connect to the URL using curl:


 $ echo $REG
default-route-openshift-image-registry.apps-crc.testing

$ curl -kv https://$REG
*   Trying 127.0.0.1:443...
* Connected to default-route-openshift-image-registry.apps-crc.testing (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.apps-crc.testing
*  start date: Jul  6 12:45:45 2022 GMT
*  expire date: Jul  5 12:45:46 2024 GMT
*  issuer: CN=ingress-operator@1657111544
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET / HTTP/1.1
> Host: default-route-openshift-image-registry.apps-crc.testing
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< cache-control: no-cache
< date: Wed, 27 Jul 2022 11:53:34 GMT
< content-length: 0
< set-cookie: 34727b82525eb26a530629c5bf0ec2f2=4b9996899c7e3a089f410965ad2bfd8a; path=/; HttpOnly; Secure; SameSite=None
< 
* Connection #0 to host default-route-openshift-image-registry.apps-crc.testing left intact

I can hit the v2 api endpoint with a bearer token and list the existing images:


$ curl -kv -H "Authorization: Bearer ${TOK}" https://$REG/v2/_catalog
*   Trying 127.0.0.1:443...
* Connected to default-route-openshift-image-registry.apps-crc.testing (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.apps-crc.testing
*  start date: Jul  6 12:45:45 2022 GMT
*  expire date: Jul  5 12:45:46 2024 GMT
*  issuer: CN=ingress-operator@1657111544
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET /v2/_catalog HTTP/1.1
> Host: default-route-openshift-image-registry.apps-crc.testing
> User-Agent: curl/7.79.1
> Accept: */*
> Authorization: Bearer sha256~A0M149SEElavwCbp7BBXkMF1ieBQ4YUzUyT0NH7L_zo
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< docker-distribution-api-version: registry/2.0
< x-registry-supports-signatures: 1
< date: Wed, 27 Jul 2022 12:04:46 GMT
< transfer-encoding: chunked
< set-cookie: 34727b82525eb26a530629c5bf0ec2f2=4b9996899c7e3a089f410965ad2bfd8a; path=/; HttpOnly; Secure; SameSite=None
< cache-control: private
< 
{"repositories":["openshift/apicast-gateway","openshift/apicurito-ui","openshift/cli","openshift/cli-artifacts","openshift/dotnet","openshift/dotnet-runtime","openshift/driver-toolkit","openshift/fis-java-openshift","openshift/fis-karaf-openshift","openshift/fuse-apicurito-generator","openshift/fuse7-console","openshift/fuse7-eap-openshift","openshift/fuse7-eap-openshift-java11","openshift/fuse7-java-openshift","openshift/fuse7-java11-openshift","openshift/fuse7-karaf-openshift","openshift/fuse7-karaf-openshift-jdk11","openshift/golang","openshift/httpd","openshift/installer","openshift/installer-artifacts","openshift/java","openshift/java-runtime","openshift/jboss-amq-62","openshift/jboss-amq-63","openshift/jboss-datagrid65-client-openshift","openshift/jboss-datagrid65-openshift","openshift/jboss-datagrid71-client-openshift","openshift/jboss-datagrid71-openshift","openshift/jboss-datagrid72-openshift","openshift/jboss-datagrid73-openshift","openshift/jboss-datavirt64-driver-openshift","openshift/jboss-datavirt64-openshift","openshift/jboss-decisionserver64-openshift","openshift/jboss-eap-xp3-openjdk11-openshift","openshift/jboss-eap-xp3-openjdk11-runtime-openshift","openshift/jboss-eap64-openshift","openshift/jboss-eap70-openshift","openshift/jboss-eap71-openshift","openshift/jboss-eap72-openjdk11-openshift-rhel8","openshift/jboss-eap72-openshift","openshift/jboss-eap73-openjdk11-openshift","openshift/jboss-eap73-openjdk11-runtime-openshift","openshift/jboss-eap73-openshift","openshift/jboss-eap73-runtime-openshift","openshift/jboss-eap74-openjdk11-openshift","openshift/jboss-eap74-openjdk11-runtime-openshift","openshift/jboss-eap74-openjdk8-openshift","openshift/jboss-eap74-openjdk8-runtime-openshift","openshift/jboss-fuse70-console","openshift/jboss-fuse70-eap-openshift","openshift/jboss-fuse70-java-openshift","openshift/jboss-fuse70-karaf-openshift","openshift/jboss-processserver64-openshift","openshift/jboss-webserver31-tomcat7-openshift","openshift/jboss-webserver31-tomcat8-openshift","openshift/jboss-webserver56-openjdk11-tomcat9-openshift-ubi8","openshift/jboss-webserver56-openjdk8-tomcat9-openshift-ubi8","openshift/jenkins","openshift/jenkins-agent-base","openshift/jenkins-agent-maven","openshift/jenkins-agent-nodejs","openshift/mariadb","openshift/must-gather","openshift/mysql","openshift/nginx","openshift/nodejs","openshift/oauth-proxy","openshift/openjdk-11-rhel7","openshift/openjdk-11-rhel8","openshift/perl","openshift/php","openshift/postgresql","openshift/python","openshift/redhat-openjdk18-openshift","openshift/redhat-sso70-openshift","openshift/redhat-sso71-openshift","openshift/redhat-sso72-openshift","openshift/redhat-sso73-openshift","openshift/redis","openshift/rhdm-decisioncentral-rhel8","openshift/rhdm-kieserver-rhel8","openshift/rhpam-businesscentral-monitoring-rhel8","openshift/rhpam-businesscentral-rhel8","openshift/rhpam-kieserver-rhel8","openshift/rhpam-smartrouter-rhel8","openshift/ruby","openshift/sso74-openshift-rhel8","openshift/sso75-openshift-rhel8","openshift/tests","openshift/tools","openshift/ubi8-openjdk-11","openshift/ubi8-openjdk-11-runtime","openshift/ubi8-openjdk-8","openshift/ubi8-openjdk-8-runtime"]}
* Connection #0 to host default-route-openshift-image-registry.apps-crc.testing left intact
ayates83 commented 2 years ago

I also tried forwarding port 5000 from the internal registry pod to my localhost and pushing there, but failed as well:

 $ podman push localhost/ocpbackup 127.0.0.1:5000/openshift/ocpbackup
Error: error pushing image "127.0.0.1:5000/openshift/ocpbackup": trying to reuse blob sha256:5bc03dec623975972bc748cb1da06ce8b4cf854d38c854a9e5f5bcad48caf6cc at destination: pinging container registry 127.0.0.1:5000: Get "https://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused
praveenkumar commented 2 years ago

I also tried forwarding port 5000 from the internal registry pod to my localhost and pushing there, but failed as well:

 $ podman push localhost/ocpbackup 127.0.0.1:5000/openshift/ocpbackup
Error: error pushing image "127.0.0.1:5000/openshift/ocpbackup": trying to reuse blob sha256:5bc03dec623975972bc748cb1da06ce8b4cf854d38c854a9e5f5bcad48caf6cc at destination: pinging container registry 127.0.0.1:5000: Get "https://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused

@ayates83 127.0.0.1:5000 is not going to work because 5000 port is only internally available not externally. Also are you using podman-machine for podman which is running parallel to crc?

ayates83 commented 2 years ago

I'm running the podman that's included with crc:

 $ which podman
/Users/$user/.crc/bin/oc/podman

$ podman --version
podman version 3.2.3

I removed podman provided by brew (was using it previously.) But it looks like my podman is pointing to the one included with crc.

praveenkumar commented 2 years ago

I'm running the podman that's included with crc:

 $ which podman
/Users/$user/.crc/bin/oc/podman

$ podman --version
podman version 3.2.3

I removed podman provided by brew (was using it previously.) But it looks like my podman is pointing to the one included with crc.

Do you have docker desktop installed in your system?

ayates83 commented 2 years ago

I did at one point but deleted it.

praveenkumar commented 2 years ago

@ayates83 I don't know why it is happening but one last debug you can do is ssh to the VM and then try to push the image to registry and see if that works. https://github.com/code-ready/crc/wiki/Debugging-guide have way to ssh to the VM.

ayates83 commented 2 years ago

It works when I ssh into the CRC VM and do podman login / podman push. Strange behavior. Any other debug info I can provide to you?

praveenkumar commented 2 years ago

If that works then it might be some left over on docker desktop side because when it works inside the crc VM the only thing we are eliminate is host <=> vm networking which podman is using to connect internal registry. I will suggest meanwhile you carry on this workaround and we will try to reproduce it on our end if possible to see what's causing that.

ayates83 commented 2 years ago

ok thanks very much :)

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

cg2p commented 1 year ago

I have the same issue. I have a podman install outside of crc as per https://github.com/crc-org/crc/issues/3246#issuecomment-1203549639 and I can curl the v2 endpoint as well.

Looking around there was a suggestion that it might be to do with gvproxy - https://podman-desktop.io/docs/troubleshooting#unable-to-set-custom-binary-path-for-podman-on-macos - but that didnt' work either.

Any more thoughts on best practice here?

e.g. only ever podman installed with crc?

kyocoolcool commented 1 year ago

I have the same issue, podman push default-route-openshift-image-registry.apps-crc.testing/demo/alpine:latest --tls-verify=false

result: Error: trying to reuse blob sha256:5e0d8111135538b8a86ce5fc969849efce16c455fd016bb3dc53131bcedc4da5 at destination: pinging container registry default-route-openshift-image-registry.apps-crc.testing: Get "http://default-route-openshift-image-registry.apps-crc.testing/v2/": dial tcp 127.0.0.1:80: connect: connection refused

NB: Apple M1 Pro CRC version: 2.27.0+71615e OpenShift version: 4.13.12 Podman version: 4.4.4

praveenkumar commented 1 year ago

@kyocoolcool can you create a new issue using template and provide all the info there for us to figure out what is wrong?

Hai-D commented 7 months ago

is fix it ? I have same issue on MacOS