porter publish fails with local docker image registry

sajuptpm commented 4 years ago

Describe the bug

Getting following error when i run "porter publish --debug"

Error: Starting to copy image myregistry1.com/porter-hello2-installer:0.1.0... Failed to copy image myregistry1.com/porter-hello2-installer:0.1.0: failed to resolve or push image for service "InvocationImage" Error: failed to resolve or push image for service "InvocationImage"

as per cnab-to-oci code, it logging actual error, but I could not find that log. Does porter not exposing it ? https://github.com/cnabio/cnab-to-oci/blob/master/remotes/fixup.go#L217
"porter publish --debug" successfully pushed CNAB invocation image to local docker registry. after that it trying to do above copy image operation and fails

Slack Post: https://cloud-native.slack.com/archives/CN8NA4F8V/p1589291511250700

To Reproduce

Steps to reproduce the behavior:

Create a local docker registry, tried with and with out ssl(nginx)
run $ porter build
Run $ porter publish
Error: Failed to copy image myregistry1.com/porter-hello2-installer:0.1.0: failed to resolve or push

Expected behavior

dont know

Porter Command and Output

$ porter publish --debug

 DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
**Pushing CNAB invocation image...**
The push refers to repository [myregistry1.com/porter-hello2-installer]
635de8915901: Preparing
f4c144b3558e: Preparing
4d28493bbaca: Preparing
f8f7a18a990e: Preparing
03d00834c9a2: Preparing
86ab2c6c5d58: Preparing
86ab2c6c5d58: Waiting
4d28493bbaca: Layer already exists
635de8915901: Layer already exists
03d00834c9a2: Pushed
f4c144b3558e: Pushed
f8f7a18a990e: Pushed
86ab2c6c5d58: Pushed
**0.1.0: digest: sha256:d1b2f84bd9aef3125a5a04b3832dcbbdcfd98a15140fc06d1494d9dcc44acd2e size: 1579**

Rewriting CNAB bundle.json...
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
**Starting to copy image myregistry1.com/porter-hello2-installer:0.1.0...
Failed to copy image myregistry1.com/porter-hello2-installer:0.1.0: failed to resolve or push image for service "InvocationImage"
Error: failed to resolve or push image for service "InvocationImage"**

Version

porter v0.26.2-beta.1 (e4aa6cd3)

vdice commented 4 years ago

Thank you for filing this issue @sajuptpm .

Indeed, I was able to reproduce the error when running a stock local registry.

As you've pointed out, the actual error is being swallowed in the cnab-to-oci library Porter uses. I think it will be worth creating an issue/inquiry as to why the library isn't explicitly returning an error if one occurs during any of the "fixups".

In the meantime, I created a branch on my fork of cnab-to-oci to do just that: explicitly error out. The diff can be seen via https://github.com/cnabio/cnab-to-oci/compare/master...vdice:fix/fixup-return-err

I created a corresponding branch on my fork of porter to use that commit: https://github.com/vdice/porter/tree/bug/1032

My registry was initially deployed insecurely, so the first result was:

$ porter publish
...
Rewriting CNAB bundle.json...
Starting to copy image vdice.registry/porter-file-param-installer:0.1.0...
Failed to copy image vdice.registry/porter-file-param-installer:0.1.0: failed to resolve or push image for service "InvocationImage": failed to do request: Head https://vdice.registry/v2/porter-file-param-installer/manifests/0.1.0: dial tcp 127.0.0.1:443: connect: connection refused
Error: failed to resolve or push image for service "InvocationImage": failed to do request: Head https://vdice.registry/v2/porter-file-param-installer/manifests/0.1.0: dial tcp 127.0.0.1:443: connect: connection refused

I'm still not sure if this is due to copying requiring TLS or if an insecure registry needs add'l flags/config to redirect properly. In any case, I redeployed the registry w/ TLS and next saw:

$ porter publish
...
Rewriting CNAB bundle.json...
Starting to copy image vdice.registry/porter-file-param-installer:0.1.0...
Failed to copy image vdice.registry/porter-file-param-installer:0.1.0: failed to resolve or push image "vdice.registry/porter-file-param-installer:0.1.0": failed to do request: Head https://vdice.registry/v2/porter-file-param-installer/manifests/0.1.0: x509: certificate signed by unknown authority
Error: failed to resolve or push image "vdice.registry/porter-file-param-installer:0.1.0": failed to do request: Head https://vdice.registry/v2/porter-file-param-installer/manifests/0.1.0: x509: certificate signed by unknown authority

Ok, so I need to properly create a CA, etc. However, before doing so, I wanted to check in and see if it would be convenient for you to build the porter CLI from my fork/branch mentioned above, to see what error(s) you may be encountering. Hopefully we'll get some extra information now.

sajuptpm commented 4 years ago

Thanks @vdice with your patch, I am getting actual error message.

I think, we are closer

# porter publish --debug

/root/.porter/mixins/terraform/terraform version --output json --debug
Starting to copy image myregistry1.com/porter-hello2-installer:0.1.0...
Failed to copy image myregistry1.com/porter-hello2-installer:0.1.0: failed to resolve or push image "myregistry1.com/porter-hello2-installer:0.1.0": failed to do request: Head https://myregistry1.com/v2/porter-hello2-installer/manifests/0.1.0: unexpected EOF
Error: failed to resolve or push image "myregistry1.com/porter-hello2-installer:0.1.0": failed to do request: Head https://myregistry1.com/v2/porter-hello2-installer/manifests/0.1.0: unexpected EOF

I am behind a proxy, curl to manifests is working when I set no_proxy=myregistry1.com Is it proxy issue ?

$ curl https://myregistry1.com/v2/porter-hello2-installer/manifests/0.1.0
{
   "schemaVersion": 1,
   "name": "porter-hello2-installer",
   "tag": "0.1.0",
   "architecture": "amd64",
   "fsLayers": [

$ curl -X HEAD https://myregistry1.com/v2/porter-hello2-installer/manifests/0.1.0

curl: (18) transfer closed with 8501 bytes remaining to read

sajuptpm commented 4 years ago

unexpected EOF was proxy issue

Fix:

export no_proxy=myregistry1.com

then run

$ porter publish --debug                 

DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Pushing CNAB invocation image...
The push refers to repository [myregistry1.com/porter-hello2-installer]
98c9f7522a52: Preparing
a4985c8da5ef: Preparing
24de638c4b50: Preparing
584c9ff67b08: Preparing
e8db0f22894f: Preparing
86ab2c6c5d58: Preparing
86ab2c6c5d58: Waiting
24de638c4b50: Layer already exists
e8db0f22894f: Layer already exists
a4985c8da5ef: Layer already exists
584c9ff67b08: Layer already exists
98c9f7522a52: Layer already exists
86ab2c6c5d58: Layer already exists
0.1.0: digest: sha256:388a4d02dd28481ee1a37a5e80c590debad05a7d2ad59c695141b48920b4c6ab size: 1579

Rewriting CNAB bundle.json...
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Starting to copy image myregistry1.com/porter-hello2-installer:0.1.0...
Completed image myregistry1.com/porter-hello2-installer:0.1.0 copy <--------------------
Bundle tag myregistry1.com/porter-hello2:v0.1.2 pushed successfully, with digest "sha256:28428d2916691d498f008cede9f024c73f62ac45fa314e1ec7330ff73f31cdef"

sajuptpm commented 4 years ago

Sharing some information I hope this will help others :)

Command to create docker registry

docker run -dp 5000:5000 --restart=always --name registry registry

Nginx config

server {
    listen       80;
    server_name  myregistry1.com;
    client_max_body_size 300M;

location / {
        proxy_pass http://127.0.0.1:5000;
    }
}

server {
    listen       443 ssl;
    server_name  myregistry1.com;
    ssl_certificate       /root/mkcert/myregistry1.com.pem;
    ssl_certificate_key   /root/mkcert/myregistry1.com-key.pem;
    client_max_body_size 300M;

location / {
        proxy_pass http://127.0.0.1:5000;

    }
}

vdice commented 4 years ago

Ah, very interesting about use of the proxy! Thank you digging in and finding the root issue, as well as supplying notes on setup for others.

I'll follow-up with a question/issue in cnab-to-oci re: the error handling as mentioned above.

In the meantime, I'll close this ticket as resolved.

jvinaya commented 4 years ago

Hi, what is the resolution for the above issue? I'm getting the same error in the latest version of porter. But the issue is being marked as closed

carolynvs commented 4 years ago

@jvinaya The solution for the OP was here: https://github.com/deislabs/porter/issues/1032#issuecomment-627977391. They used the NO_PROXY environment variable to allow docker to push directly to their registry, bypassing the proxy.

getporter / porter