The value of `PLUGIN_PUBLISH_REPOSITORY` is not getting used in `make plugin-publish-packages`

praveenrewar commented 1 year ago

Bug description I am trying to follow the plugin implementation guide to build and publish a plugin.

I have setup a local registry using minikube and set the value of PLUGIN_PUBLISH_REPOSITORY to localhost:8080/test/v1/tanzu-cli/plugins

When I try to publish the plugin using make plugin-publish-packages I get the following error.

Error: unable to push package to temporary registry for plugin: hello-world, target: kubernetes, os: darwin, arch: amd64, version: v0.0.1: output: imgpkg: Error: Writing 'localhost:5001/plugins/darwin/amd64/hello-world:v0.0.1':
  Error while preparing a transport to talk with the registry:
    Unable to create round tripper:
      Get "https://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused; Get "http://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused
: exit status 1
2023-04-13T12:30:54+05:30 [x] : unable to push package to temporary registry for plugin: hello-world, target: kubernetes, os: darwin, arch: amd64, version: v0.0.1: output: imgpkg: Error: Writing 'localhost:5001/plugins/darwin/amd64/hello-world:v0.0.1':
  Error while preparing a transport to talk with the registry:
    Unable to create round tripper:
      Get "https://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused; Get "http://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused
: exit status 1
make: *** [plugin-build-packages] Error 1

Expected behavior The value from the env variable should have been used to publish the plugin.

Steps to reproduce the bug / Relevant debug output I am following the plugin implementation guide and used the following commands to setup a registry.

minikube addons enable registry
kubectl port-forward --namespace kube-system service/registry 8080:80
export PLUGIN_PUBLISH_REPOSITORY=localhost:8080/test/v1/tanzu-cli/plugins

Output of tanzu version

version: v0.90.0-alpha.0
buildDate: 2023-04-08
sha: 198b3730

Environment where the bug was observed (cloud, OS, etc)

marckhouzam commented 1 year ago

Thanks for the report @praveenrewar. I am not able to reproduce the problem myself. However, if I set the PLUGIN_PUBLISH_REPOSITORY to a wrong value, then I do see the same error as you.

Can you confirm that you set that variable properly?

praveenrewar commented 1 year ago

Thank you for the response @marckhouzam! I doubled checked it, and the value of the environment variable is still set.

$ export PLUGIN_PUBLISH_REPOSITORY=localhost:8080/test/v1/tanzu-cli/plugins
$ echo $PLUGIN_PUBLISH_REPOSITORY
localhost:8080/test/v1/tanzu-cli/plugins
$ curl http://localhost:8080/v2/_catalog
{"repositories":["plugins/darwin/amd64/hello-world","plugins/darwin/amd64/package","test/v1/tanzu-cli/plugins/plugin-inventory","test/v1/tanzu-cli/plugins/vmware/tzcli/darwin/amd64/kubernetes/hello-world","test/v1/tanzu-cli/plugins/vmware/tzcli/darwin/amd64/kubernetes/package"]}

After this when I try to publish the plugin, I get the same error.

praveenrewar commented 1 year ago

Here's an interesting observation, if I try to use the 5001 port for my registry

kubectl port-forward --namespace kube-system service/registry 5001:80

then for some reason the value of the environment variable is getting used, but since nothing is running on 8080, it still gives the connection refused error.

Error: unable to publish plugin (name:hello-world, target:kubernetes, os:linux, arch:amd64, version:v0.0.1): output: imgpkg: Error: Error while preparing a transport to talk with the registry:
  Unable to create round tripper:
    Get "https://localhost:8080/v2/": dial tcp [::1]:8080: connect: connection refused; Get "http://localhost:8080/v2/": dial tcp [::1]:8080: connect: connection refused
: exit status 1
2023-04-13T18:27:30+05:30 [x] : unable to publish plugin (name:hello-world, target:kubernetes, os:linux, arch:amd64, version:v0.0.1): output: imgpkg: Error: Error while preparing a transport to talk with the registry:
  Unable to create round tripper:
    Get "https://localhost:8080/v2/": dial tcp [::1]:8080: connect: connection refused; Get "http://localhost:8080/v2/": dial tcp [::1]:8080: connect: connection refused
: exit status 1
make: *** [plugin-publish-packages] Error 1

I might be doing something wrong, but I am not sure what.

marckhouzam commented 1 year ago

Here is how I am trying to reproduce. In shell 1:

$ minikube start
😄  minikube v1.30.1 on Darwin 13.2.1 (arm64)
    ▪ KUBECONFIG=/Users/kmarc/.kube/config.minikube
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                      │
│    Registry addon with docker driver uses port 49825 please use that instead of default port 5000    │
│                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
📘  For more information see: https://minikube.sigs.k8s.io/docs/drivers/docker
    ▪ Using image gcr.io/google_containers/kube-registry-proxy:0.4
    ▪ Using image docker.io/registry:2.8.1
🔎  Verifying registry addon...
🌟  Enabled addons: storage-provisioner, default-storageclass, registry
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
E0413 09:01:03.987633   88097 start.go:289] kubectl info: client semver: Invalid character(s) found in build meta data "$Format:%H$"

$ k config get-contexts
CURRENT   NAME                    CLUSTER                 AUTHINFO                NAMESPACE
*         minikube                minikube                minikube                default

$ minikube addons enable registry
💡  registry is an addon maintained by Google. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                      │
│    Registry addon with docker driver uses port 49825 please use that instead of default port 5000    │
│                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
📘  For more information see: https://minikube.sigs.k8s.io/docs/drivers/docker
    ▪ Using image gcr.io/google_containers/kube-registry-proxy:0.4
    ▪ Using image docker.io/registry:2.8.1
🔎  Verifying registry addon...
🌟  The 'registry' addon is enabled

In shell 2:

kubectl port-forward --namespace kube-system service/registry 8080:80

Back to shell 1:

$ export PLUGIN_PUBLISH_REPOSITORY=localhost:8080/test/v1/tanzu-cli/plugins

$ make plugin-publish-packages
docker stop temp-package-registry && docker rm -v temp-package-registry || true
Error response from daemon: No such container: temp-package-registry
docker run -d -p 5001:5000 --name temp-package-registry mirror.gcr.io/library/registry:2
1bf751505c29aa4c915ef26163d839112654afa92046ebff5153432afb3c3ca3
tanzu builder plugin build-package \
        --binary-artifacts /Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/plugins \
        --package-artifacts /Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/packages \
        --oci-registry localhost:5001
2023-04-13T09:03:11-04:00 [i] Using plugin binary artifacts from "/Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/plugins"
2023-04-13T09:03:11-04:00 [i] Generating plugin package for 'plugin:kubectl' 'target:global' 'os:linux' 'arch:amd64' 'version:v0.0.1'
2023-04-13T09:03:18-04:00 [i] Generated plugin package at "/Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/packages/linux/amd64/global/kubectl/v0.0.1/kubectl-linux_amd64.tar.gz"
2023-04-13T09:03:18-04:00 [i] Generating plugin package for 'plugin:kubectl' 'target:global' 'os:darwin' 'arch:amd64' 'version:v0.0.1'
2023-04-13T09:03:27-04:00 [i] Generated plugin package at "/Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/packages/darwin/amd64/global/kubectl/v0.0.1/kubectl-darwin_amd64.tar.gz"
2023-04-13T09:03:27-04:00 [i] Generating plugin package for 'plugin:kubectl' 'target:global' 'os:windows' 'arch:amd64' 'version:v0.0.1'
2023-04-13T09:03:35-04:00 [i] Generated plugin package at "/Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/packages/windows/amd64/global/kubectl/v0.0.1/kubectl-windows_amd64.tar.gz"
2023-04-13T09:03:35-04:00 [i] Generating plugin package for 'plugin:kubectl' 'target:global' 'os:darwin' 'arch:arm64' 'version:v0.0.1'
2023-04-13T09:03:42-04:00 [i] Generated plugin package at "/Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/packages/darwin/arm64/global/kubectl/v0.0.1/kubectl-darwin_arm64.tar.gz"
2023-04-13T09:03:42-04:00 [i] Saved plugin manifest at "/Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/packages/plugin_manifest.yaml"
tanzu builder plugin publish-package \
        --package-artifacts /Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/packages \
        --publisher tzcli \
        --vendor vmware \
        --repository localhost:8080/test/v1/tanzu-cli/plugins
2023-04-13T09:03:43-04:00 [i] using plugin package artifacts from "/Users/kmarc/git/tanzu-kubectl-plugin-new/artifacts/packages"
2023-04-13T09:03:43-04:00 [i] publishing plugin 'name:kubectl' 'target:global' 'os:linux' 'arch:amd64' 'version:v0.0.1'
2023-04-13T09:03:48-04:00 [i] published plugin at 'localhost:8080/test/v1/tanzu-cli/plugins/vmware/tzcli/linux/amd64/global/kubectl:v0.0.1'
2023-04-13T09:03:48-04:00 [i] publishing plugin 'name:kubectl' 'target:global' 'os:darwin' 'arch:amd64' 'version:v0.0.1'
2023-04-13T09:03:51-04:00 [i] published plugin at 'localhost:8080/test/v1/tanzu-cli/plugins/vmware/tzcli/darwin/amd64/global/kubectl:v0.0.1'
2023-04-13T09:03:51-04:00 [i] publishing plugin 'name:kubectl' 'target:global' 'os:windows' 'arch:amd64' 'version:v0.0.1'
2023-04-13T09:03:55-04:00 [i] published plugin at 'localhost:8080/test/v1/tanzu-cli/plugins/vmware/tzcli/windows/amd64/global/kubectl:v0.0.1'
2023-04-13T09:03:55-04:00 [i] publishing plugin 'name:kubectl' 'target:global' 'os:darwin' 'arch:arm64' 'version:v0.0.1'
2023-04-13T09:03:59-04:00 [i] published plugin at 'localhost:8080/test/v1/tanzu-cli/plugins/vmware/tzcli/darwin/arm64/global/kubectl:v0.0.1'

Do you see something different than in your case?

praveenrewar commented 1 year ago

I have followed similar steps, I even tried deleting my existing minikube cluster. But seeing your output made me realised that the push is happening twice. Once to the temp registry, and then once again to the repository provided by using the environment variable, and for me it's failing in the first step itself. I should have shared the complete output, my bad. I will try to figure out why this is happening but if you have any pointers for me, do let me know.

make plugin-publish-packages
docker stop temp-package-registry && docker rm -v temp-package-registry || true
Error response from daemon: No such container: temp-package-registry
docker run -d -p 5001:5000 --name temp-package-registry mirror.gcr.io/library/registry:2
Unable to find image 'mirror.gcr.io/library/registry:2' locally
2: Pulling from library/registry
91d30c5bc195: Pull complete
65d52c8ad3c4: Pull complete
54f80cd081c9: Pull complete
ca8951d7f653: Pull complete
5ee46e9ce9b6: Pull complete
Digest: sha256:8c51be2f669c82da8015017ff1eae5e5155fcf707ba914c5c7b798fbeb03b50c
Status: Downloaded newer image for mirror.gcr.io/library/registry:2
ad4ff4d66541185a538411b8b13eb829e60a4e71f5c1d2c319416fdd9984b39a
tanzu builder plugin build-package \
        --binary-artifacts /Users/prewar/Workspace/praveenrewar/hello-world-plugin/artifacts/plugins \
        --package-artifacts /Users/prewar/Workspace/praveenrewar/hello-world-plugin/artifacts/packages \
        --oci-registry localhost:5001
2023-04-13T18:44:43+05:30 [i] Using plugin binary artifacts from "/Users/prewar/Workspace/praveenrewar/hello-world-plugin/artifacts/plugins"
2023-04-13T18:44:43+05:30 [i] Generating plugin package for 'plugin:hello-world' 'target:kubernetes' 'os:linux' 'arch:amd64' 'version:v0.0.1'
Error: unable to push package to temporary registry for plugin: hello-world, target: kubernetes, os: linux, arch: amd64, version: v0.0.1: output: imgpkg: Error: Writing 'localhost:5001/plugins/linux/amd64/hello-world:v0.0.1':
  Error while preparing a transport to talk with the registry:
    Unable to create round tripper:
      Get "https://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused; Get "http://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused
: exit status 1
2023-04-13T18:44:43+05:30 [x] : unable to push package to temporary registry for plugin: hello-world, target: kubernetes, os: linux, arch: amd64, version: v0.0.1: output: imgpkg: Error: Writing 'localhost:5001/plugins/linux/amd64/hello-world:v0.0.1':
  Error while preparing a transport to talk with the registry:
    Unable to create round tripper:
      Get "https://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused; Get "http://localhost:5001/v2/": dial tcp [::1]:5001: connect: connection refused
: exit status 1
make: *** [plugin-build-packages] Error 1

marckhouzam commented 1 year ago

It seems we still are tied to the 5001 registry a little but it should not prevent you from using minikube. We'll look into cleaning that better.

Now I see the "Linux" error in your output that you reported in #213 . If you work around that one, do things work?

praveenrewar commented 1 year ago

I think the issue is that I use the docker daemon that comes with minikube by using eval $(minikube docker-env), so even though the port of the temp-registry is exposed, it's still not reachable from the host. If I use the kubectl port forwarding for that as well, then it works.

$ kubectl port-forward --namespace kube-system service/registry 5001:80

$ kubectl port-forward --namespace kube-system service/registry 8080:80

So running the above two commands in different terminals worked for me.

We can close this issue now, but I think that the ideal behaviour should be that when the environment variable is provided, the temp-registry shouldn't be used

vmware-tanzu / tanzu-cli

The value of `PLUGIN_PUBLISH_REPOSITORY` is not getting used in `make plugin-publish-packages` #212