search

eclipse-jkube / jkube

Build and Deploy java applications on Kubernetes
https://www.eclipse.dev/jkube/
Eclipse Public License 2.0
733 stars 481 forks source link

failure when running k8s:push to custom container push registry when using podman #796

Open itewk opened 3 years ago

itewk commented 3 years ago

Description

When trying to execute k8s:push on quickstart apps with a custom container push registry it fails.

When trying this with a host with docker it works fine.

Reproducing steps - jkube.generator.name not specified

.commands

# setup jkube to use podman instead of docker
export DOCKER_HOST="unix:/home/$(whoami)/podman.sock"
podman system service --time=0 ${DOCKER_HOST} 1> /home/$(whoami)/podman.stdout 2> /home/$(whoami/podman.stderr &

# get project
git clone https://github.com/eclipse/jkube
cd jkube/quickstarts/maven/spring-boot-helm

# build image (pass)
mvn package k8s:build -Pkubernetes
podman image list -a | grep spring-boot-helm

# push image to internal registry (fail)
mvn k8s:push -Pkubernetes -Djkube.docker.push.registry=registry-internal-nexus.apps.tssc.rht-set.com -Djkube.docker.username=xxxx -Djkube.docker.password=xxxx

.mvn k8s:push - output

[INFO] Scanning for projects...
[INFO]
[INFO] --------< org.eclipse.jkube.quickstarts.maven:spring-boot-helm >--------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot - Helm 1.4.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:push (default-cli) @ spring-boot-helm ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc: Name or service not known
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[ERROR] k8s: Error while trying to push the image: Unable to push 'maven/spring-boot-helm:1.4.0' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/maven/spring-boot-helm:1.4.0: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/maven/spring-boot-helm:1.4.0: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500) [Error while trying to push the image: Unable to push 'maven/spring-boot-helm:1.4.0' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/maven/spring-boot-helm:1.4.0: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/maven/spring-boot-helm:1.4.0: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.976 s
[INFO] Finished at: 2021-07-28T18:36:09Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.4.0:push (default-cli) on project spring-boot-helm: Error while trying to push the image: Unable to push 'maven/spring-boot-helm:1.4.0' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/maven/spring-boot-helm:1.4.0: destination image name must be specified: invalid argument","response":500}
[ERROR] {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/maven/spring-boot-helm:1.4.0: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Reproducing steps - jkube.generator.name specified with short name

.commands

# setup jkube to use podman instead of docker
export DOCKER_HOST="unix:/home/$(whoami)/podman.sock"
podman system service --time=0 ${DOCKER_HOST} 1> /home/$(whoami)/podman.stdout 2> /home/$(whoami/podman.stderr &

# get project
git clone https://github.com/eclipse/jkube
cd jkube/quickstarts/maven/spring-boot-helm

# build image (pass)
mvn package k8s:build -Pkubernetes -Djkube.generator.name=jkube-test/spring-boot-helm-example
podman image list -a | grep jkube-test/spring-boot-helm-example

# push image to internal registry (fail)
mvn k8s:push -Pkubernetes -Djkube.docker.push.registry=registry-internal-nexus.apps.tssc.rht-set.com -Djkube.docker.username=xxxx -Djkube.docker.password=xxxx -Djkube.generator.name=jkube-test/spring-boot-helm-example

.mvn k8s:push - output

[INFO] Scanning for projects...
[INFO]
[INFO] --------< org.eclipse.jkube.quickstarts.maven:spring-boot-helm >--------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot - Helm 1.4.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:push (default-cli) @ spring-boot-helm ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc: Name or service not known
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[ERROR] k8s: Error while trying to push the image: Unable to push 'jkube-test/spring-boot-helm-example' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/jkube-test/spring-boot-helm-example:latest: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/jkube-test/spring-boot-helm-example:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500) [Error while trying to push the image: Unable to push 'jkube-test/spring-boot-helm-example' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/jkube-test/spring-boot-helm-example:latest: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/jkube-test/spring-boot-helm-example:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.001 s
[INFO] Finished at: 2021-07-28T18:41:07Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.4.0:push (default-cli) on project spring-boot-helm: Error while trying to push the image: Unable to push 'jkube-test/spring-boot-helm-example' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/jkube-test/spring-boot-helm-example:latest: destination image name must be specified: invalid argument","response":500}
[ERROR] {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/jkube-test/spring-boot-helm-example:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Reproducing steps - jkube.generator.name specified with name including repository

.commands

# setup jkube to use podman instead of docker
export DOCKER_HOST="unix:/home/$(whoami)/podman.sock"
podman system service --time=0 ${DOCKER_HOST} 1> /home/$(whoami)/podman.stdout 2> /home/$(whoami/podman.stderr &

# get project
git clone https://github.com/eclipse/jkube
cd jkube/quickstarts/maven/spring-boot-helm

# build image (pass)
mvn package k8s:build -Pkubernetes -Djkube.generator.name=registry-internal-nexus.apps.tssc.rht-set.com/jkube-test/spring-boot-helm-example
podman image list -a | grep jkube-test/spring-boot-helm-example

# push image to internal registry (fail)
mvn k8s:push -Pkubernetes -Djkube.docker.push.registry=registry-internal-nexus.apps.tssc.rht-set.com -Djkube.docker.username=xxxx -Djkube.docker.password=xxxx -Djkube.generator.name=registry-internal-nexus.apps.tssc.rht-set.com/jkube-test/spring-boot-helm-example

.mvn k8s:push - output

[INFO] Scanning for projects...
[INFO]
[INFO] --------< org.eclipse.jkube.quickstarts.maven:spring-boot-helm >--------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot - Helm 1.4.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:push (default-cli) @ spring-boot-helm ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc: Name or service not known
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[ERROR] k8s: Error while trying to push the image: Unable to push 'registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot:latest: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500) [Error while trying to push the image: Unable to push 'registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot:latest: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.728 s
[INFO] Finished at: 2021-07-28T19:33:38Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.4.0:push (default-cli) on project spring-boot-helm: Error while trying to push the image: Unable to push 'registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot' to registry 'registry-internal-nexus.apps.tssc.rht-set.com' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot:latest: destination image name must be specified: invalid argument","response":500}
[ERROR] {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/ploigos-ref/spring-boot:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Info

itewk commented 3 years ago

one difference i have found is that when using podman if i list the images then the built image name is localhost/GENERATED_IMAGE_NAME but if doing docker and listing the built images then the built image name is simply GENERATED_IMAGE_NAME i wonder if that localhost is messing it up because then tag for the push from location isn't lining up. gonna investigate more.

itewk commented 3 years ago

one difference i have found is that when using podman if i list the images then the built image name is localhost/GENERATED_IMAGE_NAME but if doing docker and listing the built images then the built image name is simply GENERATED_IMAGE_NAME i wonder if that localhost is messing it up because then tag for the push from location isn't lining up. gonna investigate more.

found more informaiton on this. podman adds localhost/ where as docker adds docker.io by default, but i guess the docker image list -a command filters out the docker.io part vs podman does not filter out the localhost. Still not sure if this is root of problem though...just a difference i see.

itewk commented 3 years ago

updated description with another attempt trying to specify a full name rather then short name with the generater name. no dice.

itewk commented 3 years ago

i have noticed each time i try to push i get the following in my podman service log

http: superfluous response.WriteHeader call from github.com/containers/podman/pkg/api/handlers/utils.WriteJSON (handler.go:195)
manusa commented 3 years ago

Are you able to push that image using Podman's CLI?

I'm doing the following, which works both with jib and docker using Minikube's registry:

# Enable Minikube registry
$ minikube addons enable registry
# Enable local access to Minikube's registry
$ docker run --rm -e REMOTE_HOST=$(minikube ip) -e REMOTE_PORT=5000 -e LOCAL_PORT=5000 -p 5000:5000 marcnuri/port-forward

With standard Docker:

$ mvn -Pkubernetes -Djkube.generator.name="localhost:5000/test" clean package k8s:build k8s:push
[INFO] k8s: Pushed localhost:5000/test in 726 milliseconds
$ curl http://localhost:5000/v2/test/tags/list
{"name":"test","tags":["latest"]}

With standard Jib:

$ mvn -Pkubernetes -Djkube.generator.name="localhost:5000/test" -Djkube.build.strategy=jib clean package k8s:build k8s:push
JIB> Pushing manifest for latest...                                                                                          
JIB> [===========================   ] 90.9% complete > launching manifest list pushers
JIB> [==============================] 100.0% complete
$ curl http://localhost:5000/v2/test/tags/list
{"name":"test","tags":["latest"]}

With Podman:

$ export DOCKER_HOST="unix:////$XDG_RUNTIME_DIR/podman/podman.sock"
$ mvn -Pkubernetes -Djkube.generator.name="localhost:5000/test" clean package k8s:build k8s:push
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.4.0:push (default-cli) on project spring-boot-helm: Error while trying to push the image: Unable to push 'localhost:5000/test' to registry 'localhost:5000' : {"cause":"Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client","message":"error pushing image \"localhost:5000/test:latest\": error copying image to the remote destination: Error trying to reuse blob sha256:3aa55ff7bca11fe107fcefec151021377ce49f4bacb40262a416b88bc34aeb64 at destination: error pinging docker registry localhost:5000: Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client","response":400} (Bad Request: 400) -> [Help 1]

But also:

$ podman images
REPOSITORY                           TAG     IMAGE ID      CREATED         SIZE
localhost:5000/test                  latest  132d07695cd5  40 seconds ago  541 MB
quay.io/jkube/jkube-java-binary-s2i  0.0.9   910caf82544b  6 months ago    522 MB
$ podman push localhost:5000/test
Getting image source signatures
Error: error copying image to the remote destination: Error trying to reuse blob sha256:3aa55ff7bca11fe107fcefec151021377ce49f4bacb40262a416b88bc34aeb64 at destination: error pinging docker registry localhost:5000: Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client

Is it possible that the issue is in Podman?

itewk commented 3 years ago

@manusa i can push with podman without issue:

> podman tag maven/spring-boot-helm:1.4.0 registry-internal-nexus.apps.tssc.rht-set.com/maven/spring-boot-helm:1.4.0
> podman push registry-internal-nexus.apps.tssc.rht-set.com/maven/spring-boot-helm:1.4.0
Getting image source signatures
Copying blob da99f938f663 done
Copying blob 00af10937683 done
Copying blob 3aa55ff7bca1 done
Copying blob e9fac1fd05df done
Copying config 76d64ea036 done
Writing manifest to image destination
Storing signatures
manusa commented 3 years ago

OK, I went too quick and didn't check the error about the insecure registry :facepalm:

With my reproducer steps it works OK:

$ export DOCKER_HOST="unix:////$XDG_RUNTIME_DIR/podman/podman.sock"
$ mvn -Pkubernetes -Djkube.generator.name="localhost:5000/test" clean package k8s:build k8s:push
[INFO] k8s: Pushed localhost:5000/test in 6 seconds 
$ podman images
REPOSITORY                           TAG     IMAGE ID      CREATED         SIZE
localhost:5000/test                  latest  3d4be2a0c3e2  21 seconds ago  541 MB
quay.io/jkube/jkube-java-binary-s2i  0.0.9   910caf82544b  6 months ago    522 MB
$ curl http://localhost:5000/v2/test/tags/list
{"name":"test","tags":["latest"]}

I'm using podman 2.2.1

itewk commented 3 years ago

@manusa im on podman 3.0.2-dev which is what installs on RHEL 8. maybe thats the diff?

13:25 $ mvn -Pkubernetes -Djkube.generator.name="registry-internal-nexus.apps.tssc.rht-set.com/test" clean package k8s:build k8s:push -Djkube.docker.username=xxxx -Djkube.docker.password=xxxx
[INFO] Scanning for projects...
[INFO]
[INFO] --------< org.eclipse.jkube.quickstarts.maven:spring-boot-helm >--------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot - Helm 1.4.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ spring-boot-helm ---
[INFO] Deleting /home/itewk/jkube/quickstarts/maven/spring-boot-helm/target
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ spring-boot-helm ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/itewk/jkube/quickstarts/maven/spring-boot-helm/src/main/resources
[INFO] skip non existing resourceDirectory /home/itewk/jkube/quickstarts/maven/spring-boot-helm/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ spring-boot-helm ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/itewk/jkube/quickstarts/maven/spring-boot-helm/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ spring-boot-helm ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/itewk/jkube/quickstarts/maven/spring-boot-helm/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ spring-boot-helm ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ spring-boot-helm ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ spring-boot-helm ---
[INFO] Building jar: /home/itewk/jkube/quickstarts/maven/spring-boot-helm/target/spring-boot-helm-1.4.0.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.2.7.RELEASE:repackage (repackage) @ spring-boot-helm ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:resource (default) @ spring-boot-helm ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc: Name or service not known
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[INFO] k8s: Using resource templates from /home/itewk/jkube/quickstarts/maven/spring-boot-helm/src/main/jkube
[INFO] k8s: jkube-service-discovery: Using first mentioned service port '8080'
[INFO] k8s: jkube-revision-history: Adding revision history limit to 2
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:build (default) @ spring-boot-helm ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[INFO] k8s: [registry-internal-nexus.apps.tssc.rht-set.com/test:latest] "spring-boot": Created docker-build.tar in 505 milliseconds
[INFO] k8s: [registry-internal-nexus.apps.tssc.rht-set.com/test:latest] "spring-boot": Built image sha256:57fd4
[INFO] k8s: [registry-internal-nexus.apps.tssc.rht-set.com/test:latest] "spring-boot": Removed old image sha256:cd95f
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:helm (default) @ spring-boot-helm ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc: Name or service not known
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Creating Helm Chart "spring-boot-helm" for Kubernetes
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:build (default-cli) @ spring-boot-helm ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[INFO] k8s: [registry-internal-nexus.apps.tssc.rht-set.com/test:latest] "spring-boot": Created docker-build.tar in 232 milliseconds
[INFO] k8s: [registry-internal-nexus.apps.tssc.rht-set.com/test:latest] "spring-boot": Built image sha256:8952e
[INFO] k8s: [registry-internal-nexus.apps.tssc.rht-set.com/test:latest] "spring-boot": Removed old image sha256:57fd4
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:push (default-cli) @ spring-boot-helm ---
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc
[WARNING] Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[ERROR] k8s: Error while trying to push the image: Unable to push 'registry-internal-nexus.apps.tssc.rht-set.com/test' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/test:latest: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/test:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500) [Error while trying to push the image: Unable to push 'registry-internal-nexus.apps.tssc.rht-set.com/test' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/test:latest: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/test:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22.778 s
[INFO] Finished at: 2021-07-29T13:25:41Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.4.0:push (default-cli) on project spring-boot-helm: Error while trying to push the image: Unable to push 'registry-internal-nexus.apps.tssc.rht-set.com/test' : {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/test:latest: destination image name must be specified: invalid argument","response":500}
[ERROR] {"cause":"invalid argument","message":"failed to find image registry-internal-nexus.apps.tssc.rht-set.com/test:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
manusa commented 3 years ago

@manusa im on podman 3.0.2-dev

Yes, I read that in the issue description, that's why I was letting you know.

maybe thats the diff?

I'm not sure, I don't have a Nexus repository available. I think that it might either be an issue with the Podman version or the Nexus Docker registry.

Could you please try my steps with Minikube + local port forward, to see if that works for you in your environment?

itewk commented 3 years ago

@manusa same issue. sooo....podman version?

setup

✘-55 ~/jkube/quickstarts/maven/spring-boot-helm [master ↓·1|✚ 1]
13:34 $ minikube start
😄  minikube v1.22.0 on Redhat 8.4 (xen/amd64)
✨  Automatically selected the podman driver
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.21.2 preload ...
    > preloaded-images-k8s-v11-v1...: 502.14 MiB / 502.14 MiB  100.00% 66.11 Mi
    > gcr.io/k8s-minikube/kicbase...: 361.08 MiB / 361.09 MiB  100.00% 29.75 Mi
E0729 13:35:49.295442  343846 cache.go:200] Error downloading kic artifacts:  not yet implemented, see issue #8426
🔥  Creating podman container (CPUs=2, Memory=2200MB) ...
🐳  Preparing Kubernetes v1.21.2 on Docker 20.10.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
✔ ~/jkube/quickstarts/maven/spring-boot-helm [master ↓·1|✚ 1]
13:37 $ minikube addons enable registry
    ▪ Using image registry:2.7.1
    ▪ Using image gcr.io/google_containers/kube-registry-proxy:0.4
🔎  Verifying registry addon...
🌟  The 'registry' addon is enabled
✔ ~/jkube/quickstarts/maven/spring-boot-helm [master ↓·1|✚ 1]
13:37 $ podman run --rm -e REMOTE_HOST=$(minikube ip) -e REMOTE_PORT=5000 -e LOCAL_PORT=5000 -p 5000:5000 marcnuri/port-forward
✔ docker.io/marcnuri/port-forward:latest
Trying to pull docker.io/marcnuri/port-forward:latest...
Getting image source signatures
Copying blob 18967800a0c1 done
Copying blob 5a3ea8efae5d done
Copying config 3f907e420c done
Writing manifest to image destination
Storing signatures
Socat started listening on 5000: Redirecting traffic to 192.168.49.2:5000 (6)

attempt

13:40 $ mvn -Pkubernetes -Djkube.generator.name="localhost:5000/test" clean package k8s:build k8s:push
[INFO] Scanning for projects...
[INFO]
[INFO] --------< org.eclipse.jkube.quickstarts.maven:spring-boot-helm >--------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot - Helm 1.4.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ spring-boot-helm ---
[INFO] Deleting /home/itewk/jkube/quickstarts/maven/spring-boot-helm/target
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ spring-boot-helm ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/itewk/jkube/quickstarts/maven/spring-boot-helm/src/main/resources
[INFO] skip non existing resourceDirectory /home/itewk/jkube/quickstarts/maven/spring-boot-helm/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ spring-boot-helm ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/itewk/jkube/quickstarts/maven/spring-boot-helm/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ spring-boot-helm ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/itewk/jkube/quickstarts/maven/spring-boot-helm/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ spring-boot-helm ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ spring-boot-helm ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ spring-boot-helm ---
[INFO] Building jar: /home/itewk/jkube/quickstarts/maven/spring-boot-helm/target/spring-boot-helm-1.4.0.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.2.7.RELEASE:repackage (repackage) @ spring-boot-helm ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:resource (default) @ spring-boot-helm ---
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[INFO] k8s: Using resource templates from /home/itewk/jkube/quickstarts/maven/spring-boot-helm/src/main/jkube
[INFO] k8s: jkube-service-discovery: Using first mentioned service port '8080'
[INFO] k8s: jkube-revision-history: Adding revision history limit to 2
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:build (default) @ spring-boot-helm ---
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[INFO] k8s: [localhost:5000/test:latest] "spring-boot": Created docker-build.tar in 197 milliseconds
[INFO] k8s: [localhost:5000/test:latest] "spring-boot": Built image sha256:cc543
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:helm (default) @ spring-boot-helm ---
[INFO] k8s: Creating Helm Chart "spring-boot-helm" for Kubernetes
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:build (default-cli) @ spring-boot-helm ---
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[INFO] k8s: [localhost:5000/test:latest] "spring-boot": Created docker-build.tar in 228 milliseconds
[INFO] k8s: [localhost:5000/test:latest] "spring-boot": Built image sha256:2fa44
[INFO] k8s: [localhost:5000/test:latest] "spring-boot": Removed old image sha256:cc543
[INFO]
[INFO] --- kubernetes-maven-plugin:1.4.0:push (default-cli) @ spring-boot-helm ---
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[ERROR] k8s: Error while trying to push the image: Unable to push 'localhost:5000/test' to registry 'localhost:5000' : {"cause":"invalid argument","message":"failed to find image localhost:5000/test:latest: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image localhost:5000/test:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500) [Error while trying to push the image: Unable to push 'localhost:5000/test' to registry 'localhost:5000' : {"cause":"invalid argument","message":"failed to find image localhost:5000/test:latest: destination image name must be specified: invalid argument","response":500}
{"cause":"invalid argument","message":"failed to find image localhost:5000/test:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 27.164 s
[INFO] Finished at: 2021-07-29T13:41:28Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.4.0:push (default-cli) on project spring-boot-helm: Error while trying to push the image: Unable to push 'localhost:5000/test' to registry 'localhost:5000' : {"cause":"invalid argument","message":"failed to find image localhost:5000/test:latest: destination image name must be specified: invalid argument","response":500}
[ERROR] {"cause":"invalid argument","message":"failed to find image localhost:5000/test:latest: destination image name must be specified: invalid argument","response":404} (Internal Server Error: 500)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
itewk commented 3 years ago

i also verified each time i attempt to push with jkube, whether with the minikube local registry or nexus the following gets logged to my podman socket error log:

http: superfluous response.WriteHeader call from github.com/containers/podman/pkg/api/handlers/utils.WriteJSON (handler.go:195)

so my guess, something different with the podman socket API or something?

manusa commented 3 years ago

so my guess, something different with the podman socket API or something?

I really don't know.

@rohanKanojia I think you might have podman 3.x in your system, right? Could you try to reproduce with my steps, see if it fails for you too?

rohanKanojia commented 3 years ago

I tried this with podman version 3.2.3. I'm able to push to my quay.io private registry with following steps:

  1. podman system service --time=0 ${DOCKER_HOST} 1> /home/$(whoami)/podman.stdout 2> /home/$(whoami)/podman.stderr
  2. Set DOCKER_HOST and run JKube goals in quickstart directory: 
    spring-boot-helm : $ echo $DOCKER_HOST
unix:/home/rokumar/podman.sock
spring-boot-helm : $ mvn -Pkubernetes k8s:build  -Djkube.generator.name="quay.io/rohankanojia/%a:%l"
[INFO] Scanning for projects...
[INFO] 
[INFO] --------< org.eclipse.jkube.quickstarts.maven:spring-boot-helm >--------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot - Helm 1.4.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.4.0:build (default-cli) @ spring-boot-helm ---
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc: Name or service not known
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[INFO] k8s: [quay.io/rohankanojia/spring-boot-helm:1.4.0] "spring-boot": Created docker-build.tar in 169 milliseconds
[INFO] k8s: [quay.io/rohankanojia/spring-boot-helm:1.4.0] "spring-boot": Built image sha256:12804
[INFO] k8s: [quay.io/rohankanojia/spring-boot-helm:1.4.0] "spring-boot": Removed old image sha256:2b4f6
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.377 s
[INFO] Finished at: 2021-07-29T20:19:34+05:30
[INFO] ------------------------------------------------------------------------
spring-boot-helm : $ mvn -Pkubernetes k8s:push  -Djkube.generator.name="quay.io/rohankanojia/%a:%l"
[INFO] Scanning for projects...
[INFO] 
[INFO] --------< org.eclipse.jkube.quickstarts.maven:spring-boot-helm >--------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot - Helm 1.4.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.4.0:push (default-cli) @ spring-boot-helm ---
[WARNING] k8s: Cannot access cluster for detecting mode: Unknown host kubernetes.default.svc: Name or service not known
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image quay.io/jkube/jkube-java-binary-s2i:0.0.9 as base / builder
[INFO] k8s: Pushed quay.io/rohankanojia/spring-boot-helm:1.4.0 in 36 seconds 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  38.701 s
[INFO] Finished at: 2021-07-29T20:20:49+05:30
[INFO] ------------------------------------------------------------------------

    Screenshot from 2021-07-29 20-20-59

itewk commented 3 years ago

so maybe a problem with 3.0.2-dev (which is what ships with RHEL 8)? I can try and figure out how to compile podman 3.2.3 and try that, then i guess if that works, open bug with rhel folks to ship a newer podman. its gonna be quite a rabbit whole to try and compile podman 3.2.3 for rhel and i can work around this by just doing a podman push vs using the jkube push, so not going to be high on my priorties to try.

manusa commented 3 years ago

I still find it very odd, I'm not really sure that the issue is with Podman itself.

Anyway, the good thing is that you can work around the issue. If we find time we'll try to reproduce with an environment equivalent to yours too.