search

eclipse-jkube / jkube

Build and Deploy java applications on Kubernetes
https://www.eclipse.dev/jkube/
Eclipse Public License 2.0
734 stars 479 forks source link

Upgrading from 1.8.0 to 1.9.0 and now failing with Message: Not Found #1747

Closed edeandrea closed 1 year ago

edeandrea commented 1 year ago

Describe the bug

I have a learning tutorial (https://kubebyexample.com/learning-paths/developing-spring-boot-kubernetes/lesson-4-deploying-spring-boot-kubernetes-eclipse) which currently uses JKube 1.8.0. It is a simple Spring Boot application that uses the JKube kubernetes-maven-plugin to deploy to minikube.

The source code is at https://github.com/RedHat-Middleware-Workshops/spring-jkube-external-config/tree/solution.

As part of the repo there are some GitHub actions which verify that things work correctly. Upon upgrading JKube from 1.8.0 to 1.9.0 the k8s:deploy task no longer works. I get this output:

╰─ ./mvnw k8s:deploy -Pk8s -DskipTests
[INFO] Scanning for projects...
[INFO] 
[INFO] ---------------< org.acme:spring-jkube-external-config >----------------
[INFO] Building spring-jkube-external-config 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] >>> kubernetes-maven-plugin:1.9.0:deploy (default-cli) > install @ spring-jkube-external-config >>>
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ spring-jkube-external-config ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.9.0:resource (jkube) @ spring-jkube-external-config ---
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image registry.access.redhat.com/ubi8/openjdk-17:latest as base / builder
[INFO] k8s: Using resource templates from /Users/edeandre/workspaces/workshops/spring-jkube-external-config/src/main/jkube
[INFO] k8s: jkube-controller: Adding a default Deployment
[INFO] k8s: jkube-namespace: Adding a default Namespace: spring-jkube-external-config
[INFO] k8s: jkube-healthcheck-spring-boot: Adding readiness probe on port 8080, path='/actuator/health', scheme='HTTP', with initial delay 10 seconds
[INFO] k8s: jkube-healthcheck-spring-boot: Adding liveness probe on port 8080, path='/actuator/health', scheme='HTTP', with initial delay 180 seconds
[INFO] k8s: jkube-service-discovery: Using first mentioned service port '8080' 
[INFO] k8s: jkube-revision-history: Adding revision history limit to 2
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-deployment.yml resource
[WARNING] Unknown keyword $module - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
[WARNING] Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
[WARNING] Unknown keyword javaOmitEmpty - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-service.yml resource
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-rolebinding.yml resource
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-namespace.yml resource
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-configmap.yml resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.10.1:compile (default-compile) @ spring-jkube-external-config ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ spring-jkube-external-config ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /Users/edeandre/workspaces/workshops/spring-jkube-external-config/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.10.1:testCompile (default-testCompile) @ spring-jkube-external-config ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ spring-jkube-external-config ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:3.2.2:jar (default-jar) @ spring-jkube-external-config ---
[INFO] Building jar: /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/spring-jkube-external-config-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.7.3:repackage (repackage) @ spring-jkube-external-config ---
[INFO] Replacing main artifact with repackaged archive
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.9.0:build (jkube) @ spring-jkube-external-config ---
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image registry.access.redhat.com/ubi8/openjdk-17:latest as base / builder
[INFO] k8s: [acme/spring-jkube-external-config:latest] "spring-boot": Created docker-build.tar in 137 milliseconds
[INFO] k8s: [acme/spring-jkube-external-config:latest] "spring-boot": Built image sha256:6f7e1
[INFO] k8s: [acme/spring-jkube-external-config:latest] "spring-boot": Removed old image sha256:7f96b
[INFO] k8s: [acme/spring-jkube-external-config:latest] "spring-boot": Tag with latest
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ spring-jkube-external-config ---
[INFO] Installing /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/spring-jkube-external-config-0.0.1-SNAPSHOT.jar to /Users/edeandre/.m2/repository/org/acme/spring-jkube-external-config/0.0.1-SNAPSHOT/spring-jkube-external-config-0.0.1-SNAPSHOT.jar
[INFO] Installing /Users/edeandre/workspaces/workshops/spring-jkube-external-config/pom.xml to /Users/edeandre/.m2/repository/org/acme/spring-jkube-external-config/0.0.1-SNAPSHOT/spring-jkube-external-config-0.0.1-SNAPSHOT.pom
[INFO] Installing /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes.yml to /Users/edeandre/.m2/repository/org/acme/spring-jkube-external-config/0.0.1-SNAPSHOT/spring-jkube-external-config-0.0.1-SNAPSHOT-kubernetes.yml
[INFO] 
[INFO] <<< kubernetes-maven-plugin:1.9.0:deploy (default-cli) < install @ spring-jkube-external-config <<<
[INFO] 
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.9.0:deploy (default-cli) @ spring-jkube-external-config ---
[INFO] k8s: Using Kubernetes at https://127.0.0.1:59256/ in namespace spring-jkube-external-config with manifest /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes.yml 
[INFO] k8s: Creating currentNamespace: spring-jkube-external-config
[INFO] k8s: Created Namespace: target/jkube/applyJson/spring-jkube-external-config/namespace-spring-jkube-external-config-4.json
[INFO] k8s: Creating a Service from kubernetes.yml namespace spring-jkube-external-config name spring-jkube-external-config
[INFO] k8s: Created Service: target/jkube/applyJson/spring-jkube-external-config/service-spring-jkube-external-config-4.json
[INFO] k8s: Creating project: spring-jkube-external-config
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.200 s
[INFO] Finished at: 2022-09-13T08:22:23-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.9.0:deploy (default-cli) on project spring-jkube-external-config: Execution default-cli of goal org.eclipse.jkube:kubernetes-maven-plugin:1.9.0:deploy failed: Failure executing: GET at: https://127.0.0.1:59256/apis/project.openshift.io/v1/projects. Message: Not Found. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

It looks as if its trying to create a project.openshift.io/v1/project, but I'm not sure why it would be doing that. This is the kubernetes-maven-plugin, not the openshift-maven-plugin.

This issue is preventing me from upgrading.

Eclipse JKube version

other (please specify in additional context)

Component

Kubernetes Maven Plugin

Apache Maven version

3.8.4

Gradle version

No response

Steps to reproduce

  1. Clone https://github.com/RedHat-Middleware-Workshops/spring-jkube-external-config/tree/solution
  2. In pom.xml, update <jkube.version>1.8.0</jkube.version> to <jkube.version>1.9.0</jkube.version>
  3. Startup minikube
  4. Run ./mvnw clean k8s:deploy -Pk8s -DskipTests
  5. Observe error described in description

In addition to failing on my local machine (Mac M1), it is failing in GitHub actions too (https://github.com/RedHat-Middleware-Workshops/spring-jkube-external-config/actions/runs/3034325286)

Expected behavior

I would expect the deployment to minikube to be successful, just as it is with version 1.8.0

Runtime

minikube

Kubernetes API Server version

other (please specify in additional context)

Environment

Linux, macOS

Eclipse JKube Logs

[INFO] Scanning for projects...
[INFO] 
[INFO] ---------------< org.acme:spring-jkube-external-config >----------------
[INFO] Building spring-jkube-external-config 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] >>> kubernetes-maven-plugin:1.9.0:deploy (default-cli) > install @ spring-jkube-external-config >>>
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ spring-jkube-external-config ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.9.0:resource (jkube) @ spring-jkube-external-config ---
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image registry.access.redhat.com/ubi8/openjdk-17:latest as base / builder
[INFO] k8s: Using resource templates from /Users/edeandre/workspaces/workshops/spring-jkube-external-config/src/main/jkube
[INFO] k8s: jkube-controller: Adding a default Deployment
[INFO] k8s: jkube-namespace: Adding a default Namespace: spring-jkube-external-config
[INFO] k8s: jkube-healthcheck-spring-boot: Adding readiness probe on port 8080, path='/actuator/health', scheme='HTTP', with initial delay 10 seconds
[INFO] k8s: jkube-healthcheck-spring-boot: Adding liveness probe on port 8080, path='/actuator/health', scheme='HTTP', with initial delay 180 seconds
[INFO] k8s: jkube-service-discovery: Using first mentioned service port '8080' 
[INFO] k8s: jkube-revision-history: Adding revision history limit to 2
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-deployment.yml resource
[WARNING] Unknown keyword $module - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
[WARNING] Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
[WARNING] Unknown keyword javaOmitEmpty - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-service.yml resource
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-rolebinding.yml resource
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-namespace.yml resource
[INFO] k8s: validating /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes/spring-jkube-external-config-configmap.yml resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.10.1:compile (default-compile) @ spring-jkube-external-config ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ spring-jkube-external-config ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /Users/edeandre/workspaces/workshops/spring-jkube-external-config/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.10.1:testCompile (default-testCompile) @ spring-jkube-external-config ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ spring-jkube-external-config ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:3.2.2:jar (default-jar) @ spring-jkube-external-config ---
[INFO] Building jar: /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/spring-jkube-external-config-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.7.3:repackage (repackage) @ spring-jkube-external-config ---
[INFO] Replacing main artifact with repackaged archive
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.9.0:build (jkube) @ spring-jkube-external-config ---
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Running generator spring-boot
[INFO] k8s: spring-boot: Using Docker image registry.access.redhat.com/ubi8/openjdk-17:latest as base / builder
[INFO] k8s: [acme/spring-jkube-external-config:latest] "spring-boot": Created docker-build.tar in 137 milliseconds
[INFO] k8s: [acme/spring-jkube-external-config:latest] "spring-boot": Built image sha256:6f7e1
[INFO] k8s: [acme/spring-jkube-external-config:latest] "spring-boot": Removed old image sha256:7f96b
[INFO] k8s: [acme/spring-jkube-external-config:latest] "spring-boot": Tag with latest
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ spring-jkube-external-config ---
[INFO] Installing /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/spring-jkube-external-config-0.0.1-SNAPSHOT.jar to /Users/edeandre/.m2/repository/org/acme/spring-jkube-external-config/0.0.1-SNAPSHOT/spring-jkube-external-config-0.0.1-SNAPSHOT.jar
[INFO] Installing /Users/edeandre/workspaces/workshops/spring-jkube-external-config/pom.xml to /Users/edeandre/.m2/repository/org/acme/spring-jkube-external-config/0.0.1-SNAPSHOT/spring-jkube-external-config-0.0.1-SNAPSHOT.pom
[INFO] Installing /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes.yml to /Users/edeandre/.m2/repository/org/acme/spring-jkube-external-config/0.0.1-SNAPSHOT/spring-jkube-external-config-0.0.1-SNAPSHOT-kubernetes.yml
[INFO] 
[INFO] <<< kubernetes-maven-plugin:1.9.0:deploy (default-cli) < install @ spring-jkube-external-config <<<
[INFO] 
[INFO] 
[INFO] --- kubernetes-maven-plugin:1.9.0:deploy (default-cli) @ spring-jkube-external-config ---
[INFO] k8s: Using Kubernetes at https://127.0.0.1:59256/ in namespace spring-jkube-external-config with manifest /Users/edeandre/workspaces/workshops/spring-jkube-external-config/target/classes/META-INF/jkube/kubernetes.yml 
[INFO] k8s: Creating currentNamespace: spring-jkube-external-config
[INFO] k8s: Created Namespace: target/jkube/applyJson/spring-jkube-external-config/namespace-spring-jkube-external-config-4.json
[INFO] k8s: Creating a Service from kubernetes.yml namespace spring-jkube-external-config name spring-jkube-external-config
[INFO] k8s: Created Service: target/jkube/applyJson/spring-jkube-external-config/service-spring-jkube-external-config-4.json
[INFO] k8s: Creating project: spring-jkube-external-config
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.200 s
[INFO] Finished at: 2022-09-13T08:22:23-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.9.0:deploy (default-cli) on project spring-jkube-external-config: Execution default-cli of goal org.eclipse.jkube:kubernetes-maven-plugin:1.9.0:deploy failed: Failure executing: GET at: https://127.0.0.1:59256/apis/project.openshift.io/v1/projects. Message: Not Found. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Sample Reproducer Project

https://github.com/RedHat-Middleware-Workshops/spring-jkube-external-config/tree/solution

Additional context

JKube version is 1.9.0 (no option for 1.9.0 is available in the Eclipse JKube version dropdown.

Minikube/Kubernetes version:

╰─ minikube start                
😄  minikube v1.26.1 on Darwin 12.6 (arm64)
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.24.3 on Docker 20.10.17 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image kubernetesui/dashboard:v2.6.0
    ▪ Using image kubernetesui/metrics-scraper:v1.0.8
🌟  Enabled addons: storage-provisioner, default-storageclass, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
manusa commented 1 year ago

That's odd (the project check), we'll take a look and provide a fix.

rohanKanojia commented 1 year ago

Thanks for the bug report. I can reproduce this issue. I think this is caused by upgrade of KubernetesClient. I see this line:

https://github.com/eclipse/jkube/blob/c0ac9b2a2327c6568c5707e675b266067ed74ec0/jkube-kit/common/src/main/java/org/eclipse/jkube/kit/common/util/OpenshiftHelper.java#L48

in v1.8.0, it was throwing OpenShiftNotAvailableException but on v1.9.0 it's just returning a regular DefaultOpenShiftClient object

manusa commented 1 year ago

Could you provide the stack-trace, that method shouldn't be called to determine if the underlying cluster is OpenShift (I think most of these checks were replaced when upgrading the client version).

Also, do you have any idea why our tests haven't showed this behavior?

rohanKanojia commented 1 year ago

I think it's called from ApplyService while applying RoleBinding

ApplyService.applyRoleBinding(ApplyService.java:712)
   ApplyService.applyNamespace(ApplyService.java:954)
       ApplyService.applyNamespace(ApplyService.java:961)
           ApplyService.getOpenShiftClient(ApplyService.java:349)
                OpenShiftHelper.asOpenShiftClient(OpenShiftHelper.java:49) (should return null but returns valid object)

We do seem to have a test for this scenario in OpenShiftHelperTest but there we're mocking client's adapt method to throw exception (which is not happening with 6.x version)

manusa commented 1 year ago

OK; so this is due to the usage of a RoleBinding resource.

manusa commented 1 year ago

In the original Kubernetes Client bump PR (#1464), we left out a few places where we checked if the cluster was OpenShift.

To verify if a cluster is an OpenShift cluster, we need to reuse the method OpenshiftHelper.isOpenShift.

So basically we need to find all of the usages of asOpenShiftClient and replace any if-else with the asOpenShiftClient(client) != null with the proper check (isOpenShift).

For additional safety, we should also modify the asOpenShiftClient to make sure that if isOpenShift(client) returns false, the method returns null.

edeandrea commented 1 year ago

Thank you @manusa and @rohanKanojia for the quick turnaround! Is there a timeline for when 1.9.1 might be released?

manusa commented 1 year ago

It was released this morning, but there might be some problems with the repository synchronization. I've been waiting for more than 8 hours now to complete the release process :sweat:

https://twitter.com/MarcNuri/status/1569957559188029440

https://oss.sonatype.org/service/local/repo_groups/public/content/org/eclipse/jkube/kubernetes-maven-plugin/maven-metadata.xml

manusa commented 1 year ago

New version is now available from Maven Central. @edeandrea please confirm if the fix works for you :pray: :crossed_fingers:

edeandrea commented 1 year ago

Doesn't look like its in maven central yet (or at least hasn't dispersed through their CDN).

image

edeandrea commented 1 year ago

Now its there. Looks good! Confirmed it worked.

edeandrea commented 1 year ago

Again - many thanks for the extremely quick turnaround!

manusa commented 1 year ago

Now its there. Looks good! Confirmed it worked.

Awesome news :rocket:

Again - many thanks for the extremely quick turnaround!

Thank you for trying and reporting. This was a critical one, I'm glad we caught it so quickly.