Cannot use a template variable for `service.yml` on `spec.ports[].port`

ajeans commented 11 months ago

Describe the bug

I believe there is a templating problem. Here are the steps, apologies if I failed to read the documentation.

I am trying to push the service port into a template yaml fragment:

File: src/main/jkube/service-template.yml

kind: Template
parameters:
  - name: service.server.port
    value: 8080

While I can use this variable in an ingress fragment... File: src/main/jkube/ingress-internal.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: ${common.namespace}
spec:
  ingressClassName: rfr-internal
  rules:
    - host: ${project.artifactId}.qa1.rfr-tech.net
      http:
        paths:
          - backend:
              service:
                name: ${project.artifactId}
                port:
                  number: ${service.server.port}
            path: /
            pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
      - ip: 192.168.212.67

... this fails with an error when I try to do the same for the service fragment.

File:src/main/jkube/service.yml

metadata:
  namespace: ${common.namespace}
spec:
  ports:
    - name: http
      port: ${service.server.port}
      protocol: TCP
      targetPort: http-main
  type: NodePort

Eclipse JKube version

SNAPSHOT

Component

Kubernetes Maven Plugin

Apache Maven version

3.8.5

Gradle version

None

Steps to reproduce

Add all three fragments from the description to a maven project (maybe quickstarts/maven/spring-boot-helm)

Build the project with

./mvnw clean install -Dmaven.test.skip
./mvnw k8s:resource k8s:helm

And notice the error output on the resource goal

[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.16-SNAPSHOT:resource (default-cli) on project microservice-app: Execution default-cli of goal org.eclipse.jkube:kubernetes-maven-plugin:1.16-SNAPSHOT:resource failed: Service null .spec.ports[0].port: required value -> [Help 1]

Expected behavior

Maven generates the k8s resources successfully.

Runtime

other (please specify in additional context)

Kubernetes API Server version

1.25.3

Environment

macOS

Eclipse JKube Logs

[INFO] 
[INFO] --- k8s:1.16-SNAPSHOT:resource (default-cli) @ microservice-app ---
[INFO] k8s: Using Dockerfile: /Volumes/Workspace/b2c/template-springboot-microservice/microservice-app/Dockerfile
[INFO] k8s: Using Docker Context Directory: /Volumes/Workspace/b2c/template-springboot-microservice/microservice-app
[INFO] k8s: Using resource templates from /Volumes/Workspace/b2c/template-springboot-microservice/microservice-app/src/main/jkube
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.820 s
[INFO] Finished at: 2023-11-27T18:17:39+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.16-SNAPSHOT:resource (default-cli) on project microservice-app: Execution default-cli of goal org.eclipse.jkube:kubernetes-maven-plugin:1.16-SNAPSHOT:resource failed: Service null .spec.ports[0].port: required value -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Sample Reproducer Project

No response

Additional context

No response

manusa commented 11 months ago

I am trying to push the service port into a template yaml fragment:

This is no longer the recommended approach, I'd suggest to configure the parameters using the pom.xml file.

For example:

<plugin>
  <groupId>org.eclipse.jkube</groupId>
  <artifactId>kubernetes-maven-plugin</artifactId>
  <version>${jkube.version}</version>
  <configuration>
    <helm>
      <parameters>
        <parameter>
          <name>ingress.host</name>
          <value>host.example.com</value>
        </parameter>
      </parameters>
    </helm>
  </configuration>
</plugin>

Regarding the problem, it's due to the type of the value (I think). Could you please try something like:

<plugin>
  <groupId>org.eclipse.jkube</groupId>
  <artifactId>kubernetes-maven-plugin</artifactId>
  <version>${jkube.version}</version>
  <configuration>
    <helm>
      <parameters>
        <parameter>
          <name>service.server.port</name>
          <value>{{ .Values.service.server.port | default 8080 }}</value>
        </parameter>
      </parameters>
    </helm>
  </configuration>
</plugin>

manusa commented 11 months ago

It might also interest you the demo I delivered on EclipseCon this year: https://youtu.be/DvMickoLlXM?si=ouqkEY5079ZLuFDA

This is the repo for the demo https://github.com/marcnuri-demo/2023-eclipsecon-spring-petclinic/tree/jkube (note that the appropriate branch is jkube)

ajeans commented 11 months ago

I am trying to push the service port into a template yaml fragment:

This is no longer the recommended approach, I'd suggest to configure the parameters using the pom.xml file.

For example:

<plugin>
  <groupId>org.eclipse.jkube</groupId>
  <artifactId>kubernetes-maven-plugin</artifactId>
  <version>${jkube.version}</version>
  <configuration>
    <helm>
      <parameters>
        <parameter>
          <name>ingress.host</name>
          <value>host.example.com</value>
        </parameter>
      </parameters>
    </helm>
  </configuration>
</plugin>

Regarding the problem, it's due to the type of the value (I think). Could you please try something like:

<plugin>
  <groupId>org.eclipse.jkube</groupId>
  <artifactId>kubernetes-maven-plugin</artifactId>
  <version>${jkube.version}</version>
  <configuration>
    <helm>
      <parameters>
        <parameter>
          <name>service.server.port</name>
          <value>{{ .Values.service.server.port | default 8080 }}</value>
        </parameter>
      </parameters>
    </helm>
  </configuration>
</plugin>

This fails with the same error message. This errors out on the resources goal, not the helm one, so I assume that the helm configuration change has no effect on this problem.
Adding a global property in the pom.xml file, as you did in the eclipsecon jkube branch makes it work for k8s:resource, but does not add the value in the values.ymlof the helm chart.

    <properties>
        <service.server.port>8080</service.server.port>
    </properties>

Trying to do a go expression inside the property also fails similarly on the resource goal

    <properties>
        <service.server.port>{{ .Values.service.server.port | default 8080 }}</service.server.port>
    </properties>

Looking at the source code, it seems to be due to `ServiceDiscoveryEnricher#addAnnotations" when it tries to add the "discovery.3scale.net/port" annotation.

I will try to disable this enricher and see if that helps...

ajeans commented 11 months ago

Yes, excluding this enricher seems to make it all work

                     </images>
+                    <enricher>
+                        <excludes>
+                            <!-- Avoid https://github.com/eclipse/jkube/issues/2479 -->
+                            <exclude>jkube-service-discovery</exclude>
+                        </excludes>
+                    </enricher>
                 </configuration>
             </plugin>

With that, I get the expected values.yaml.

[...]
service:
  server:
    port: "8080"

Thanks for the youtube link, I know I am focusing on the outer loop, but making the outer loop work fine is my primary goal. Making k8s:apply work too is a nice secondary goal, but worst case, people can use helm locally on their k8s clusters.

manusa commented 11 months ago

Adding a global property in the pom.xml file, as you did in the eclipsecon jkube branch makes it work for k8s:resource, but does not add the value in the values.ymlof the helm chart.

This is the point that's hard to understand and that I try to make clear in this presentations.

The k8s:resource goal is mostly related to the inner-loop. This means that while working on your project as a developer, you'll be iterating and deploying to a development cluster where you'll be testing your changes. So in this case, you need the k8s:resource and k8s:apply goals.

Note how the project I linked has a dev profile to set this properties. So while developing, the placeholders will always be replaced by the values that should work in your development environment.

The k8s:helm goal is mostly related to the outer-loop. You will be using it to generate the Helm charts that will be used to deploy your application to production. Since (for now) in JKube the k8s:helm goal requires the k8s:resource goal to generate its input templates, in this case the generated YAMLs are invalid (hence the warning). However, when k8s:helm picks them up it outputs valid Helm templates (which are still invalid YAML by the way).

This is why in the demo project there is a prodprofile. Note that in this profile, none of the properties in the YAML fragments are set, this way the Helm generation logic is able to replace those placeholders with valid Helm expressions.

ajeans commented 11 months ago

Hello @manusa

Thanks for all the feedback and the updated example in the quickstarts.

I understand that the inner loop requires shadowing all template entries into pom properties, in a maven profile. I tested it with my POC and shadowed the .Release.Namespace -> it works fine.

So that's one way to fix this, thanks a lot.

I feel that there is an underlying bug ; in my understanding an enricher should enrich if it is activated, or not enrich if it cannot, but never fail the build / generation. But that's obviously your call to make :)

I have another question / issue regarding helm tests and how they can work within the inner loop, now that I started working on it. I will open another ticket, please bear with me :)

ajeans commented 11 months ago

@manusa OK I think there really is a bug with the jkube-service-discovery enricher.

I started from master (as of this morning) and with the updated project quickstarts/maven/spring-boot-helm.

With the following patch applied...

diff --git a/quickstarts/maven/spring-boot-helm/pom.xml b/quickstarts/maven/spring-boot-helm/pom.xml
index 37f478d05..b1c9cc2e1 100644
--- a/quickstarts/maven/spring-boot-helm/pom.xml
+++ b/quickstarts/maven/spring-boot-helm/pom.xml
@@ -94,6 +94,7 @@
         <deployment.resources.limits_memory/>
         <deployment.resources.requests_memory/>
         <service.spec.type>NodePort</service.spec.type>
+        <service.server.port>8080</service.server.port>
         <service.annotations>api.service.kubernetes.io/path: /dev</service.annotations>
       </properties>
     </profile>
diff --git a/quickstarts/maven/spring-boot-helm/src/main/jkube/service.yml b/quickstarts/maven/spring-boot-helm/src/main/jkube/service.yml
index 9f13db0d7..4b2e6de00 100644
--- a/quickstarts/maven/spring-boot-helm/src/main/jkube/service.yml
+++ b/quickstarts/maven/spring-boot-helm/src/main/jkube/service.yml
@@ -16,4 +16,9 @@ metadata:
   annotations:
     ${service.annotations}
 spec:
+  ports:
+    - name: http
+      port: ${service.server.port}
+      protocol: TCP
+      targetPort: http-main
   type: ${service.spec.type}

The inner loop documented in the README.md works fine: mvn clean package k8s:build k8s:resource generates a service with port 8080

The outer loop documented in the README.md fails though: mvn -Pprod clean package k8s:build k8s:resource k8s:helm fails on the k8s:resource goal, because we don't shadow the variable and helm is not in the picture yet.

[ERROR] Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.15.0:resource (default-cli) on project spring-boot-helm: Execution default-cli of goal org.eclipse.jkube:kubernetes-maven-plugin:1.15.0:resource failed: Service spring-boot-helm .spec.ports[0].port: required value -> [Help 1]

Am I missing something? Does this work for you with the attached patch?

rohanKanojia commented 11 months ago

That looks strange, Is service.server.port set in prod profile or in dev profile?

ajeans commented 11 months ago

@rohanKanojia

I pushed the diff onto a branch on a forked jkube, you can look at https://github.com/ajeans/jkube/tree/issue-2479/quickstarts/maven/spring-boot-helm (I also added the template entry to make it more understandable, even if it fails before that).

It is set as a property in the dev profile (value 8080). https://github.com/ajeans/jkube/blob/issue-2479/quickstarts/maven/spring-boot-helm/pom.xml#L97

It is not set as a property in the prod profile, It is set in the template (value 9090). https://github.com/ajeans/jkube/blob/issue-2479/quickstarts/maven/spring-boot-helm/src/main/jkube/template.yaml#L22

I would expect it to create an entry in the helm values.yaml with a default value of 9090. But it fails before that, as stated above.

eclipse-jkube / jkube