terrakube default values should work out of the box

[Diliz]

Hello there!

The helm chart should be installable as it is in a simple kubernetes environment, like minikube, kind, or whatever clusters.

It would provide a better "accessibility" to this tool, and facilitate adoption, I currently had to check the templates and values.schema.json to make it work, should not really be the case for "default" kubernetes context (like quick homelabs stuff)

Can you provide a "sandbox" version of the values that would work just by using helm install as this?

EDIT:

By doing:

helm repo add terrakube-community https://AzBuilder.github.io/terrakube-helm-chart
helm repo update
helm install terrakube-community/terrakube -n terrakube --generate-name

I got the following output:

Error: INSTALLATION FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
terrakube:
- ui.serviceType: ui.serviceType must be one of the following: "ClusterIP", "NodePort", "LoadBalancer", "ExternalName"
- ingress.ui.pathType: ingress.ui.pathType must be one of the following: "ImplementationSpecific", "Exact", "Prefix"
- ingress.api.pathType: ingress.api.pathType must be one of the following: "ImplementationSpecific", "Exact", "Prefix"
- ingress.registry.pathType: ingress.registry.pathType must be one of the following: "ImplementationSpecific", "Exact", "Prefix"
- registry.serviceType: registry.serviceType must be one of the following: "ClusterIP", "NodePort", "LoadBalancer", "ExternalName"
- security: patSecret is required
- security: internalSecret is required
- api.properties.databaseType: api.properties.databaseType must be one of the following: "H2", "SQL_AZURE", "POSTGRESQL", "MYSQL"
- api.serviceType: api.serviceType must be one of the following: "ClusterIP", "NodePort", "LoadBalancer", "ExternalName"
- executor.serviceType: executor.serviceType must be one of the following: "ClusterIP", "NodePort", "LoadBalancer", "ExternalName"

[alfespa17]

Hello @Diliz

I understand your point that is something that we need to improve in the helm chart.

We have been thinking how to do acomplish this when installing for example in minikube with some default values but rigth now I am not really sure how to do it because of the following:

• Dex configuration (this needs external dependency like ldap, azure ad, google identity, etc) to make it work.

Any suggestion of how we can accomplish this is welcome.

This is why we provide several examples in inside this folder , there you can find examples for the values that you will need when installing with different cloud providers:

Google Identity Authentication Ngnix Ingress + H2 Database + GCP Storage Bucket Ngnix Ingress + PostgreSQL + GCP Storage Bucket Ngnix Ingress + MySQL + GCP Storage Bucket Azure Authentication Ngnix Ingress + H2 Database + Azure Storage Account Ngnix Ingress + SQL Azure + Azure Storage Account Ngnix Ingress + PostgreSQL + Azure Storage Account Amazon Load Balancer + PostgreSQL + S3 Bucket Github Authentication Ngnix Ingress + H2 Database + Azure Storage Account Amazon AWS Cognito Ngnix Ingress + H2 Database + AWS S3 Bucket

[Diliz]

Hello!

Yep, I was more thinking about maybe having a user password authentication without a dex thing per default so you could use it everywhere

And I'm wondering why the official dex chart was not used here? (https://github.com/dexidp/helm-charts)

[alfespa17]

Hello.

Local accounts coud be a good idea this will help so we can deploy the helm chart without any external dependency.

Maybe this could be implemented using the spring-authorization-server it implement OIDC so we could basically just put DEX on top and have something like local accounts, we will have to do some research to add a feature like this one. I will create an issue around this in our main repository

For your question about the dex helm chart, well we tried to keep it simple so we just add a very basic dex implementation.

[Diliz]

Re!

Had some ideas on how to do that, related to https://github.com/AzBuilder/terrakube/issues/329 There should be 2 things here to make it "installable" locally without too much to do I think:

• add local/private storage/databases values in here as well (same as https://github.com/AzBuilder/terrakube/issues/329)
• more an advice than else, use third party helm charts when you can, like minio, dex, and postgres ones at least, so you just have to maintain your application values and stuff in the beginning

I'll try to provide you a simple openshift values (doing openshift here + s3 operator as an example), but I'm struggling with bad user rights in the ui pod right now.

Your chart should always be launchable in sandboxed mode from my point of view, from an active developper or as a newcomer, if the user is struggling to make your product work, he will lose faith in your product

(I'll try to help on this chart because I think this is a good product, just need some "helm install . and everything's running" magic here)

[alfespa17]

I opened this issue to explore options for a future implementation of local accounts.

Azbuilder/terrakube#330

All help is welcome 👍 if you want to contribute with some example or enhancements feel free to send a pull request.

[alfespa17]

By the way after I saw you comment about the user righst in the UI pod I noticed the image was using a "root" user, I did a small update in the Dockerfile using the bitnami/nginx image and now the image is not running as root anymore, you could check terrakube version 2.11.2 with the helm chart 2.3.1 maybe that help with your issue with bad user rigths.

[Diliz]

As a Quick win, using keycloak + dex on minikube should work for you hère, I'll try that on my side, still need the ability to declare the local storage/dex stuff in the chart without using kustomize (which I did to override the application properties given to the apps

Thanks for the nginx image, I'll check this out

EDIT: The nginx image is working well without any modification on openshift now thanks

Tried to activate the registry, and there's the internalsecret declared twice, so it's not working: https://github.com/AzBuilder/terrakube-helm-chart/blob/main/charts/terrakube/templates/secrets-registry.yaml#L14

I'll make a pull request here when everything is working on my side to enable this chart to be runnable in a minikube environment

EDIT2:

The executor seems to have an issue with the local one, got this error on my side:

[main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'executorJobImpl' defined in file [/workspace/BOOT-INF/classes/org/terrakube/executor/service/executor/ExecutorJobImpl.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'terraformExecutorServiceImpl' defined in file [/workspace/BOOT-INF/classes/org/terrakube/executor/service/terraform/TerraformExecutorServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'terraformClient' defined in class path resource [org/terrakube/terraform/spring/autoconfigure/TerraformAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:731)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292)
    at org.terrakube.executor.ExecutorApplication.main(ExecutorApplication.java:10)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'terraformExecutorServiceImpl' defined in file [/workspace/BOOT-INF/classes/org/terrakube/executor/service/terraform/TerraformExecutorServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'terraformClient' defined in class path resource [org/terrakube/terraform/spring/autoconfigure/TerraformAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
    ... 27 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'terraformClient' defined in class path resource [org/terrakube/terraform/spring/autoconfigure/TerraformAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
    ... 41 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
    ... 55 more
Caused by: java.lang.RuntimeException: Error creating TerraformDownloader
    at org.terrakube.terraform.TerraformDownloader.<init>(TerraformDownloader.java:44)
    at org.terrakube.terraform.TerraformClient.<init>(TerraformClient.java:33)
    at org.terrakube.terraform.TerraformClient$TerraformClientBuilder.build(TerraformClient.java:12)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration.terraformClient(TerraformAutoConfiguration.java:19)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration$$EnhancerBySpringCGLIB$$ad0dc1da.CGLIB$terraformClient$0(<generated>)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration$$EnhancerBySpringCGLIB$$ad0dc1da$$FastClassBySpringCGLIB$$28964ca5.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration$$EnhancerBySpringCGLIB$$ad0dc1da.terraformClient(<generated>)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
    ... 56 more

I'm not a java/springboot developper (I tried to look at the terrakube codebase but didn't find what was wrong here), if you have a clue I'll be glad to help on fixing it as well 👍

EDIT3:

It seems that the secret for the ui are not loaded correctly on my side, it's in the secrets file and everything seems ok with it, even after adding REACTCONFIG* values in the terrakube-ui-secrets I still got the following issue in the UI:

env-config.js:1 

       Uncaught SyntaxError: Unexpected token '<'
authConfig.js:6 

       Uncaught TypeError: Cannot read properties of undefined (reading 'REACT_APP_AUTHORITY')
    at authConfig.js:6:27
    at index.js:18:1
    at index.js:18:1

[alfespa17]

Let me check the helm chart I will make it work with minikube, I will do the changes so you dont need to add the database configuration or the storage part I think I can make it work with MINIO like we do in GITPOD.

For the UI error, make sure you are using the latest version 2.11.2 and the helm chat 2.3.1.

## UI Properties
ui:
  enabled: true
  version: "2.11.2"
  replicaCount: "1"
  serviceType: "ClusterIP"

You should see a file /app/env-config.js with the information like this if the UI setup was done correctly. .

I never noticed the duplicated secret I will fix that in the registry secret thank you.

Not sure about the executor error I guess is related to the helm chart values. that you are using maybe the storage configuration. Maybe you can share the storage part of the helm chart that you are using and the executor properties.

It will be something like this if you are using minio:

storage:
  aws:
    accessKey: ""
    secretKey: ""
    bucketName: ""
    region: ""
    endpoint: ""

## Executor properties
executor:
  enabled: true
  version: "2.11.2"  
  replicaCount: "1"
  serviceType: "ClusterIP"
  properties:
    toolsRepository: "https://github.com/AzBuilder/terrakube-extensions"
    toolsBranch: "main"

[alfespa17]

Now you should be able to use minikube @Diliz .

Please check this PR for more information https://github.com/AzBuilder/terrakube-helm-chart/pull/49/

I tested minikube in Ubuntu and it should work.

[alfespa17]

By the way I used openldap for the default user and groups because I used the same approach in GITPOD, I am not really familiar with keycloak so I was not sure how to implemement it with minikube but if you have any suggestion let me know so I can learn a little bit about that.

[alfespa17]

Added some documentation here

https://docs.terrakube.org/getting-started/deployment/minikube

[Diliz]

Oh nice! Didn't saw your comment before! I'll try that, thank you for the work :)

EDIT: tested this out, seems to work on minikube

I got issues on openshift with the minio and pstgres which seem priviledged, I'll try to make unpriviledged ones so it would work on any kind of kubernetes cluster by default

EDIT2:

My bad, was thinking it was on minikube, but still have the executor error (Didn't saw the pod was not in my list of pods at first)

So, for openshift side: I just removed the security context things in the postgres and minio deployment/statefulsets and it worked

And the issues I'm still facing: So I got the following issue with the executor:

main] WARN org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'executorJobImpl' defined in file [/workspace/BOOT-INF/classes/org/terrakube/executor/service/executor/ExecutorJobImpl.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'terraformExecutorServiceImpl' defined in file [/workspace/BOOT-INF/classes/org/terrakube/executor/service/terraform/TerraformExecutorServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'terraformClient' defined in class path resource [org/terrakube/terraform/spring/autoconfigure/TerraformAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
Mar 02, 2023 3:56:59 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Tomcat]
[main] INFO org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener - 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
[main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'executorJobImpl' defined in file [/workspace/BOOT-INF/classes/org/terrakube/executor/service/executor/ExecutorJobImpl.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'terraformExecutorServiceImpl' defined in file [/workspace/BOOT-INF/classes/org/terrakube/executor/service/terraform/TerraformExecutorServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'terraformClient' defined in class path resource [org/terrakube/terraform/spring/autoconfigure/TerraformAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:731)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292)
    at org.terrakube.executor.ExecutorApplication.main(ExecutorApplication.java:10)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'terraformExecutorServiceImpl' defined in file [/workspace/BOOT-INF/classes/org/terrakube/executor/service/terraform/TerraformExecutorServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'terraformClient' defined in class path resource [org/terrakube/terraform/spring/autoconfigure/TerraformAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
    ... 27 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'terraformClient' defined in class path resource [org/terrakube/terraform/spring/autoconfigure/TerraformAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
    ... 41 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.terrakube.terraform.TerraformClient]: Factory method 'terraformClient' threw exception; nested exception is java.lang.RuntimeException: Error creating TerraformDownloader
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
    ... 55 more
Caused by: java.lang.RuntimeException: Error creating TerraformDownloader
    at org.terrakube.terraform.TerraformDownloader.<init>(TerraformDownloader.java:44)
    at org.terrakube.terraform.TerraformClient.<init>(TerraformClient.java:33)
    at org.terrakube.terraform.TerraformClient$TerraformClientBuilder.build(TerraformClient.java:12)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration.terraformClient(TerraformAutoConfiguration.java:19)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration$$EnhancerBySpringCGLIB$$f60a2e28.CGLIB$terraformClient$0(<generated>)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration$$EnhancerBySpringCGLIB$$f60a2e28$$FastClassBySpringCGLIB$$ae5db79d.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration$$EnhancerBySpringCGLIB$$f60a2e28.terraformClient(<generated>)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
    ... 56 more

Native Memory Tracking:

Total: reserved=51507053904, committed=595150160
       malloc: 149198160 #248315
       mmap:   reserved=51357855744, committed=445952000

-                 Java Heap (reserved=49023025152, committed=335544320)
                            (mmap: reserved=49023025152, committed=335544320) 

-                     Class (reserved=139103276, committed=5082156)
                            (classes #7793)
                            (  instance classes #7249, array classes #544)
                            (malloc=691244 #16144) 
                            (mmap: reserved=138412032, committed=4390912) 
                            (  Metadata:   )
                            (    reserved=33554432, committed=30081024)
                            (    used=29891568)
                            (    waste=189456 =0.63%)
                            (  Class space:)
                            (    reserved=138412032, committed=4390912)
                            (    used=4250192)
                            (    waste=140720 =3.20%)

-                    Thread (reserved=43186256, committed=1751120)
                            (thread #41)
                            (stack: reserved=43077632, committed=1642496)
                            (malloc=66432 #274) 
                            (arena=42192 #78)

-                      Code (reserved=254491824, committed=17509552)
                            (malloc=863408 #4933) 
                            (mmap: reserved=253628416, committed=16646144) 

-                        GC (reserved=1887825328, committed=81063344)
                            (malloc=34295216 #2961) 
                            (mmap: reserved=1853530112, committed=46768128) 

-                  Compiler (reserved=201256, committed=201256)
                            (malloc=22856 #461) 
                            (arena=178400 #15)

-                  Internal (reserved=327080, committed=327080)
                            (malloc=290216 #6016) 
                            (mmap: reserved=36864, committed=36864) 

-                    Symbol (reserved=8766704, committed=8766704)
                            (malloc=7907144 #211908) 
                            (arena=859560 #1)

-    Native Memory Tracking (reserved=3981760, committed=3981760)
                            (malloc=8704 #120) 
                            (tracking overhead=3973056)

-        Shared class space (reserved=12582912, committed=10833920)
                            (mmap: reserved=12582912, committed=10833920) 

-               Arena Chunk (reserved=99629048, committed=99629048)
                            (malloc=99629048) 

-                   Tracing (reserved=345, committed=345)
                            (malloc=345 #9) 

-                 Arguments (reserved=99, committed=99)
                            (malloc=99 #2) 

-                    Module (reserved=182624, committed=182624)
                            (malloc=182624 #1526) 

-                 Safepoint (reserved=8192, committed=8192)
                            (mmap: reserved=8192, committed=8192) 

-           Synchronization (reserved=53344, committed=53344)
                            (malloc=53344 #641) 

-            Serviceability (reserved=672, committed=672)
                            (malloc=672 #6) 

-                 Metaspace (reserved=33680256, committed=30206848)
                            (malloc=125824 #64) 
                            (mmap: reserved=33554432, committed=30081024) 

-      String Deduplication (reserved=704, committed=704)
                            (malloc=704 #8) 

-           Object Monitors (reserved=7072, committed=7072)
                            (malloc=7072 #34) 

And the openldap pod seems not working as well, got restarts then crashloopbackoff, I'll give some logs after I get some

[alfespa17]

Hello @Diliz the openldap wont work because the container is running as root (I will change that later but for now it will work for minikube for a quick sandbox to test terrakube)

Just disable the openLDAP using the following:

security:
  useOpenLDAP: false

You will have to add your DEX configuration, now it is using the official helm chart, you can read a little in this doc.

https://docs.terrakube.org/getting-started/deployment/user-authentication-dex

For the error in your executor component, I think is failing right here.

Caused by: java.lang.RuntimeException: Error creating TerraformDownloader
    at org.terrakube.terraform.TerraformDownloader.<init>(TerraformDownloader.java:44)
    at org.terrakube.terraform.TerraformClient.<init>(TerraformClient.java:33)
    at org.terrakube.terraform.TerraformClient$TerraformClientBuilder.build(TerraformClient.java:12)
    at org.terrakube.terraform.spring.autoconfigure.TerraformAutoConfiguration.terraformClient(TerraformAutoConfiguration.java:19)

https://github.com/AzBuilder/terraform-spring-boot/blob/master/terraform-client/src/main/java/org/terrakube/terraform/TerraformDownloader.java#L44

Internally it is calling this hashicorp endpoint to downlod the list of available terraform versions with this URL

https://releases.hashicorp.com/terraform/index.json

Maybe you can check if you are able to call that endpoint because it is just downloading a json file here.

https://github.com/AzBuilder/terraform-spring-boot/blob/master/terraform-client/src/main/java/org/terrakube/terraform/TerraformDownloader.java#L99

Maybe some proxy setting but not sure.

[Diliz]

Internally it is calling this hashicorp endpoint to downlod the list of available terraform versions with this URL

Ah! That should be the issue! For private kubernetes clusters (air gapped ones), I'll try to mock the call so it would work without connecting to internet at all, but can be good to be able to at least provide a private registry or this index.json file (I'll check what kind of registry it is after mocking it)

Thanks for your answers/work here, it's been helping greatly!

[alfespa17]

I will add support to have a custom index.json it is very easy to implement. It wont require a lot of time and It will be need it for private cluster because terrakube download the terraform binary from hashicorp url.

If I add support for that URL you can even compile your custom terraform cli and use it inside the platform. Your could upload your custom terraform cli to some internal storage and just use it from terrakube

[alfespa17]

Hello @Diliz I did the quick fix to support a custom terraform releases url.

I will release this in terrakube 2.12.0 later this month but for now you could test it.

Try adding this to your values.yaml when deploying the helm chart

api:
  version: "2.12.0-beta.1"
  terraformReleasesUrl: "https://eov1ys4sxa1bfy9.m.pipedream.net/"

executor:
  version: "2.12.0-beta.1"

terraformReleasesUrl should be your internal endpoint

You will have to expose GET request with the following structure:

{
   "name":"terraform",
   "versions":{
      "1.3.9":{
         "builds":[
            {
               "arch":"amd64",
               "filename":"terraform_1.3.9_linux_amd64.zip",
               "name":"terraform",
               "os":"linux",
               "url":"https://releases.hashicorp.com/terraform/1.3.9/terraform_1.3.9_linux_amd64.zip",
               "version":"1.3.9"
            }
         ],
         "name":"terraform",
         "shasums":"terraform_1.3.9_SHA256SUMS",
         "shasums_signature":"terraform_1.3.9_SHA256SUMS.sig",
         "shasums_signatures":[
            "terraform_1.3.9_SHA256SUMS.72D7468F.sig",
            "terraform_1.3.9_SHA256SUMS.sig"
         ],
         "version":"1.3.9"
      }
   }
}

You can change this field to any internal storage URL so terrrakube can download the terraform cli from a custom location

"url":"https://releases.hashicorp.com/terraform/1.3.9/terraform_1.3.9_linux_amd64.zip",

You could even use this to build a cutom terraform cli version or restrict the terraform version available in the UI.

Example:

When running executore should look like this:

[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Running Terraform show
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformClient - Creating terraform downloader using custom terraform release URL: https://releases.hashicorp.com/terraform/index.json
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Initialize TerraformDownloader using custom URL

Documentation

[Diliz]

Nice thanks! I'll try it, I redirected the call in my cluster for now, managed to make it all working except the openshift authentication in the ui, it seems to have cors issues when fetching the .well-known/openid-configuration file when I'm on the ui, try to figure it out, and I'll provide a default openshift values file as a pull request when everything is setted up here, everything seems fine right now except this, the last step :)

[alfespa17]

I think you can add this web.allowedOrigins to your dex config:

Example:

    config:
      issuer: https://terrakube-api.minikube.net/dex
      storage:
        type: memory
      oauth2:
        responseTypes: ["code", "token", "id_token"] 
        skipApprovalScreen: true
      web:
        allowedOrigins: ['*']

[Diliz]

Hello again! Yep I already did this, and it's working, have cors issues when contacting the terrakube api from the terrakube ui now, do you know where I can put custom certs to allow them to talk to each other?

[alfespa17]

If you want to add custom ca certs you could check this.

https://github.com/AzBuilder/terrakube-helm-chart#5-custom-ca-certificates-at-runtime

[Diliz]

For helm purpose (general comment- not related to the main github issue), you should use this annotation for helm charts: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments

It will help you trigger a deployment rollout each time a related configmap (works with secrets as well) is changed (when you're frequently using helm install/upgrade it's simpler to iterate)

[alfespa17]

By the way for cors issues you could check the API logs, you should see the setup there.

It will look like this with the your domain

And it is taking the value from here. https://github.com/AzBuilder/terrakube-helm-chart/blob/main/charts/terrakube/templates/secrets-api.yaml#L17

[Diliz]

I'll check this, probably an issue with only this yep, got an issue when putting my certs to the pods right now, it seems that the layers/.../jre folder is read only and the application try to put the certs I gave to him in this folder, dunno if I did something wrong here, I'm not a java dude xD

EDIT: I'm trying to use cert manager to provide certs to the app, my issue is that I can't connect to dex now, the cors issue was just because I forgot to add the my own ca chain in pod certs (not the application one), it seems that fixed the cors issue at some point

EDIT2: Got the following issue when starting the api container

WARNING: Unable to add container CA certificates to JVM because /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts is read-onlySpring Cloud Bindings Enabled

EDIT3: The following issue seems to be the exact same as I have: https://github.com/paketo-buildpacks/libjvm/issues/190

EDIT4: Tried with CNB_BINDINGS as well, but it doesn't seem to be working as well, I have the following message when starting:

Added 145 additional CA certificate(s) to system truststore
Setting Active Processor Count to 20
Calculating JVM memory based on 47642488K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx46884464K -XX:MaxMetaspaceSize=246023K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 47642488K, Thread Count: 250, Loaded Class Count: 41022, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 284 container CA certificates to JVM truststore
WARNING: Unable to add container CA certificates to JVM because /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts is read-onlySpring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=20 -XX:MaxDirectMemorySize=10M -Xmx46884464K -XX:MaxMetaspaceSize=246023K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary

[alfespa17]

Hello @Diliz

Let me explain you a little bit more, the terrakube componentes (api, registry and executor) are using buildpacks as you can see in this build log:

When you are using buildpack to add a custom CA certificate at runtime you need to do the following:

You need to provide the following environment variable to the container:

SERVICE_BINDING_ROOT: /mnt/platform/bindings

If you connecto to the container you will see that inside the path there is a folder call ca-certificates

cnb@terrakube-api-678cb68d5b-ns5gt:/mnt/platform/bindings$ ls
ca-certificates

Inside the folder you will find your PEM CA certs and one additional file call type

cnb@terrakube-api-678cb68d5b-ns5gt:/mnt/platform/bindings/ca-certificates$ ls
terrakubeDemo1.pem  terrakubeDemo2.pem  type

If you see the content of the file called type

cnb@terrakube-api-678cb68d5b-ns5gt:/mnt/platform/bindings/ca-certificates$ cat type
ca-certificates

When you put the security.caCerts in your yaml the information is save inside this secret: https://github.com/AzBuilder/terrakube-helm-chart/blob/main/charts/terrakube/templates/secrets-certs.yaml

You will see the type file with content "ca-certificates"

apiVersion: v1
kind: Secret
metadata:
  name: terrakube-ca-secrets
type: Opaque
stringData: 
  type: |
    ca-certificates

  {{- with .Values.security.caCerts }}
  {{- toYaml . | nindent 2 }}
  {{- end }}

Finally your helm values.yaml should look something like this because we are mounting your CA certs and the file called type in the following path " /mnt/platform/bindings/ca-certificates"

## API properties
api:
  version: "2.12.0-beta.1"
  terraformReleasesUrl: "https://eov1ys4sxa1bfy9.m.pipedream.net/"
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "terrakubeDemo1.pem"
          path: "terrakubeDemo1.pem"
        - key: "terrakubeDemo2.pem"
          path: "terrakubeDemo2.pem"
        - key: "type'
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true

And if you go to your API pods logs you will see the following:

Added 2 additional CA certificate(s) to system truststore
Setting Active Processor Count to 2
Calculating JVM memory based on 5791152K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx5033128K -XX:MaxMetaspaceSize=246023K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 5791152K, Thread Count: 250, Loaded Class Count: 41022, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 126 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=2 -XX:MaxDirectMemorySize=10M -Xmx5033128K -XX:MaxMetaspaceSize=246023K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.7.8)

You can find how to handle ca certs when your container are usig buildpack in the following links:

• https://blog.dahanne.net/2021/02/06/customizing-cloud-native-buildpacks-practical-examples/#Add_certificates_binding_at_runtime:~:text=ca%2Dcertificates-,Add%20certificates%20binding%20at%20runtime,-If%20your%20image
• https://github.com/paketo-buildpacks/ca-certificates
• https://github.com/paketo-buildpacks/spring-boot

I hope this information can help you.

Added information to our documentation. https://docs.terrakube.org/getting-started/deployment/custom-ca-certs

[Diliz]

Humm, I can see that my certs are taken into account with the SERVICE_BINDING_ROOT env var, but it seems that it was an issue with the libjvm at some point, do you use a version >= 1.4.0?

I clearly see the following line:

Added 145 additional CA certificate(s) to system truststore
...
Adding 284 container CA certificates to JVM truststore
WARNING: Unable to add container CA certificates to JVM because /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts is read-only

So I'm pretty sure that the certifates are loaded since I putted them only there, it seems to be working, the only thing I think should be the culprit is the warning here, which seems related to: https://github.com/paketo-buildpacks/libjvm/releases/tag/v1.40.0 (In the enhancement section: Sets relevant paths as group-writable to support Jammy stack)

[alfespa17]

Cool nice to hear that is working for you now 👍.

According to the logs it is using the spring boot build pack 5.23.0

2023-03-06T18:49:29.8931746Z [INFO]     [creator]     Paketo Buildpack for Spring Boot 5.23.0
2023-03-06T18:49:29.8932826Z [INFO]     [creator]       https://github.com/paketo-buildpacks/spring-boot
2023-03-06T18:49:29.8937207Z [INFO]     [creator]       Build Configuration:
2023-03-06T18:49:29.8938426Z [INFO]     [creator]         $BP_SPRING_CLOUD_BINDINGS_DISABLED   false  whether to contribute Spring Boot cloud bindings support
2023-03-06T18:49:29.8939079Z [INFO]     [creator]       Launch Configuration:
2023-03-06T18:49:29.8941931Z [INFO]     [creator]         $BPL_SPRING_CLOUD_BINDINGS_DISABLED  false  whether to auto-configure Spring Boot environment properties from bindings
2023-03-06T18:49:29.8942855Z [INFO]     [creator]         $BPL_SPRING_CLOUD_BINDINGS_ENABLED   true   Deprecated - whether to auto-configure Spring Boot environment properties from bindings
2023-03-06T18:49:30.6903545Z [INFO]     [creator]       Creating slices from layers index
2023-03-06T18:49:30.6915770Z [INFO]     [creator]         dependencies (175.3 MB)
2023-03-06T18:49:30.6922261Z [INFO]     [creator]         spring-boot-loader (281.8 KB)
2023-03-06T18:49:30.6925294Z [INFO]     [creator]         snapshot-dependencies (0.0 B)
2023-03-06T18:49:30.6951907Z [INFO]     [creator]         application (1.0 MB)
2023-03-06T18:49:30.6956718Z [INFO]     [creator]       Launch Helper: Contributing to layer
2023-03-06T18:49:30.6975251Z [INFO]     [creator]         Creating /layers/paketo-buildpacks_spring-boot/helper/exec.d/spring-cloud-bindings
2023-03-06T18:49:30.6983213Z [INFO]     [creator]       Spring Cloud Bindings 1.11.0: Contributing to layer
2023-03-06T18:49:30.7027680Z [INFO]     [creator]         Downloading from https://repo1.maven.org/maven2/org/springframework/cloud/spring-cloud-bindings/1.11.0/spring-cloud-bindings-1.11.0.jar
2023-03-06T18:49:30.7222292Z [INFO]     [creator]         Verifying checksum
2023-03-06T18:49:30.7228254Z [INFO]     [creator]         Copying to /layers/paketo-buildpacks_spring-boot/spring-cloud-bindings
2023-03-06T18:49:30.7235481Z [INFO]     [creator]       Web Application Type: Contributing to layer
2023-03-06T18:49:30.7236216Z [INFO]     [creator]         Servlet web application detected
2023-03-06T18:49:30.7237393Z [INFO]     [creator]         Writing env.launch/BPL_JVM_THREAD_COUNT.default
2023-03-06T18:49:30.7237954Z [INFO]     [creator]       4 application slices
2023-03-06T18:49:30.7238246Z [INFO]     [creator]       Image labels:
2023-03-06T18:49:30.7238693Z [INFO]     [creator]         org.opencontainers.image.title
2023-03-06T18:49:30.7239078Z [INFO]     [creator]         org.opencontainers.image.version
2023-03-06T18:49:30.7239558Z [INFO]     [creator]         org.springframework.boot.version

And I think it is using libjvm 1.41.0 https://github.com/paketo-buildpacks/spring-boot/blob/v5.23.0/go.mod#L11

[Diliz]

Ok found the culprit I think, it seems the api image is using a specific group/user, when trying to add certs with SERVICE_BINDING_ROOT, the certs are not added to the jvm correctly because the folder /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts is read-only for a random user, dunno how it's usually done for springboot/buildpack for other apps in openshift (Only have this issue on openshift, so I think it's the main issue here)

[alfespa17]

Hello @Diliz

There is a second option to add the ca certs at build time instead of at runtime.

Check this: https://github.com/AzBuilder/terrakube/pull/265

There is a folder in the repository where you need to put your ca certs and just build the image following the example build script.

Maybe that approach works for you.

[alfespa17]

Hello @Diliz

I did some changes in the buildpack builder image, now it is using ubuntu jammy, can you test maybe that fix the issue.

According the fix it was for jammy and I was using bionic. https://github.com/paketo-buildpacks/libjvm/releases/tag/v1.40.0

You can test with this version 2.12.0-beta.2

## API properties
api:
  version: "2.12.0-beta.2"
  terraformReleasesUrl: "https://eov1ys4sxa1bfy9.m.pipedream.net/"
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "terrakubeDemo1.pem"
          path: "terrakubeDemo1.pem"
        - key: "terrakubeDemo2.pem"
          path: "terrakubeDemo2.pem"
        - key: "type"
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true

[Diliz]

Oh thanks! I currently added an openshift security context constraint to fix it, as I was expecting, the main culprit was the rights on the cacerts file here: /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts

So I had to use the user 1000 in this case to make it work, which I don't really want to do for the helm chart to be installable as is in an openshift environment :/

EDIT: I'll try the new image as soon as I manage to make the whole other things to work on my side, mostly comprehension on how terrakube is working behind the hood now, how do you define groups for authentication on the api side?

[alfespa17]

Nice to hear that is finally working for you 👍 .

Well at least there is an alternative to make it work in Openshift, we can add that constraint in the documentation and explain the reason.

Terrakube basically take the dex token and check the groups inside the token to validate in which organization the user has access.

[alfespa17]

basically the group inside the token should be the name that you use inside the teams setup https://docs.terrakube.org/user-guide/organizations/team-management

[alfespa17]

Hello @Diliz I am closing this issue, now the helm chart can work out of the box thanks to your suggestions, feel free to open a new issue if you find any new problem