search

AzBuilder / terrakube-helm-chart

Helm chart to install Terrakube in any Kubernetes cluster
Apache License 2.0
30 stars 19 forks source link

Backend configuration help #124

Open MARVELMafia opened 6 days ago

MARVELMafia commented 6 days ago

So I am using CEPH S3 as my storage backend and this is the config:

storage:
    existingSecret: terrakube-minio
    defaultStorage: true
    minio:
      bucketName: "iaac-infrastructure"
      endpoint: "http://cephs3endpoint"
    default:
      endpoint: "http://cephs3endpoint"
minio:
    existingSecret: terrakube-minio
    defaultBuckets: iaac-infrastructure

I get the following error on the output on terrrakube gui:

image

I get the following error on my api

2024-06-25 16:32:07.168 ERROR 1 --- [nio-8080-exec-9] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : S3 Not found: InvalidAccessKeyId (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: tx000007c1dc09c1a2b8a4b-00667a6467-7c2e0c7-default; S3 Extended Request ID: 7c2e0c7-default-default; Proxy: null)

But when I update the configuration to (when I directly specify the minio.auth.rootPassword and minio.auth.rootUser):

storage:
    existingSecret: terrakube-minio
    defaultStorage: true
    minio:
      bucketName: "iaac-infrastructure"
      endpoint: "http://xxx"
    default:
      endpoint: "http://xxx"
minio:
    auth:
      rootPassword: abc
      rootUser: xyz
    defaultBuckets: iaac-infrastructure

I get the following error on the api:

2024-06-25 02:42:29.096 ERROR 1 --- [io-8080-exec-10] o.t.a.p.s.aws.AwsStorageTypeServiceImpl : S3 Not found: NoSuchKey (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: tx00000cfcf6be4b62e8332-00667a2eec-7c2e0c7-default; S3 Extended Request ID: 7c2e0c7-default-default; Proxy: null)

BUT it works fine and the job on the terrakube gui runs properly and the iaac gets applied. So I am thinking the minio.auth.{} is not being able to get the value from the secret terrakube-minio properly.

Can somoene tell me how do I use the secret in the minio.auth.{} configuration?

I checked the terrakube-minio deployment and it says: image

My secret terrakube-minio is getting created properly with keys username : root-user and password : root-password as below: image

what am I missing?

MARVELMafia commented 5 days ago

I created the secrets manually and now am facing the next hurdle with the following error on the executor:

Where does this token get generated from and what is this used for? Currently I have a valid InternalSecret string in the secret which is being used by the executor. How does this executor workflow actually run? There is not enough info at https://docs.terrakube.io/user-guide/reference/executor/terraform-execution-flow

[threadPoolTaskExecutor-1] ERROR org.terrakube.client.dex.DexCredentialAuthentication - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] INFO org.terrakube.client.dex.DexCredentialAuthentication - Authentication error 401
[threadPoolTaskExecutor-1] ERROR org.terrakube.client.dex.DexCredentialAuthentication - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] ERROR org.springframework.aop.interceptor.SimpleAsyncUncaughtExceptionHandler - Unexpected exception occurred invoking async method: public void org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(org.terrakube.executor.service.mode.TerraformJob)
feign.RetryableException: Too many follow-up requests: 21 executing GET http://terrakube-api-service:8080/api/v1/organization/4edae022-e858-4f91-97c0-ecc4120cb2b0/job/10?include=step
        at feign.FeignException.errorExecuting(FeignException.java:268)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:131)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:91)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
        at jdk.proxy2/jdk.proxy2.$Proxy89.getJobById(Unknown Source)
        at org.terrakube.executor.service.status.UpdateJobStatusImpl.setRunningStatus(UpdateJobStatusImpl.java:37)
        at org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(ExecutorJobImpl.java:47)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ProtocolException: Too many follow-up requests: 21
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:127)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at org.terrakube.client.dex.DexCredentialAuthentication.intercept(DexCredentialAuthentication.java:79)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
        at feign.okhttp.OkHttpClient.execute(OkHttpClient.java:180)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:121)
        ... 17 more
alfespa17 commented 4 days ago

Hello @MARVELMafia , I think this could work for you and the helm chart to setup your cephs3 storage

storage:
  defaultStorage: false
  minio:
    accessKey: "XXXX"
    secretKey: "XXXX"
    bucketName: "XXXX"
    endpoint: "http://cephs3endpoint"

Dont forget to put "defaultStorage: false", the example config can be found here

Those are the values that you need as you can see in this part of the code.

https://github.com/AzBuilder/terrakube-helm-chart/blob/9b2d816e7a958f9714b66e70854a4b4e982c1cb5/charts/terrakube/templates/secrets-api.yaml#L59

alfespa17 commented 4 days ago

I created the secrets manually and now am facing the next hurdle with the following error on the executor:

Where does this token get generated from and what is this used for? Currently I have a valid InternalSecret string in the secret which is being used by the executor. How does this executor workflow actually run? There is not enough info at https://docs.terrakube.io/user-guide/reference/executor/terraform-execution-flow

[threadPoolTaskExecutor-1] ERROR org.terrakube.client.dex.DexCredentialAuthentication - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] INFO org.terrakube.client.dex.DexCredentialAuthentication - Authentication error 401
[threadPoolTaskExecutor-1] ERROR org.terrakube.client.dex.DexCredentialAuthentication - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] ERROR org.springframework.aop.interceptor.SimpleAsyncUncaughtExceptionHandler - Unexpected exception occurred invoking async method: public void org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(org.terrakube.executor.service.mode.TerraformJob)
feign.RetryableException: Too many follow-up requests: 21 executing GET http://terrakube-api-service:8080/api/v1/organization/4edae022-e858-4f91-97c0-ecc4120cb2b0/job/10?include=step
        at feign.FeignException.errorExecuting(FeignException.java:268)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:131)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:91)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
        at jdk.proxy2/jdk.proxy2.$Proxy89.getJobById(Unknown Source)
        at org.terrakube.executor.service.status.UpdateJobStatusImpl.setRunningStatus(UpdateJobStatusImpl.java:37)
        at org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(ExecutorJobImpl.java:47)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ProtocolException: Too many follow-up requests: 21
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:127)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at org.terrakube.client.dex.DexCredentialAuthentication.intercept(DexCredentialAuthentication.java:79)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
        at feign.okhttp.OkHttpClient.execute(OkHttpClient.java:180)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:121)
        ... 17 more

The executor needs to communicate with the API, the API requires authentication so to communicate with the API the Executor component generates a JWT that is signed with the security.internalSecret that you put in your helm chart.

I am not really sure why you are getting the following:

feign.RetryableException: Too many follow-up requests: 21 executing GET http://terrakube-api-service:8080/api/v1/organization/4edae022-e858-4f91-97c0-ecc4120cb2b0/job/10?include=step

It looks like is failing in this part of the code when the executor receive the request from the API and tries to update the job status

https://github.com/AzBuilder/terrakube/blob/e585cebbf6280065199e1516428487f253e4fb1d/executor/src/main/java/org/terrakube/executor/service/status/UpdateJobStatusImpl.java#L34

You mentioned that you created the secret manually so I guess you could check the value for "AzBuilderApiUrl" by default it is using the following value "http://terrakube-api-service:8080" if the executor is deployed in the same kubernetes namespace

alfespa17 commented 4 days ago

I created the secrets manually and now am facing the next hurdle with the following error on the executor: Where does this token get generated from and what is this used for? Currently I have a valid InternalSecret string in the secret which is being used by the executor. How does this executor workflow actually run? There is not enough info at https://docs.terrakube.io/user-guide/reference/executor/terraform-execution-flow

[threadPoolTaskExecutor-1] ERROR org.terrakube.client.dex.DexCredentialAuthentication - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] INFO org.terrakube.client.dex.DexCredentialAuthentication - Authentication error 401
[threadPoolTaskExecutor-1] ERROR org.terrakube.client.dex.DexCredentialAuthentication - Generate Dex Authentication Private Token
[threadPoolTaskExecutor-1] ERROR org.springframework.aop.interceptor.SimpleAsyncUncaughtExceptionHandler - Unexpected exception occurred invoking async method: public void org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(org.terrakube.executor.service.mode.TerraformJob)
feign.RetryableException: Too many follow-up requests: 21 executing GET http://terrakube-api-service:8080/api/v1/organization/4edae022-e858-4f91-97c0-ecc4120cb2b0/job/10?include=step
        at feign.FeignException.errorExecuting(FeignException.java:268)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:131)
        at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:91)
        at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
        at jdk.proxy2/jdk.proxy2.$Proxy89.getJobById(Unknown Source)
        at org.terrakube.executor.service.status.UpdateJobStatusImpl.setRunningStatus(UpdateJobStatusImpl.java:37)
        at org.terrakube.executor.service.executor.ExecutorJobImpl.createJob(ExecutorJobImpl.java:47)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ProtocolException: Too many follow-up requests: 21
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:127)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at org.terrakube.client.dex.DexCredentialAuthentication.intercept(DexCredentialAuthentication.java:79)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
        at feign.okhttp.OkHttpClient.execute(OkHttpClient.java:180)
        at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:121)
        ... 17 more

The executor needs to communicate with the API, the API requires authentication so to communicate with the API the Executor component generates a JWT that is signed with the security.internalSecret that you put in your helm chart.

I am not really sure why you are getting the following:

feign.RetryableException: Too many follow-up requests: 21 executing GET http://terrakube-api-service:8080/api/v1/organization/4edae022-e858-4f91-97c0-ecc4120cb2b0/job/10?include=step

It looks like is failing in this part of the code when the executor receive the request from the API and tries to update the job status

https://github.com/AzBuilder/terrakube/blob/e585cebbf6280065199e1516428487f253e4fb1d/executor/src/main/java/org/terrakube/executor/service/status/UpdateJobStatusImpl.java#L34

You mentioned that you created the secret manually so I guess you could check the value for AzBuilderApiUrl in your executor secrets by default it is using the following value "http://terrakube-api-service:8080" if the executor is deployed in the same kubernetes namespace

alfespa17 commented 4 days ago

You could try connecting to your executor pod and do a curl to the API, if you see a 401 error when you do a curl to GET http://terrakube-api-service:8080/api/v1/organization/4edae022-e858-4f91-97c0-ecc4120cb2b0 the communication should be working

MARVELMafia commented 4 days ago 
storage:
  defaultStorage: false
  minio:
    accessKey: "XXXX"
    secretKey: "XXXX"
    bucketName: "XXXX"
    endpoint: "http://cephs3endpoint"

This is not enough. I tried this and still got the

Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId

error on the api.

So I had to add the following config as well

minio:
  auth:
    rootPassword: "XXX"
    rootUser: "XXX"
  defaultBuckets: "iaac-infrastructure"

And then I got the following error

AwsStorageTypeServiceImpl : S3 Not found: NoSuchKey (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey)

But the Plan executed. image

alfespa17 commented 4 days ago

These are the only values need it to setup MINIO for example for the API as you can see here.

storage:
  defaultStorage: false
  minio:
    accessKey: "XXXX"
    secretKey: "XXXX"
    bucketName: "XXXX"
    endpoint: "http://cephs3endpoint"

The code can be found here:

https://github.com/AzBuilder/terrakube-helm-chart/blob/9b2d816e7a958f9714b66e70854a4b4e982c1cb5/charts/terrakube/templates/secrets-api.yaml#L59

You only need this part when you use "defaultStorage=true"

minio:
  auth:
    rootPassword: "XXX"
    rootUser: "XXX"
  defaultBuckets: "iaac-infrastructure"

You can check the following helm chart file:

https://github.com/AzBuilder/terrakube-helm-chart/blob/9b2d816e7a958f9714b66e70854a4b4e982c1cb5/charts/terrakube/templates/secrets-api.yaml#L30