Bug when using configMap with ephemeral agents

[BenjaminDecreusefond]

Bug description 🐞

Hi !

Following the PR I implemented, it appears that the configMap is not mounted on the job when it's running. I've set the TERRAKUBE_ENABLE_EPHEMERAL_EXECUTOR, EPHEMERAL_CONFIG_MAP_NAME, EPHEMERAL_CONFIG_SERVICE_ACCOUNT and the EPHEMERAL_CONFIG_MAP_MOUNT_PATH but no volumes are mounted. I tripled check the new code I added but it seem correct to me and I'm having hard time to trouble shot where it comes from.

I'm still trying to figure out what is happening, if you have any ideas I'd be happy to hear !

thanks !

Steps to reproduce

Try to run an EPHEMERAL_JOB with the parameter above, no volume are mounted on the ephemeral job's container.

Expected behavior

No response

Example repository

No response

Anything else?

No response

[BenjaminDecreusefond]

Hello @alfespa17 !

Do I have good understanding of these values by setting workspace environment variables like ? Or we do need to use them differently ? From what i can see in the kubectl describe job-<job_id>

Containers:
  job-376:
    Environment:
      EphemeralFlagBatch:           true
      EphemeralJobData:   data
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:          my-region
      AWS_REGION:                   my-region
      AWS_ROLE_ARN:                 my-role
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  kube-api-access-46wdc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true

No environment variables like EPHEMERAL_CONFIG_MAP_NAME are set so maybe I misunderstood something ?

Regards!

[alfespa17]

Hello @alfespa17 !

Do I have good understanding of these values by setting workspace environment variables like ? Or we do need to use them differently ? From what i can see in the kubectl describe job-<job_id>

Containers:
  job-376:
    Environment:
      EphemeralFlagBatch:           true
      EphemeralJobData:   data
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:          my-region
      AWS_REGION:                   my-region
      AWS_ROLE_ARN:                 my-role
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  kube-api-access-46wdc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true

No environment variables like EPHEMERAL_CONFIG_MAP_NAME are set so maybe I misunderstood something ?

Regards!

Did you update the value in ExecutorEphemeralImage???

[alfespa17]

From you logic here, I think it should work

https://github.com/AzBuilder/terrakube/blob/3114c2bf9fd768120216cf4aa5f5aaccb227e599/api/src/main/java/org/terrakube/api/plugin/scheduler/job/tcl/executor/ephemeral/EphemeralExecutorService.java#L88

You only need to add the environment values EPHEMERAL_CONFIG_MAP_NAME and EPHEMERAL_CONFIG_MAP_MOUNT_PATH

[BenjaminDecreusefond]

Hello @alfespa17 ! Do I have good understanding of these values by setting workspace environment variables like ? Or we do need to use them differently ? From what i can see in the kubectl describe job-<job_id>

Containers:
  job-376:
    Environment:
      EphemeralFlagBatch:           true
      EphemeralJobData:   data
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:          my-region
      AWS_REGION:                   my-region
      AWS_ROLE_ARN:                 my-role
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  kube-api-access-46wdc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true

No environment variables like EPHEMERAL_CONFIG_MAP_NAME are set so maybe I misunderstood something ? Regards!

Did you update the value in ExecutorEphemeralImage???

Yep ! I'm using this config for the API

api:
  defaultDatabase: false
  loadSampleData: false
  serviceAccountName: ${svc_name}
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  - name: JAVA_OPTS
    value: "-Xmx512m -XX:MaxMetaspaceSize=1024m"
  - name: ExecutorEphemeralNamespace
    value: terrakube
  - name: ExecutorEphemeralImage
    value: azbuilder/executor:2.24.0-beta.5
  - name: ExecutorEphemeralSecret
    value: terrakube-executor-secrets

[BenjaminDecreusefond]

From you logic here, I think it should work

https://github.com/AzBuilder/terrakube/blob/3114c2bf9fd768120216cf4aa5f5aaccb227e599/api/src/main/java/org/terrakube/api/plugin/scheduler/job/tcl/executor/ephemeral/EphemeralExecutorService.java#L88

You only need to add the environment values EPHEMERAL_CONFIG_MAP_NAME and EPHEMERAL_CONFIG_MAP_MOUNT_PATH

You mean through the helm release or through the workspace environment variables ?

When doing a printenv inside the job container I do not see any of my ephemeral variables

[alfespa17]

Workspace environment variables

[alfespa17]

Did you try killing the API pod?

[BenjaminDecreusefond]

I'll try to reinstall it !

[BenjaminDecreusefond]

I didn't fix it :/

[alfespa17]

I didn't fix it :/

Try checking the Kubernetes job specification using kubectl maybe you can find what is missing there

[BenjaminDecreusefond]

Figured it out ! Actually need to set the API to the same image as the executor ! thanks !