search

AzBuilder / terrakube

Open source IaC Automation and Collaboration Software.
https://docs.terrakube.io
Apache License 2.0
529 stars 47 forks source link

Use plugin cache dir with ephemeral agents #1582

Open BenjaminDecreusefond opened 1 day ago

BenjaminDecreusefond commented 1 day ago

Feedback

Hi !

We tried to follow this documentation to use cache dir with ephemeral agents. Since this settings is made at the executor level we are not sure that effectively taken into account when a new ephemeral job is started. Beside that, using the option TF_PLUGIN_CACHE_DIR=/home/cnb/.terraform.d/plugin-cache in a workspace running on ephemeral agents throw error in plan Error: The specified plugin cache dir /home/cnb/.terraform.d/plugin-cache/ cannot be opened: stat /home/cnb/.terraform.d/plugin-cache/: no such file or directory even though after checking in the ephemeral job that the folder actually exists. Would you know what we've missed ?

Regards !

alfespa17 commented 1 day ago

It make sense that it is failing because this volume information is not mounted in the ephemeral executor here.

You could use TF_PLUGIN_CACHE_DIR=/home/cnb/ that directory exist inside the container, the other/home/cnb/.terraform.d/plugin-cache is only created inside the container when the volume is mounted using the below information:

executor:
  volumes:
    - name: cache-volume
      emptyDir:
        sizeLimit: 1024Mi
  volumeMounts:
  - mountPath: /home/cnb/.terraform.d/plugin-cache
    name: cache-volume

I don't think using the terraform plugin cache will have any benefits inside the ephemeral executor because even if you add the code to mount an Empty dir to the container the volume will be empty every time the container is starting so terraform/tofu will have to download all the providers information every time.

The only way that maybe could work if you mount some kind of share storage like a file share that can be mounted across all the ephemeral executors instead of using EmptyDir

I hope this help to clarify why is failing with the above error

BenjaminDecreusefond commented 1 day ago

Yeah ! that's what I was thinking about :/ Thanks you ! I'll try to figure out how we can mitigate those storage issues !

BenjaminDecreusefond commented 1 day ago

Btw, I tested the toleration feature and it works good !

alfespa17 commented 1 day ago

Btw, I tested the toleration feature and it works good !

Cool, I was planing to release the new version tomorrow by the end of the day, let me know if you find any other issue or if you have any suggestion to add to the new version

BenjaminDecreusefond commented 1 day ago

I'm also curious to know. When we run an ephemeral agents, it uses the storage of the executor right ? Would it make sense to create several executor to have different disk space ?

alfespa17 commented 1 day ago

I'm also curious to know. When we run an ephemeral agents, it uses the storage of the executor right ? Would it make sense to create several executor to have different disk space ?

It will use the container storage, it is writing files to the home directory for user cnb