search

telia-oss / concourse-tasks

Common concourse tasks.
MIT License
3 stars 5 forks source link

Feature request: terraform task should be able to recurse over directories and apply #20

Closed itsdalmo closed 4 years ago

itsdalmo commented 5 years ago

It would be nice if the terraform task was able to recurse over subdirectories and run the command over all subdirectories, and even better if we could support glob patterns while doing so. Given that the bash script is getting more and more complex, it might be an idea to write a small Go binary and bake it into a docker image.

In order to be backwards compatible we'd need to have multiple Dockerfiles in the repository, one per terraform version, and rebuild everything on merge to master 😂

mikael-lindstrom commented 5 years ago

I think multiple subdirectories are a bit of an edge case since the majority of projects will deploy to a single account. That said we have several pipelines that does require this.

Another this that we should take into consideration is how we can run things in parallell, the serial approach we are running now does not scale. Should this be a part of this Go binary?

itsdalmo commented 5 years ago

That is a good point! With a Go binary we could spawn multiple go routines and run several terraform tasks concurrently and/or in parallel - the drawback would be that we'd have to somehow sort the output, or they'd become jumbled together?

mikael-lindstrom commented 5 years ago

Yes and how would we handle access/secret keys? Maybe this is outside the scope for this task but we should make sure we have a plan for this before we start implementing this.

itsdalmo commented 5 years ago

Yes and how would we handle access/secret keys?

The easy solution would be that we run all the terraform actions (e.g. apply) with the same set of credentials. So one task could do multiple apply's at once if they were destined for the same account.

Yes and how would we handle access/secret keys? Maybe this is outside the scope for this task but we should make sure we have a plan for this before we start implementing this.

Agreed. I think doing parallel apply (or whatever) needs some more thinking. Should we go ahead and get a POC up and running for a sample Go version and then take it from there?

itsdalmo commented 5 years ago

Oh damn. Access keys and secret keys affect the feature request in the issue as well... 🤔

itsdalmo commented 4 years ago

We've ended up using shell-scripts inside a Taskfile to do this for pull request testing. And for actual plan/apply I think it's better to be explicit instead of recursing 😅 I.e., will not implement.