Added ability to add customisation scripts for terraform

[garethjevans]

This PR adds the ability to apply extra terraform scripts when performing a bbl up. This is useful when using those scripts to provision extra infrastructure such as nat-gateways, cloud sql instances etc.

[cfdreddbot]

Hey garethjevans!

Thanks for submitting this pull request! I'm here to inform the recipients of the pull request that you and the commit authors have already signed the CLA.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/154499054

The labels on this github issue will be updated when the story is started.

[dsabeti]

Hi @garethjevans. Thanks for the PR. Since this is a breaking change to the interface -- it requires all Concourse users to configure an additional input, which they may not even technically need it -- we'll probably hold off on merging for a bit. Otherwise, this is probably a worthwhile change in the long-term.

[garethjevans]

Thanks @dsabeti, my concourse knowledge is fairly new - is there a way of making an input optional? or a better approach for doing this?

[garethjevans]

ah looks like there is - https://github.com/concourse/concourse/issues/1923

[dsabeti]

Hey, look at that! The feature hasn't been shipped in an official release of Concourse yet, so we'd still have to wait until that, at the very least. More importantly, I have a meeting in (literally) ten minutes with the team that develops bbl to discuss Concourse-ability of bbl in general, so we may come out with a different strategy. I'll keep this thread posted.

[dsabeti]

Hi @garethjevans. After discussing with the infrastructure team (which makes bbl), I think we're moving toward using the env-repo/bbl-state as the mechanism for delivering customizations. For example, if you want to add an additional terraform template, you can simply drop the additional template into the terraform directory and bbl will automatically include it when it paves infrastructure.

Have you given that workflow a shot? If so, is there a reason you prefer using the Concourse task interface?

[garethjevans]

@dsabeti that's actually what this PR uses, it copies the resources into that terraform directory after the bbl plan but before the bbl up.

Is that the workflow you are describing? Or are you suggesting copying them into the env-repo/bbl-state/terraform directory before running the bbl-up task

[dsabeti]

Yeah, I was curious if you'd tried updating the env-repo prior to running the bbl-up task. The reason we're asking is that, with bbl 6, we have to re-work the solution for the director ops-files (the --ops-file flag has been removed). I was curious how burdensome it would be to manage all bbl customizations through the env-repo instead of the concourse task.

[garethjevans]

@dsabeti I've not tried that approach, I'll give that a go and get back to you

[garethjevans]

Hi @dsabeti, I've done quite a bit of testing using the suggested approach today and can confirm it works nicely, want me to close this PR?

I'd also be very interested to understand how ops-files will be added with bbl6 like you mentioned above, not sure where the best place is to have that conversation!

[dsabeti]

Thanks for the feedback @garethjevans. Yeah, we can go ahead and close this PR. For reference, we've go this release marker in Pivotal Tracker for having the concourse tasks allow straightforward use of bbl6. This story is the main one to follow.