search

navapbc / template-infra

A template to set up foundational infrastructure for your application in AWS
Apache License 2.0
9 stars 2 forks source link

Enable parallel runs of template-only-ci-infra.yml #658

Open lorenyu opened 2 weeks ago

lorenyu commented 2 weeks ago

Now that the OIDC has been moved out of terraform and into the set-up-current-account script, which only creates the OIDC if it doesn't already exist, this unblocks the ability to enable template only CI runs in parallel.

Some other things that need to happen before we can do this:

  1. Make each run of template-only-ci-infra use a different project name, rather than a hardcoded one, so that resources don't conflict
  2. Make all resource names unique by utilizing the project name, such as the ECS cluster's name

Still not 100% sure it's a good idea. Due to the fact that template-only-ci-infra can be a bit brittle, sometimes it's nice that we can't run too many instances of it at a time, so we don't end up with a big cleanup job if it is failing to clean up for some reason.

lorenyu commented 2 weeks ago

cc @rocketnova just realized that this might be possible now, which could be nice since these workflows take 20 minutes to run

rocketnova commented 1 week ago

@lorenyu OIDC has moved out of terraform? I thought it was still https://github.com/navapbc/template-infra/tree/main/infra/modules/auth-github-actions?

lorenyu commented 1 week ago

it was moved to set-up-current-account here. it was a change that krista introduced on another project that i ported over.

auth-github-actions still creates the role and iam policy, but the oidc provider is now a data bock

rocketnova commented 1 week ago

it was moved to set-up-current-account here. it was a change that krista introduced on another project that i ported over.

auth-github-actions still creates the role and iam policy, but the oidc provider is now a data bock

Oh interesting. I see. 👍

Still not 100% sure it's a good idea. Due to the fact that template-only-ci-infra can be a bit brittle, sometimes it's nice that we can't run too many instances of it at a time, so we don't end up with a big cleanup job if it is failing to clean up for some reason.

I'm not positive it's a good idea either. Based on the docs, it looks like we cannot set a limit on concurrency (e.g. only allow max 3 jobs within this concurrency group to run). I think that even though the jobs take a long time, they do feel like a pretty good safety check that's worth waiting for.

lorenyu commented 1 week ago

k will leave this ticket in the backlog then