Open eeaton opened 2 months ago
Very good idea and walkthrough.
I like either option. Currently we stand up just over 50 projects. For clean orgs not already under quota management - there is a minor issue with not project quota but billing quota - the default is 5. At least 2 requests are required to keep the additions under 50 to kick in the 3 min response per request. I think one BU with the 3 example floating/peering/shared options is ok as an example. I vote for option 1 to reduce project count.
However for shared east-west traffic where one BU is an API provider workload for the other BU - 2 of them work well and exercise canary flows. Option 2 is good as well - however the client can dup BU1 at design time.
/Appreciated
Partially addressed by https://github.com/terraform-google-modules/terraform-example-foundation/pull/1241, which had led to a noticeable increase in CI pass rate (though still low). Will continue evaluating if there's a good mechanism to enforce approach 2.
TL;DR
Currently it is difficult to merge even small PR because of flaky CI tests. The CI test pipelines build and destroy the entire environment. This CI test occurs on every pull or merge, runs twice in parallel (to accommodate both network patterns), takes up to 3 hours, and has a high rate of flaky failures.
There are various contributing factors, but it appears there are race conditions that randomly cause the creation of some projects and some service accounts to fail. Some of these errors have been improved with retry logic, but some errors, particulary 409 errors, need more clever logic to unpick the state and retry. While I'm investigating the root cause of those errors, I'm also exploring tactical fixes to improve the success rate of the CI pipeline.
One potential improvement is reducing the number of projects. Every time we create a project there is some small chance of a flaky failure, so for the CI pipeline to pass we currently must get lucky and hope that every project passes. The blueprint creates a large number of projects to illustrate what a real/complex customer environment might look like. However, many of the projects are placeholders or duplicates. So by reducing the number of projects that don't add substantive value for the architecture or illustrate how to use terraform modules, we can reduce the chance of flaky errors preventing the CI test from passing.
Terraform Resources
Detailed design
Additional information
No response