Busywork: Update build process and make CICD accessible by adding docs.

[maunzCache]

During preparation of #2180 i noticed some issue regarding the CICD pipeline which does the testing for this project. To cut it short in this introduction i'd generally suggest to write some documentation how CICD works for this project and how to contribute to it. Additionally prepare a roadmap on how to update all CICD components that are outdated:

• Clarify if both appveyor.yml and .circleci/config.yml are required. I rarely know the services but it looks to me that AppVeyor was replaced by CircleCI. Fixed in #2194
• Have a guide in docs/ that explains contribution to the CICD pipeline which does the testing and building. Initial focus would be which files have to be updated and how to upgrade runtimes such as a new Terraform or Go version.
  • Links to existing guides would be fine. We don't need to copy & paste CircleCI docs here.
• Consider updating components used in CICD to ensure support and decrease security issues.
  • Even though everything is containerized does not mean this project cannot be harmed by security flaws of some libs.
  • In .circleci/config.yml
  • Check if the windows orb is up-to-date
  • Give access to the test image or at least reference its source so that we can openly contribute to it.
  • Consider updating pre-commit pip to next major, including all dependencies. Python 3.6 is EOL so everything should be compatible with supported versions
    • Also update the gruntworks pre-commit repository version. This project uses 0.1.10 but 0.1.17 is latest and contains relevant changes for terraform-fmt
  • In gofmtcheck.sh update script to reflect changes in terraform. They have removed the vendor specific lookup.
  • I have seen some additional checks for this somewhere else.
  • Maybe partially relevant: Update the docs generation docs/Dockerfile and docs/Gemfile
  • The container uses Ruby 2.6 which is EOL since three years. Migrate this to Ruby 3.
  • Of course all gems have to be updated and checked. If there is issues with updating this, write docs that reflect this blocks e.g. create a new issue here

[yorinasub17]

Thanks for your effort in improving our CI/CD flow!

You bring up a lot of good points. Here are a few top level comments that come to mind:

• Regarding appveyor v. circleci, appveyor was a failed experiment so we can remove it to reduce confusion: https://github.com/gruntwork-io/terragrunt/pull/2194
• The bulk of our CI infrastructure (specifically, the base docker image) is using scripts from our private repo from our proprietary library, which makes it very difficult to open it up for public collaboration. I suspect that a good number of confusion comes from that. We currently don't have plans to open this up. In the future, we may be open to adjusting our testing infrastructure to only use open source, but at this moment in time that is not feasible for our small team to maintain.
• The changes for terraform-fmt in pre-commit are actually not relevant for this repo. That said, agree that it is a good idea to keep those up to date as well. Which brings me to...
• We are internally working on a new product to maintain these version updates in a more meaningful manner, and keeping it up to date over time, which we plan to roll out here, which should help with the story of keeping a lot of these things up to date.

Hope this makes sense! Happy to keep discussing options for better collaborating on this despite the challenges above! In the meantime, any PRs that propose version bumps in dependabot fashion (e.g., a PR that bumps versions in the docker files and gemfiles for the docs folder) is much appreciated, and would be accepted.

[maunzCache]

Thank you for the response and fast removal of the appveyor. This got me pretty confused.

I'd like to keep this issue open until you released your fancy stuff then and see if there are still open issues to discuss or if my lists shrinks with that. If i or someone else has a nice solution for the above, feel free to pursue. And still docs are always good so i'd like to put your last words regarding bot PRs somewhere into the contribution docs.