Split VCS using a plugin based architecture

[nitrocode]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

• [ ] I'd be willing to implement this feature (contributing guide)

Describe the user story

It would be nice to separate VCS code from this repo using a plugin based architecture.

This has been successful in projects such as terraform and tflint.

This would give us the following benefits

• reduce code in atlantis repo
• faster builds and tests of atlantis
• allow maintainers to split github, gitlab, bitbucket, azuredevops, etc into its own repos
  • the repo may not even be managed by maintainers since most of the maintainers are on github
• allow users and devs to contribute their own custom VCS providers

Describe the solution you'd like

1. Evaluate
   1. golang's plugin system
   2. separate golang modules
      1. This would have isolation. One drawback would be that when a module like runatlantis/atlantis-vcs-github is updated, then runatlantis/atlantis's go.mod would have to be updated to use the new version. This would be a 2 PR system and may make e2e testing more difficult.
   3. https://github.com/hashicorp/go-plugin (grpc)
      1. This would have true isolation but is not well documented.
      2. used by hashicorp terraform providers
      3. used by tflint rulesets https://github.com/terraform-linters/tflint-plugin-sdk
   4. other methods
2. Decide on a method
3. Identify a VCS to split up i.e. github
4. Isolate VCS code in this repo
5. Create a separate repo for the VCS in runatlantis org
6. Enable auto-downloading the VCS to this repo
7. Remove old VCS code from this repo
8. Release atlantis
9. Repeat for other VCSs

Describe the drawbacks of your solution

Some effort

Describe alternatives you've considered

Do nothing (status quo)

[acastle]

Couple comments on the different approaches to plugins here:

golang's plugin system From my experience go's plugin system has a pretty major limitation that makes it very difficult to work with. Specifically, plugins must all be built from the exact same version of the go toolchain as the host system. If I am not mistaken, they also require that any package dependencies are versioned in lockstep between the plugin and host. This feels to me like it wouldn't be a great fit since it comes with most the same downsides of the code living in the same repository.

separate golang modules This is definitely the closest approach to the current structure. My only concern here is that this doesn't quite get us to the stated goals. It would reduce the LOC in atlantis true and, by virtue of the VCS specific tests moving to their repos with them, would reduce the test run time. The module code would still ultimately need to be included in the atlantis build and not have any real impact on build times for Atlantis itself. In some respects this also makes the prospect of including custom VCS providers a bit more of a black box than they are currently. (As it is now just a version bump and likely a include _ at the root of the Atlantis server)

go-plugin Since this creates a true process boundary between plugin and host, it seems to be the most in line with the stated goals. This is also most likely the biggest change for atlantis users as we will need to come up with a method of finding and "installing" these plugins. This could be as simple as a known location on disk to scan on startup, and we could likely make this change transparent to customers by including the current, in-tree, plugins in our docker images. Anyone who is just consuming the Atlantis binary we release directly would need an outlined upgrade path laid out for them to not be broken by this type of change.

My 2c - embracing go-plugin is the way to go here