Support Git Submodules - Githubissues

osterman commented 5 years ago

what

Support git submodules

why

It appears that clones are not recursive: https://github.com/runatlantis/atlantis/blob/f057d6d7d9a76c7b8d85c874a10c4f25f7be7465/server/events/working_dir.go#L113
Support importing terraform root modules without needing terragrunt

use-case

We're trying to adapt our deployment methodology to work well with Atlantis
We keep all of our "root" modules in one repo called terraform-root-modules
We then import these modules into various stages (aka environments or accounts)
We have been using docker multi-stage, but that won't play well with atlantis detecting changes, so we're willing to compromise
We want to use git submodules instead, with the expectation that atlantis would be able to detect changes for autoplans

@lkysow do you think this would work?

Technically, looks very easy to implement here and we'd be happy contribute the capability.

lkysow commented 5 years ago

Yeah that looks good! Please push a PR. Can you also look into if there are any unintended consequences?

lkysow commented 5 years ago

We should version detect git though and make sure we use the right command for the right version.

tomer-1 commented 5 years ago

@osterman a more "backward compatible" approach to this would be:

"cd" into the repo
run "git submodules update --init --recursive" that'll do for all repos, even ones not containing any submodules. but this operation should be coupled with any "git clone" operation. i'm not aware of the way atlantis manages the repositories after the first clone. but if it's doing any git manipulations to prevent re-clones, then those manipulations should contain the steps presented above to achieve an up to date working copy.

osterman commented 5 years ago

The problem with this approach is we cannot detect changes automatically, which is why we implemented custom support for it in our fork.

tomer-1 commented 5 years ago

The problem with this approach is we cannot detect changes automatically, which is why we implemented custom support for it in our fork.

could you please elaborate how did you manage implement this? we have many environments and many repositories, and there are some things that implementing as a submodules is the best way to go

osterman commented 5 years ago

@tomer-1 we ended up having to compromise. Our change detects if a submodule folder is modified and then executes. We were not able to detect changes inside the submodule due to the current implementation relying on the files changed as returned by the github API and not the files changed using git calculus.

osterman commented 5 years ago

@aknysh can add more details

tomer-1 commented 5 years ago

@osterman i understand you had to compromise for the changes in the modules, what i don't fully understand is how did you manage to make atlantis sync the submodules after the clone?

osterman commented 5 years ago

@tomer-1 did you have a chance to look at the PRs where we implement it?

https://github.com/cloudposse/atlantis/pull/11 https://github.com/cloudposse/atlantis/pull/18 https://github.com/cloudposse/atlantis/pull/19

@aknysh can answer any specific questions about these.

krishnanandchoudhary commented 4 years ago

@osterman HI, I am exploring atlantis to use, While the repo has submodules for terraform vars file, it does not checkout submodules. Wondering how to make it work. Thanks

goobysnack commented 4 years ago

I'm blocked by this too. Can we add extra args for the git clone so that it can --recurse-submodules TIA.

lkysow commented 4 years ago

I'm blocked by this too. Can we add extra args for the git clone so that it can --recurse-submodules TIA.

Does this work? @osterman was saying that that's not enough because the github api doesn't return the list of files changed in the submodule so Atlantis won't know that the files in the submodule have changed and it won't trigger a plan.

goobysnack commented 4 years ago

Are there extra args so that we can configure git in our server config? So we can add --recurse-submodules if we need to.

lkysow commented 4 years ago

Are there extra args so that we can configure git in our server config? So we can add --recurse-submodules if we need to.

No, you can't add extra args to the git clone command.

goobysnack commented 4 years ago

@lkysow I figured this one out too. I added this to my plan workflow:

- run: git submodule update --init --recursive && ...<removed>

dcatalano-figure commented 3 years ago

another submodule usecase to support is forcing a plan b/c of symlinks ... we have common terraform code IE. tfvars, in a directory that also happens to be a git submodule. We are able to check out the submodule with a custom workflow as described above. There are files in the submodule that are shared / accesses via a symlink in each respective directory. Of course the symlink doesn't change so there is nothing to trigger the plan.

I'd be really awesome to be able to force a plan via GitHub regardless if Atlantis thinks there is a change or not.

ganeshk1928 commented 2 years ago

Is there any update on this one?

dgokcin commented 2 years ago

run: git submodule update --init --recursive && ...<removed

@lkysow do you know how to make this work with a github app? any suggestions

mnhat3896 commented 2 years ago

To workaround, here is my configuration, I did a test with Atlantis on local and it worked. hope it can help somehow. --remote will always pull from the submodule repo, so you don't need to run submodule update on your root module and push the change

dgokcin commented 2 years ago

@mnhat3896 did you use a github app or your PAT to authenticate between atlantis and your github account on this test?

mnhat3896 commented 2 years ago

@dgokcin Sorry for the response late. No, I was using the ssh key, you just need to add your ssh key to the personal setting, don't use HTTPS in .gitmodules config file. when running the plan. Atlantis will expose your username/password as the plain-text. ==> Atlantis does not work well with HTTPS protocol because of the security risk above

dgokcin commented 2 years ago

@dgokcin Sorry for the response late.

No, I was using the ssh key, you just need to add your ssh key to the personal setting, don't use HTTPS in .gitmodules config file. when running the plan. Atlantis will expose your username/password as the plain-text.

==> Atlantis does not work well with HTTPS protocol because of the security risk above

hmm. can i do this with a github app? I want to avoid using a PAT or anything personal.

ganeshk1928 commented 2 years ago

@dgokcin Sorry for the response late. No, I was using the ssh key, you just need to add your ssh key to the personal setting, don't use HTTPS in .gitmodules config file. when running the plan. Atlantis will expose your username/password as the plain-text. ==> Atlantis does not work well with HTTPS protocol because of the security risk above

hmm. can i do this with a github app? I want to avoid using a PAT or anything personal.

@dgokcin Even before you run the plan/init or submodule commands, you have to authenticate with Github and you can do that via a Github app too. Please check this for reference on how to authenticate using Github app.

mnhat3896 commented 2 years ago

well, yes indeed, as @ganesh-katakam-T18451 said you have to authenticate with VCS. The thing I mentioned above about submodule. when you run git submodule add ... will decide your protocol for Atlantis run in the plan. for example, if you run git submodule add https://github.com/<YOUR_REPO_URL>. then Atlantis will use HTTPS to run git submodule init and expose your authentication like clone of 'https://userA:password@github.com. therefore using SSH protocol if possible

FRABUCHI commented 1 year ago

@dgokcin If you are using the github app and using --write-git-creds, .gitconfig and .gitcredential will be created in the /home/atlantis path. The contents of gitconfig should be in the following path.

[url "https://x-access-token@[your git address]"]
    insteadOf = ssh://git@[your git ssh address]

If I write gitconfig like that code, the request will go to https.

[url "https://x-access-token@[your git address]/"]
    insteadOf = git@[your git ssh address]:

The contents of this file may vary depending on how you use git submodules.

bml1g12 commented 1 year ago

In case it's useful, when using git app with --write-git-creds, I found the following allows me to use git submodules:

workflows:
  default:
    plan:
      steps:
        - run: |
            #!/bin/ash
            cat >/home/atlantis/.gitconfig <<EOF
            [credential]
                    helper = store
            [url "https://x-access-token@github.com"]
                    insteadOf = ssh://git@github.com
            [url "https://x-access-token@github.com/"]
                    insteadOf = git@github.com:
            EOF
        - run: echo "init submodule" && git submodule init
        - run: echo "updating submodule" && git submodule update --init --recursive

bdellegrazie commented 3 weeks ago

For github apps, see also https://github.com/runatlantis/atlantis/issues/4704

runatlantis / atlantis

Support Git Submodules #311

what

why

use-case