Open mss opened 3 months ago
@mss thanks for testing it.
Atmos uses the go-getter
lib https://github.com/hashicorp/go-getter to download from different sources.
Can you please test the URL ssh://git@git.example.com/tf/tf-modules.git//modules/account-vpc?ref=v0.1.0
instead of git::ssh://git@git.example.com/tf/tf-modules.git//modules/account-vpc?ref=v0.1.0
Another problem could be that it uses both ssh
and //modules
, and only in this case it does not respect .gitconfig
settings.
Can you please test a few things:
ssh
URL that does not use //modules
and let us know the results.
Thank you
Sorry, I see you point to this now in your "Context" section.
This relates to:
There is a notable exception when it comes to SSH settings in .gitconfig
. Configurations related to SSH, such as URL replacements (insteadOf
), do not work as expected with go-getter
. This is because go-getter
checks out the repository into a temporary directory first, which can bypass certain conditional Git configurations.
@aknysh sounds like the fix is pointed to here:
It does work in Terraform because they resolve the double-slash syntax themselves as pointed out in this comment (I linked to the OpenTofu source due to the current Terraform license but the code is the same).
We should be able to implement the same fix in atmos.
@mss I see your latest opened issue here https://github.com/hashicorp/go-getter/issues/493
If you have tested what I asked above ("Any other ssh URL that does not use //modules"), does it means that go-getter
respects .gitconfig
for the root of the repo, but does not respect it when using modules (b/c the $GIT_DIR
does not match the directory from the includeIf
)?
Thanks again for all the testing. Your help is appreciated, it will allow us to understand the root of the problem and fix it in Atmos
Wow, that's some quick responses :-)
Ok, maybe some additional info: We did not check if URLs without //
work but instead verified that .gitconfig
is not generally ignored by setting $TMPDIR
to a directory within the work
dir. Ie. this worked as expected: mkdir -p $HOME/code/work/tmp && env TMPDIR=$HOME/code/work/tmp atmos vendor pull
.
I now tried to use an URL without the double-slash and that does not work at all, even without the magic in .gitconfig
.
Here is my use case: The first component is vendored properly but the second isn't (I put both into a single vendor file; the behaviour is the same when I use two files)
apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
name: account-vpc
description: account components
spec:
sources:
- component: 'aws-vpc-endpoints-v5.8.1'
source: 'git::https://github.com/terraform-aws-modules/terraform-aws-vpc.git//modules/vpc-endpoints?ref=v5.8.1'
targets:
- 'components/terraform/aws-vpc-endpoints/v5.8.1'
- component: 'aws-vpc-v5.8.1'
source: 'git::https://github.com/terraform-aws-modules/terraform-aws-vpc.git?ref=v5.8.1'
targets:
- 'components/terraform/aws-vpc/v5.8.1'
The weird thing is that if I strace the process then no git clone
is called at all for the second component (same when using two vendor files). But I am tired so there is probably a typo in my tests.
One thing I also noticed in the strace output: Something first creates a /tmp/17174332073449747459
directory and tries to use that. For both cases. Is that atmos doing its thing there? (You can see the go-getter /tmp/getter3662728735/temp
right after that in the first usecase.)
@mss thanks.
The second issue is different from the first one :)
Anyway, I see both issue now. The second one is b/c of a combination of how go-getter
works and the fact that Atmos always created tmp directories b/c it later processes the Included_paths
and excluded_paths
attributes.
The fixes will be in a new Atmos release.
Thanks again for all the testing
@mss please try this latest release https://github.com/cloudposse/atmos/releases/tag/v1.78.0
It fixes the second issue you raised ("I now tried to use an URL without the double-slash").
Regarding the first issue (go-getter
not respecting .gitconfig
), let me know if it's still an issue for you (if mkdir -p $HOME/code/work/tmp && env TMPDIR=$HOME/code/work/tmp atmos vendor pull
is not what you want to use), and we'll look into what could be done here.
Thanks again
Hey there. I have these exact issues. For the second issue (no //..
). With 1.78.0 I now have the same behaviour for both double-slash and no-double-slash URLs. That's good.
I still have the behaviour that I have to set TMPDIR ([includeIf "gitdir:~/...]
).
thanks @verygitmuchhub
We'll have to look if we can set TMPDIR
in Atmos to the temp directory
Describe the Bug
I must admit that this is a slightly esoteric use case and thus maybe some documentation at https://atmos.tools/cli/commands/vendor/pull#description would be sufficient.
Let's assume that we have some module source with the URL
git::ssh://git@git.example.com/tf/tf-modules.git//modules/account-vpc?ref=v0.1.0
(note the double-slash syntax to use the given subdirectory).For some reason are certain users or processes not able to access the repository via SSH but need to use HTTPS instead. Since it only affects this one server they add some config to
~/.gitconfig
like this:And that
work.config
file contains something like this:This configuration works and the modules are pulled via HTTPS instead of SSH if one creates a plain old Terraform root module and one calls
terraform init
.Now we want to use Atmos vendoring and add
vendor.yaml
:This won't use the given mirror but will (try to) use the original URL which may fail due to whatever networking issues are the reason the config was added in the first place.
Expected Behavior
I should not have to strace the atmos command to find out why my Git config which worked with Terraform does not work anymore with Atmos vendoring :-) It would be nice if it just worked as expected (like for Terraform) or the behaviour (ie. that a subdirectory-based module will be cloned to a subdirectory below
$TMPDIR
) was documented.Steps to Reproduce
See above (there is probably a more minimalistic reproducer possible). Some other Git features (badly written hooks?) than the one described might be affected, too.
Screenshots
No response
Environment
Additional Context
This is caused by an undocumented behaviour of go-getter (cf. hashicorp/go-getter#493) to pull an URL which refers to a subdirectory to
$TMPDIR
first and then copy over the wanted contents. So the$GIT_DIR
does not match the directory from theincludeIf
because the code is actually checked out to a temporary location like/tmp/getter12345/temp
.It does work in Terraform because they resolve the double-slash syntax themselves as pointed out in this comment (I linked to the OpenTofu source due to the current Terraform license but the code is the same).