search

buildkite / elastic-ci-stack-s3-secrets-hooks

🕵️‍♀️ Expose secrets to your buildkite build steps via Amazon S3
MIT License
24 stars 17 forks source link

Unable to download git-credentials on windows agents after updating to 5.1.0 #46

Closed kszucs closed 3 years ago

kszucs commented 3 years ago

Perhaps the path should be quoted:

Cloning into '.'...
--
  | C:/Program Files/Git/usr/local/buildkite-aws-stack/plugins/secrets/git-credential-s3-secrets ursa-computing-buildkite-secrets git-credentials get: C:/Program: No such file or directory
  | fatal: could not read Username for 'https://github.com': terminal prompts disabled
pda commented 3 years ago

Thanks @kszucs — I'll look into this, and hopefully add a regression test.

keithduncan commented 3 years ago

I’m looking into the escaping on this today. Using the command line git config to set values with spaces I have managed to figure out how Linux paths should be escaped in the file format and environment variable:

File:

[ec2-user@ip-10-0-2-19 test]$ cat .git/config 
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = https://github.com/keithduncan/hello-world-private
    fetch = +refs/heads/*:refs/remotes/origin/*
[credential]
    helper = /usr/local/buildkite-aws-stack/plugins/secrets/test\\ test/git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials
[ec2-user@ip-10-0-2-19 test]$ GIT_TRACE=1 AWS_DEFAULT_REGION=us-east-1 git fetch
22:49:18.494276 git.c:455               trace: built-in: git fetch
22:49:18.495583 run-command.c:667       trace: run_command: GIT_DIR=.git git remote-https origin https://github.com/keithduncan/hello-world-private
22:49:18.496685 git.c:743               trace: exec: git-remote-https origin https://github.com/keithduncan/hello-world-private
22:49:18.496720 run-command.c:667       trace: run_command: git-remote-https origin https://github.com/keithduncan/hello-world-private
22:49:18.530891 run-command.c:667       trace: run_command: '/usr/local/buildkite-aws-stack/plugins/secrets/test\ test/git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials get'
22:49:20.304770 run-command.c:667       trace: run_command: '/usr/local/buildkite-aws-stack/plugins/secrets/test\ test/git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials store'
22:49:20.371247 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
22:49:20.373087 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
22:49:20.374160 git.c:455               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
22:49:20.377870 run-command.c:1628      run_processes_parallel: preparing to run up to 1 tasks
22:49:20.377892 run-command.c:1660      run_processes_parallel: done
22:49:20.377909 run-command.c:667       trace: run_command: git maintenance run --auto --no-quiet
22:49:20.379109 git.c:455               trace: built-in: git maintenance run --auto --no-quiet

Environment variable:

GIT_TRACE=1 AWS_DEFAULT_REGION=us-east-1 GIT_CONFIG_PARAMETERS="'credential.helper=/usr/local/buildkite-aws-stack/plugins/secrets/test\\ test/git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials'" git fetch
22:38:07.989143 git.c:455               trace: built-in: git fetch
22:38:07.990199 run-command.c:667       trace: run_command: GIT_DIR=.git git remote-https origin https://github.com/keithduncan/hello-world-private
22:38:07.991481 git.c:743               trace: exec: git-remote-https origin https://github.com/keithduncan/hello-world-private
22:38:07.992596 run-command.c:667       trace: run_command: git-remote-https origin https://github.com/keithduncan/hello-world-private
22:38:08.081159 run-command.c:667       trace: run_command: '/usr/local/buildkite-aws-stack/plugins/secrets/test\ test/git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials get'
22:38:09.869674 run-command.c:667       trace: run_command: '/usr/local/buildkite-aws-stack/plugins/secrets/test\ test/git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials store'
22:38:09.941712 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
22:38:09.944470 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
22:38:09.945597 git.c:455               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
22:38:09.948561 run-command.c:1628      run_processes_parallel: preparing to run up to 1 tasks
22:38:09.948582 run-command.c:1660      run_processes_parallel: done
22:38:09.948599 run-command.c:667       trace: run_command: git maintenance run --auto --no-quiet
22:38:09.949980 git.c:455               trace: built-in: git maintenance run --auto --no-quiet

Now to look at Windows.

keithduncan commented 3 years ago

Windows working file config:

$ cat .git/config
[core]
        repositoryformatversion = 0
        filemode = false
        bare = false
        logallrefupdates = true
        symlinks = false
        ignorecase = true
[credential]
        helper = C:\\\\Program\\ Files\\\\Git\\\\usr\\\\local\\\\buildkite-aws-stack\\\\plugins\\\\secrets\\\\git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials
[remote "origin"]
        url = https://github.com/keithduncan/hello-world-private
        fetch = +refs/heads/*:refs/remotes/origin/*

Administrator@EC2AMAZ-P2H7VP5 MINGW64 ~/test (master)
$ AWS_DEFAULT_REGION=us-east-1 GIT_TRACE=1 git fetch
23:39:41.485394 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
23:39:41.485394 git.c:447               trace: built-in: git fetch
23:39:41.485394 run-command.c:667       trace: run_command: GIT_DIR=.git git remote-https origin https://github.com/keithduncan/hello-world-private
23:39:41.515631 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
23:39:41.515631 git.c:734               trace: exec: git-remote-https origin https://github.com/keithduncan/hello-world-private
23:39:41.515631 run-command.c:667       trace: run_command: git-remote-https origin https://github.com/keithduncan/hello-world-private
23:39:41.547705 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
23:39:41.624585 run-command.c:667       trace: run_command: 'C:\\Program\ Files\\Git\\usr\\local\\buildkite-aws-stack\\plugins\\secrets\\git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials get'
23:39:44.367213 run-command.c:667       trace: run_command: 'C:\\Program\ Files\\Git\\usr\\local\\buildkite-aws-stack\\plugins\\secrets\\git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials store'
23:39:44.484822 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
23:39:44.503259 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
23:39:44.519011 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
23:39:44.519011 git.c:447               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
23:39:44.606581 run-command.c:1628      run_processes_parallel: preparing to run up to 1 tasks
23:39:44.606581 run-command.c:1660      run_processes_parallel: done
23:39:44.606581 run-command.c:667       trace: run_command: git maintenance run --auto --no-quiet
23:39:44.606581 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
23:39:44.622395 git.c:447               trace: built-in: git maintenance run --auto --no-quiet

Environment variable:

Administrator@EC2AMAZ-P2H7VP5 MINGW64 ~/test (master)
$ export GIT_CONFIG_PARAMETERS="'credential.helper=C:\\\\Program\ Files\\\\Git\\\\usr\\\\local\\\\buildkite-aws-stack\\\\plugins\\\\secrets\\\\git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials'"
00:20:20.044287 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
00:20:20.044287 git.c:447               trace: built-in: git config --bool core.sparseCheckout

Administrator@EC2AMAZ-P2H7VP5 MINGW64 ~/test (master)
$ git fetch
00:20:22.058356 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
00:20:22.082150 git.c:447               trace: built-in: git fetch
00:20:22.088134 run-command.c:667       trace: run_command: GIT_DIR=.git git remote-https origin https://github.com/keithduncan/hello-world-private
00:20:22.152109 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
00:20:22.167730 git.c:734               trace: exec: git-remote-https origin https://github.com/keithduncan/hello-world-private
00:20:22.167730 run-command.c:667       trace: run_command: git-remote-https origin https://github.com/keithduncan/hello-world-private
00:20:22.273486 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
00:20:22.334672 run-command.c:667       trace: run_command: 'C:\\Program\ Files\\Git\\usr\\local\\buildkite-aws-stack\\plugins\\secrets\\git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials get'
00:20:25.054371 run-command.c:667       trace: run_command: 'C:\\Program\ Files\\Git\\usr\\local\\buildkite-aws-stack\\plugins\\secrets\\git-credential-s3-secrets buildkite-crossregiontest pipeline-with-spaces/git-credentials store'
00:20:25.274908 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
00:20:25.300099 run-command.c:667       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs
00:20:25.315937 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
00:20:25.315937 git.c:447               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs
00:20:25.361009 run-command.c:1628      run_processes_parallel: preparing to run up to 1 tasks
00:20:25.361009 run-command.c:1660      run_processes_parallel: done
00:20:25.361009 run-command.c:667       trace: run_command: git maintenance run --auto --no-quiet
00:20:25.376753 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
00:20:25.376753 git.c:447               trace: built-in: git maintenance run --auto --no-quiet
00:20:25.662160 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
00:20:25.662160 git.c:447               trace: built-in: git config --bool core.sparseCheckout
keithduncan commented 3 years ago

The key appears to be that it has to show up in trace: run_command as escaped once. The file requires two levels of escaping, the environment variable also requires two levels of escaping though bash/the-shell will interpolate one so the environment variable will hold a singly-escaped value.