Open paslandau opened 2 years ago
joshrabinowitz
commented
2 years ago I haven't dug in yet but when you mention this:
It does not appear when git secret is used in a non shared folder.
it sounds like it might be a file/directory permissions/ownership issue; perhaps which is keeping gpg-agent from starting.
Can you try running the whole example with the env SECRETS_VERBOSE=1 ? This might offer some hints.
paslandau
commented
2 years ago Just added more info on the host system (sry for omitting before):
My Host system is Windows 10 and I use Docker Desktop (non-WSL).
Mentioning this, because in this setup the permissions are not transferred from the host system (instead, they are 0777 by default which can also not be changed, see https://stackoverflow.com/a/66390200 - thus I don't believe it's a permission issue)
Can you try running the whole example with the env SECRETS_VERBOSE=1 ? This might offer some hints.
Unfortunately no additional info:
root:/var/www/app# export SECRETS_VERBOSE=1
root:/var/www/app# git secret killperson john.doe@example.com
gpg: can't connect to the agent: IPC connect call failedBut: It revealed that the same error is happening on git secret tell:
root:/var/www/app# git secret tell john.doe@example.com
gpg: key 49393E27EDF49163: public key "John Doe <john.doe@example.com>" imported
gpg: can't connect to the agent: IPC connect call failed
gpg: Total number processed: 1
gpg: imported: 1
git-secret: cleaning up: /tmp/_git_secret.CcjBCL(that was "hidden" before).
FYI: Everything "still works", i.e. I am still able to decrypt the files with john.doe@example.com when adding via tell resp. am not able to decrypt after I ran killperson.
https://github.com/sobolevn/git-secret/issues/594 mentions
bash -x `which git-secret` add file.txtso I'm also attaching the output of that command (sry for the wall of text - there more info at the bottom of this post)
root:/shared# bash -x `which git-secret` killperson john.doe@example.com
+ GITSECRET_VERSION=0.4.0
+ _SECRETS_DIR=.gitsecret
+ _SECRETS_DIR_KEYS=.gitsecret/keys
+ _SECRETS_DIR_PATHS=.gitsecret/paths
+ _SECRETS_DIR_KEYS_TRUSTDB=.gitsecret/keys/trustdb.gpg
+ _SECRETS_DIR_PATHS_MAPPING=.gitsecret/paths/mapping.cfg
+ [[ -n 1 ]]
+ [[ 1 -ne 0 ]]
+ _SECRETS_VERBOSE=1
+ : .secret
+ : gpg
+ : '_os_based __sha256'
+ : '_os_based __get_octal_perms'
+ : '_os_based __epoch_to_date'
+ : /tmp
+ AWK_FSDB_HAS_RECORD='
BEGIN { FS=":"; OFS=":"; cnt=0; }
{
if ( key == $1 )
{
cnt++
}
}
END { if ( cnt > 0 ) print "0"; else print "1"; }
'
+ AWK_FSDB_RM_RECORD='
BEGIN { FS=":"; OFS=":"; }
{
if ( key != $1 )
{
print $1,$2;
}
}
'
+ AWK_FSDB_CLEAR_HASHES='
BEGIN { FS=":"; OFS=":"; }
{
print $1,"";
}
'
+ AWK_GPG_VER_CHECK='
/^gpg/{
version=$3
n=split(version,array,".")
if( n >= 2) {
if(array[1] >= 2)
{
if(array[2] >= 1)
{
print 1
}
else
{
print 0
}
}
else
{
print 0
}
}
else if(array[1] >= 2)
{
print 1
}
else
{
print 0
}
}
'
++ gpg --version
++ gawk '
/^gpg/{
version=$3
n=split(version,array,".")
if( n >= 2) {
if(array[1] >= 2)
{
if(array[2] >= 1)
{
print 1
}
else
{
print 0
}
}
else
{
print 0
}
}
else if(array[1] >= 2)
{
print 1
}
else
{
print 0
}
}
'
+ GPG_VER_MIN_21=1
+ AWK_GPG_KEY_CNT='
BEGIN { cnt=0; OFS=":"; FS=":"; }
flag=0; $1 == "pub" { cnt++ }
END { print cnt }
'
+ AWK_ADD_TO_GITIGNORE='
BEGIN {
cnt=0
}
function check_print_line(line){
if (line == pattern) {
cnt++
}
print line
}
# main function
{
check_print_line($0) # check and print first line
while (getline == 1) { # check and print all other
check_print_line($0)
}
}
END {
if ( cnt == 0) { # if file did not contain pattern add
print pattern
}
}
'
+ AWK_FSDB_UPDATE_HASH='
BEGIN { FS=":"; OFS=":"; }
{
if ( key == $1 )
{
print key,hash;
}
else
{
print $1,$2;
}
}
'
+ set -e
+ _init_script killperson john.doe@example.com
+ [[ 2 == 0 ]]
+ local dry_run=0
+ [[ 2 -gt 0 ]]
+ local opt=killperson
+ case "$opt" in
+ break
+ [[ 0 == 0 ]]
+ _check_setup
+ local is_tree
++ _is_inside_git_tree
++ local result
+++ git rev-parse --is-inside-work-tree
+++ echo 0
++ result=0
++ echo 0
+ is_tree=0
+ [[ 0 -ne 0 ]]
+ _secrets_dir_is_not_ignored
+ local git_secret_dir
++ _get_secrets_dir
++ _append_root_path .gitsecret
++ local path=.gitsecret
++ local root_path
+++ _get_git_root_path
+++ local result
+++++ git rev-parse --show-toplevel
++++ _clean_windows_path /shared
++++ echo /shared
++++ sed 's#^\([a-zA-Z]\):/#/\1/#'
+++ result=/shared
+++ echo /shared
++ root_path=/shared
++ echo /shared/.gitsecret
+ git_secret_dir=/shared/.gitsecret
+ local ignores
++ _check_ignore /shared/.gitsecret
++ local filename=/shared/.gitsecret
++ local result
+++ git check-ignore -q /shared/.gitsecret
+++ echo 1
++ result=1
++ echo 1
+ ignores=1
+ [[ ! 1 -eq 1 ]]
+ local keys_dir
++ _get_secrets_dir_keys
++ _append_root_path .gitsecret/keys
++ local path=.gitsecret/keys
++ local root_path
+++ _get_git_root_path
+++ local result
+++++ git rev-parse --show-toplevel
++++ _clean_windows_path /shared
++++ sed 's#^\([a-zA-Z]\):/#/\1/#'
++++ echo /shared
+++ result=/shared
+++ echo /shared
++ root_path=/shared
++ echo /shared/.gitsecret/keys
+ keys_dir=/shared/.gitsecret/keys
+ local secring=/shared/.gitsecret/keys/secring.gpg
+ [[ -f /shared/.gitsecret/keys/secring.gpg ]]
+ local function_exists
++ _function_exists killperson
++ local function_name=killperson
++ declare -f -F killperson
++ echo 0
+ function_exists=0
+ [[ 0 == 0 ]]
+ [[ ! killperson == _* ]]
+ killperson john.doe@example.com
+ OPTIND=1
+ getopts h opt
+ shift 0
+ '[' john.doe@example.com = -- ']'
+ _user_required
+ _secrets_dir_exists
+ local full_path
++ _get_secrets_dir
++ _append_root_path .gitsecret
++ local path=.gitsecret
++ local root_path
+++ _get_git_root_path
+++ local result
+++++ git rev-parse --show-toplevel
++++ _clean_windows_path /shared
++++ ++++ echo sed /shared
's#^\([a-zA-Z]\):/#/\1/#'
+++ result=/shared
+++ echo /shared
++ root_path=/shared
++ echo /shared/.gitsecret
+ full_path=/shared/.gitsecret
+ [[ ! -d /shared/.gitsecret ]]
+ local trustdb
++ _get_secrets_dir_keys_trustdb
++ _append_root_path .gitsecret/keys/trustdb.gpg
++ local path=.gitsecret/keys/trustdb.gpg
++ local root_path
+++ _get_git_root_path
+++ local result
+++++ git rev-parse --show-toplevel
++++ _clean_windows_path /shared
++++ sed 's#^\([a-zA-Z]\):/#/\1/#'
++++ echo /shared
+++ result=/shared
+++ echo /shared
++ root_path=/shared
++ echo /shared/.gitsecret/keys/trustdb.gpg
+ trustdb=/shared/.gitsecret/keys/trustdb.gpg
+ local 'error_message=no public keys for users found. run '\''git secret tell email@address'\''.'
+ [[ ! -f /shared/.gitsecret/keys/trustdb.gpg ]]
+ local secrets_dir_keys
++ _get_secrets_dir_keys
++ _append_root_path .gitsecret/keys
++ local path=.gitsecret/keys
++ local root_path
+++ _get_git_root_path
+++ local result
+++++ git rev-parse --show-toplevel
++++ _clean_windows_path /shared
++++ echo /shared
++++ sed 's#^\([a-zA-Z]\):/#/\1/#'
+++ result=/shared
+++ echo /shared
++ root_path=/shared
++ echo /shared/.gitsecret/keys
+ secrets_dir_keys=/shared/.gitsecret/keys
+ local keys_exist
++ gpg --homedir /shared/.gitsecret/keys --no-permission-warning -n --list-keys
+ keys_exist='/shared/.gitsecret/keys/pubring.kbx
----------------------------------------
pub rsa2048 2022-03-31 [SCEA]
99A33E29616B496257C7E10B49393E27EDF49163
uid [ unknown] John Doe <john.doe@example.com>
sub rsa2048 2022-03-31 [SEA]'
+ local exit_code=0
+ [[ -z /shared/.gitsecret/keys/pubring.kbx
----------------------------------------
pub rsa2048 2022-03-31 [SCEA]
99A33E29616B496257C7E10B49393E27EDF49163
uid [ unknown] John Doe <john.doe@example.com>
sub rsa2048 2022-03-31 [SEA] ]]
+ [[ 0 -ne 0 ]]
+ emails=('john.doe@example.com')
+ local emails
+ [[ 1 -eq 0 ]]
+ local secrets_dir_keys
++ _get_secrets_dir_keys
++ _append_root_path .gitsecret/keys
++ local path=.gitsecret/keys
++ local root_path
+++ _get_git_root_path
+++ local result
+++++ git rev-parse --show-toplevel
++++ _clean_windows_path /shared
++++ echo /shared
++++ sed 's#^\([a-zA-Z]\):/#/\1/#'
+++ result=/shared
+++ echo /shared
++ root_path=/shared
++ echo /shared/.gitsecret/keys
+ secrets_dir_keys=/shared/.gitsecret/keys
+ _assert_keyring_contains_emails /shared/.gitsecret/keys 'git-secret keyring' john.doe@example.com
+ local homedir=/shared/.gitsecret/keys
+ local 'keyring_name=git-secret keyring'
+ local emails=john.doe@example.com
+ _assert_keyring_emails /shared/.gitsecret/keys 'git-secret keyring' john.doe@example.com 1
+ local homedir=/shared/.gitsecret/keys
+ local 'keyring_name=git-secret keyring'
+ local emails=john.doe@example.com
+ local expected=1
+ local gpg_uids
++ _get_users_in_gpg_keyring /shared/.gitsecret/keys
++ local homedir=/shared/.gitsecret/keys
++ local result
++ args=()
++ local args
++ [[ -n /shared/.gitsecret/keys ]]
++ args+=("--homedir" "$homedir")
+++ gpg --homedir /shared/.gitsecret/keys --no-permission-warning --list-public-keys --with-colon --fixed-list-mode
+++ gawk -F: '$1=="uid"'
++ result='uid:-::::1648707594::454A9C7753721D33C846A7FFEDB690FD548BCE5C::John Doe <john.doe@example.com>::::::::::0:'
++ local invalid_lines
+++ echo 'uid:-::::1648707594::454A9C7753721D33C846A7FFEDB690FD548BCE5C::John Doe <john.doe@example.com>::::::::::0:'
+++ gawk -F: '$2=="i" || $2=="d" || $2=="r" || $2=="e" || $2=="n"'
++ invalid_lines=
++ local emails
+++ _extract_emails_from_gpg_output 'uid:-::::1648707594::454A9C7753721D33C846A7FFEDB690FD548BCE5C::John Doe <john.doe@example.com>::::::::::0:'
+++ local 'result=uid:-::::1648707594::454A9C7753721D33C846A7FFEDB690FD548BCE5C::John Doe <john.doe@example.com>::::::::::0:'
+++ local emails
++++ echo 'uid:-::::1648707594::454A9C7753721D33C846A7FFEDB690FD548BCE5C::John Doe <john.doe@example.com>::::::::::0:'
++++ gawk -F: '{print gensub(/.*<(.*)>.*/, "\\1", "g", $10); }'
++++ sed 's/([^)]*)//g'
+++ emails=john.doe@example.com
+++ echo john.doe@example.com
++ emails=john.doe@example.com
++ local emails_with_invalid_keys
+++ _extract_emails_from_gpg_output ''
+++ local result=
+++ local emails
++++ echo ''
++++ gawk -F: '{print gensub(/.*<(.*)>.*/, "\\1", "g", $10); }'++++
sed 's/([^)]*)//g'
+++ emails=
+++ echo ''
++ emails_with_invalid_keys=
++ [[ -n '' ]]
++ echo john.doe@example.com
+ gpg_uids=john.doe@example.com
+ for email in "${emails[@]}"
+ [[ john.doe@example.com != *\@* ]]
+ local emails_found=0
+ for uid in $gpg_uids
+ [[ john.doe@example.com == \j\o\h\n\.\d\o\e\@\e\x\a\m\p\l\e\.\c\o\m ]]
+ emails_found=1
+ [[ 1 -eq 1 ]]
+ [[ 1 -eq 0 ]]
+ [[ 1 -gt 1 ]]
+ for email in "${emails[@]}"
+ gpg --homedir /shared/.gitsecret/keys --no-permission-warning --batch --yes --delete-key john.doe@example.com
gpg: can't connect to the agent: IPC connect call failedExecuting the last command with -v for additional debug info yields
root:/var/www/app# gpg -v --homedir /shared/.gitsecret/keys --no-permission-warning --batch --yes --delete-key john.doe@example.com
gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg: waiting for the agent to come up ... (5s)
gpg: waiting for the agent to come up ... (4s)
gpg: waiting for the agent to come up ... (3s)
gpg: waiting for the agent to come up ... (2s)
gpg: waiting for the agent to come up ... (1s)
gpg: can't connect to the agent: IPC connect call failed
gpg: using pgp trust modelRunning the same thing in /tmp:
root:/tmp# gpg -v --homedir /tmp/.gitsecret/keys --no-permission-warning --batch --yes --delete-key john.doe@example.com
gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg: waiting for the agent to come up ... (5s)
gpg: connection to agent established
gpg: using pgp trust modelShort follow up:
I checked the running gpg-agents via ps aux and noticed, that an agent is started for /tmp but not for /shared (after running e.g. git secret tell)
root:/var/www/app# ps aux
PID USER TIME COMMAND
1 root 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
2608 root 0:00 gpg-agent --homedir /tmp/.gitsecret/keys --use-standard-socket --daemon
2679 root 0:00 ps auxKilling and restarting it works:
root:/tmp# kill 2608
root:/tmp# gpg-agent --homedir /tmp/.gitsecret/keys --use-standard-socket --daemon
gpg-agent[2704]: WARNING: "--use-standard-socket" is an obsolete option - it has no effect
gpg-agent[2705]: gpg-agent (GnuPG) 2.2.31 startedTrying the same in the /shared folder fails with
root:/shared# gpg-agent --homedir /shared/.gitsecret/keys --use-standard-socket --daemon
gpg-agent[2661]: WARNING: "--use-standard-socket" is an obsolete option - it has no effect
gpg-agent[2661]: error binding socket to '/shared/.gitsecret/keys/S.gpg-agent': I/O errorI believe I understand the issue now - though I'm not sure if/how this can be solved by git secret, because it requires adding some additional files to the --home-dir of the gpg-agent in order to change the socket location.
Here's my understanding of the issue:
gpg needs an gpg-agent for some (?) thingsgpg command somewhere (?) within git-secret--home-dir at ./S.gpg-agentThere is unfortunately no easy way to configure the location of the socket (in fact, it's 4 sockets), but one has to create three files in the --home-dir as outlined below.
Note, that i created the empty directory /foo before that is not shared with the host system to have a safe place for the sockets.
S.gpg-agent
%Assuan%
socket=/foo/S.gpg-agentS.gpg-agent.ssh
%Assuan%
socket=/foo/S.gpg-agent.sshgpg-agent.conf
extra-socket /foo/S.gpg-agent.extra
browser-socket /foo/S.gpg-agent.browserSee also https://askubuntu.com/a/1053594/1583296
Afterwards, all commands work as expected without errors:
root:/shared# git secret tell john.doe@example.com
gpg: key 49393E27EDF49163: public key "John Doe <john.doe@example.com>" imported
gpg: Total number processed: 1
gpg: imported: 1
git-secret: done. john.doe@example.com added as user(s) who know the secret.
root:/shared# git secret killperson john.doe@example.com
git-secret: removed keys.
git-secret: now [john.doe@example.com] do not have an access to the repository.
git-secret: make sure to hide the existing secrets again.The big problem that I see here, is that the location of the sockets would have to be "hard-coded" in the repository (i.e. in the git-secret/keys/ folder) - which might break the setup on different systems, e.g. if git-secret set up locally by different developers.
This is okay for "my" use case, because the whole point of setting git-secret up in a docker container is avoiding that every dev in the team has to do it "on their machine" - so I would still be great if there would be some way to support this use case.
Maybe something like git-secret init --gpg-socket-location="/path/to/directory" that would create the corresponding files as mentioned above?
Soo.. any thoughts?
joshrabinowitz
commented
2 years ago I'm not exactly clear what your use case is; I can't quite figure out what problem you're trying to solve. What are you trying to accomplish in the alpine VM?
paslandau
commented
2 years ago I try to reduce the amount of tools that each individual dev in a team needs to set up locally as much as possible. We use docker to provide the local development infrastructure and I try to set up git-secret and gpg in a container that shares the codebase with the host system. With this setup, everybody can use git-secret "through docker".
I'll release a longer article on the exact setup with a proof of concept next weeks. The corresponding code is already live at https://github.com/paslandau/docker-php-tutorial/tree/part-6-git-secret-encrypt-repository-docker (though the build might currently fail due to https://github.com/codecasts/php-alpine/issues/160 )
From the article:
To see it in action, check out branch part-6-git-secret-encrypt-repository-docker and run the following commands:
# checkout the branch
git checkout part-6-git-secret-encrypt-repository-docker
# build and start the docker setup
make make-init
make docker-build
make docker-up
# "create" the secret key - the file "secret.gpg.example" would usually NOT live in the repo!
cp secret.gpg.example secret.gpg
# intialize gpg
make gpg-init
# decrypt the secret file
make secret-decrypt
# show the content of the secret file
cat passwords.txtIn the following gif, all commands are run "through docker" (abstracted via make targets), git-secret is not set up on the host machine.
joshrabinowitz
commented
2 years ago Please explain your use case much more simply. This feels like an XY problem: https://xyproblem.info/
Your example is so removed from the normal case of 'add a user's public key to a git-secret repo's keychain and let them decrypt/encrypt secrets until their key changes or they should no longer have access' that I can't understand how you intend to use git-secret in real life. For example, in your examples above, there is one keypair that can encrypt/decrypt secrets, so it doesn't make sense to me to use 'removeperson' at the end of your example (although it is a great example of how to generate an error, which is valid, and I think we have largely addressed as a permissions issue)
In your target case (not a idealized example), who controls private keys, how many keypairs are you planning on using (generally), where are they stored (generally), who is allowed to decrypt secrets. (Previous paragraph edited for clarity).
Edit: lastly, I love the animated gifs showing your tests, but since I can't easily pause or slow down the animations it's difficult to follow along (also would be best if your examples just used basic commands and git-secret ones instead of a level of abstraction on top of them, but, no big deal).
paslandau
commented
2 years ago Hey @joshrabinowitz,
sorry for the late reply and the confusion. Here's where I was coming from (probably helpful for context): I was writing a blog post about using git-secret in a dev team that uses docker to run the application locally. It's now finally published at https://www.pascallandau.com/blog/git-secret-encrypt-repository-docker/
In this setup I believe it's quite common to share the codebase on the host system with the docker container that runs the application, so that any changes are reflected immediately. Running the application on docker instead of "locally on the host system" has the big benefit that everybody uses the same infrastructure - no more problems with different PHP versions, missing extensions etc.
Following this logic, I went one step further and also added the tooling for the development process itself directly in the docker container. Tooling being e.g. git-secret, terraform, gcloud-cli, etc. Again: The idea is that a developer does not have to set up tools locally any longer because "everything" can run through docker.
I've also described that in more detail at https://www.pascallandau.com/blog/git-secret-encrypt-repository-docker/#local-git-secret-and-gpg-setup
=> Does this make more sense?
In your target case (not a idealized example), who controls private keys, how many keypairs are you planning on using (generally), where are they stored (generally), who is allowed to decrypt secrets. (Previous paragraph edited for clarity).
docker and gpg runs inside of docker, using that shared keygpg running inside dockerI have also added the common workflow scenarios (onboarding a new developer, adding/deleting files, offboarding a dev) under https://www.pascallandau.com/blog/git-secret-encrypt-repository-docker/#scenarios
Edit: lastly, I love the animated gifs showing your tests, but since I can't easily pause or slow down the animations it's difficult to follow along (also would be best if your examples just used basic commands and git-secret ones instead of a level of abstraction on top of them, but, no big deal).
Yeah sorry for that... The second gif was actually part of the tutorial and not meant to be used "standalone". Just had it available and thus added it here. Is it still useful? I hope https://www.pascallandau.com/blog/git-secret-encrypt-repository-docker/ should clarify any questions - though I do realize it's quite a lot to ask reading a whole article... so let me know if I can clarify anything else directly.
Cheers Pascal
joshrabinowitz
commented
2 years ago Hello, @paslandau
Thanks for following up.
You present an unusual use case, but it's an important one perhaps, and you do present a reasonable solution to the problem.
If you submit a PR, we will consider it for inclusion. Alternately it may make sense to release the needed code as a git-secret "external plugin": https://git-secret.io/plugins
I try to build an alpine docker image that contains
git-secretand can be used to encry/decrypt files in a codebase that is shared with the host system. My Host system is Windows 10 and I use Docker Desktop (non-WSL).When running
git secret killpersonin the container within the shared folder, the erroris shown. It does not appear when
git secretis used in a non-shared folder.What were you expecting to happen?
No error is shown
What are the steps to reproduce this issue?
alpine:3.15.2as base imagegit, gnupg and git-secretjohn.doe@example.comcreate an empty folder on the host system
Start a container with the image, share the folder and attach a shell session
(FYI: I'm using a MinGW shell on Windows and need to run the following command:
navigate to the
sharedfolderinitialize git, git secret, add
john.doe@example.comjohn.doe@example.comAny logs, error output, etc?
-
Any other comments?
When running the same commands on a non-shared folder (e.g.
/tmp), everything works as expected. Interstingly,ps auxwill show agpg-agentprocess after running the command.=> this process is no running when used in a shared folder
What versions of software are you using?
Operating system: (
uname -a) …=> Alpine 3.15.2 via docker
git-secretpath: (which git-secret) …git-secretversion: (git secret --version) …gitversion: (git --version) …Shell type and version: (
$SHELL --version) … not workinggpgversion: (gpg --version) …