search

StackExchange / blackbox

Safely store secrets in Git/Mercurial/Subversion
MIT License
6.69k stars 370 forks source link

gpg: decryption failed: No secret key #234

Open aymericbeaumet opened 6 years ago

aymericbeaumet commented 6 years ago
Blackbox 89566f7
Gpg 2.2.3

We are facing a Blackbox issue on both several local machines (up to date macOS) and our CI (amazonlinux:2016.09.1.20161221 running on CircleCI). We extensively rely on Blackbox with nearly a 100 encrypted files.

The decryption goes well most of the time, but around 5% of the invocations result in a non-deterministic issue causing gpg to crash. Here goes the relevant part of the error logs:

$ blackbox_decrypt_all_files
gpg: key ABCD: public key "abcd <abcd@key>" imported
gpg: key ABCD: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
========== Importing keychain: START
gpg: key 98A87F81DCF732EA: public key "a <a@a>" imported
gpg: key DFB3BB7E72A6AE86: public key "b <b@b>" imported
gpg: key E718D80E5FACEB2B: public key "c <c@c>" imported
gpg: key EFAF45AA074BFF85: public key "d <d@d>" imported
gpg: key E6D589AD1CBB467F: public key "e <e@e>" imported
gpg: key CB36B534CB358125: public key "f <f@f>" imported
gpg: key F591723101148D38: public key "g <g@g>" imported
gpg: Total number processed: 8
gpg:               imported: 7
gpg:              unchanged: 1
========== Importing keychain: DONE
========== Decrypting new/changed files: START
========== EXTRACTED some/secret/file.js
# ...
# Successfully extracting ~90 files
# ...
========== EXTRACTED some/other/secret/file.js
gpg: decryption failed: No secret key
Exited with code 2

Hard to track down, any leads?

TomOnTime commented 6 years ago

If you get "gpg: decryption failed: No secret key" it often means a file was not re-encrypted with the new key. Does that help?

aymericbeaumet commented 6 years ago

I'm pretty sure the files are properly encrypted as Blackbox might fail the first time it is being invoked, but succeed the second time (without any change to the files). I'm trying to guess why that non-deterministic behavior is happening.

You never had a similar issue at SE?

tlimoncelli commented 6 years ago

Sadly (?) we've never seen this behavior at SE. We do a blackbox_decrypt_all_files in our CI system many times each day. I'd love to track this down since it seems like an edge-case that could bite others.

aymericbeaumet commented 6 years ago

I've been trying to find a consistent repro (on Linux/macOS), but did not succeed yet. Was hoping to find someone having the same problem when opening this issue.

Maybe it is related to the number of files we encrypt with Blackbox? I'm wondering whether it's putting too much pressure on the gpg daemon. Causing it to randomly not being able to find the appropriate private gpg key to decrypt.

tlimoncelli commented 6 years ago

I hadn't considered that. If it helps any... at SE, our repo with the most BB files has 61. The others have 2-3 files.

You could test your hypothesis pretty easily: put a "sleep 1" in the loop. That will slow it down considerably (not that you'd want it always that way).

aymericbeaumet commented 6 years ago

I cannot reproduce the bug locally on OSX, but I found a way on CircleCI (with the aforementioned base image).

First attempt

Hypothesis: corrupted gpg-agent / missing env var

Attempt: kill the gpg-agent / declare the GPG_TTY env var

for i in {1..9999} ; do
  echo "TRY: $i"; git clean -ffdx
  export GPG_TTY=$(tty); gpgconf --kill gpg-agent; killall gpg-agent
  blackbox_decrypt_all_files || break
done

Second attempt

Hypothesis: too much requests are being sent to the gpg-agent

Attempt: delay each loop turn in blackbox_postdeploy (/cc @TomOnTime)

The loop is the same as for the first attempt.

It takes longer but the decrypt eventually always fails.

I could not manage to make it fail with a 1s sleep (after ~30 minutes). Maybe I have not been patient enough.

Temporary fix

This is not perfect, but retrying a few times at least makes the build work.

for i in {1..3}; do
  blackbox_decrypt_all_files && exit 0
done
exit 1

Next steps

Investigate and try decrypt_file_overwrite.

Related readings

jbonzo commented 6 years ago

I've noticed similar problems recently, is this issue still relevant for @aymericbeaumet ?

aymericbeaumet commented 6 years ago

We are still facing this issue, and have been staying with the temporary fix ever since 🙄

rrggrr commented 6 years ago

Issue persists

coding-red-panda commented 6 years ago

I'm running into a very similar issue, especially when swapping between branches and keys getting added/removed as well from new employees. To give a more detailed overview of the problem:

And this happens to various projects, and we're talking about maybe 5 files maximum being encrypted in these projects. Killing the GPG agent sometimes helps but now always.

tlimoncelli commented 6 years ago

Interesting new issue!

Is this a problem with GPG or with Blackbox? Do you think gpg-agent is getting confused by git replacing so many files?

coding-red-panda commented 6 years ago

I honestly don't know. I can for example fire up gpg --edit-key <KEYID> and this works fine. But running decrypt commands or blackbox commands tells me that the wrong key is being used somehow.

I'm trying to figure out if gpg is not using something from the macOS keychain or something? I'm really running out of ideas at the moment.

The weird part is that I can work just fine other branches, or by copying the files. But as soon as I make a change everything breaks.

coding-red-panda commented 6 years ago

resolved the issue. The problem is caused by old public keys, that no longer have the private key available. Once we cleaned up those keys it all worked again

aya-inno commented 6 years ago

Had the same problem. I installed blackbox using debian and tried to decrypt the files. I used curses for my pinentry and restarted/killed the gpg agent then it worked for me.

My guess was I needed to restart my gpg agent when I changed my pinenty in the gpg-agent.conf

gpg in command line using pinentry-curses: https://chaosfreakblog.wordpress.com/2013/06/21/gpg-problem-with-the-agent-no-pinentry-solved/

and also thanks to this thread as for references.

Thanks

coding-red-panda commented 6 years ago

I actually ran into a curious problem again today. The blackbox failed to decrypt again, and even the normal gpg -d command didn't work after exchanging the keys.

I removed all duplicate entries on my system for invalid keys and noticed that multiple gpg-agents had been spawned. Once I killed those everything started to work again properly. It seems blackbox has some issue when multiple agents are spawned and keys are replaced for an admin. As if the wrong keys are being used for encrypting, or not all public known keys are used or something.

xavieramoros commented 5 years ago

I solved a similar issue by setting the GPG_TTY environment variable: 

GPG_TTY=$(tty)
export GPG_TTY

in my .zshrc file (.bashrc, ...), as seen in gnupg documentation: https://gnupg.org/documentation/manuals/gnupg-2.0/Invoking-GPG_002dAGENT.html https://gnupg.org/documentation/manuals/gnupg/Invoking-GPG_002dAGENT.html

Specially this line caught my attention:

You should always add the following lines to your .bashrc or whatever initialization file is used for all shell invocations:

GPG_TTY=$(tty) export GPG_TTY It is important that this environment variable always reflects the output of the tty command. For W32 systems this option is not required.

I was having similar issues, gpg: decryption failed: No secret key mostly when pulling from git encrypted files (but with no key added or removed, just reencrypt). Using blackbox_update_all_files was only solving the issue until the next git pull. Setting GPG_TTY seems to be definitive.

This post guided me to the solution: https://juliansimioni.com/blog/troubleshooting-gpg-git-commit-signing/

mig8447 commented 4 years ago

Just in case someone is going through the same error and none of the solutions above worked:

I had an Ubuntu 18.04 machine with the Gnome GUI and by default, gpg was using /usr/bin/pinentry-gnome3, which is a GUI-only pinentry not able to be accessed through SSH, which is exactly what i was doing. To fix it:

  1. Install pinentry-tty by executing (In Ubuntu's case): 
    sudo apt-get install pinentry-tty
  2. Set the GPG Agent's pinentry program to pinentry-tty by editing (or creating) the ~/.gnupg/gpg-agent.conf file and adding the following line: 
    pinentry-program /usr/bin/pinentry-tty
  3. Reload the GPG Agent by executing 
    gpg-connect-agent reloadagent /bye

References: