search

sbt / sbt-pgp

PGP plugin for sbt
Other
144 stars 56 forks source link

Signing with gpg must be done serially to avoid gpg-agent running out of memory #168

Closed smarter closed 4 years ago

smarter commented 5 years ago

When running publishSigned on aggregated projects, we might end up calling gpg multiple times in parallel, this can lead to the gpg-agent running out of "secmem" (secure memory) (see https://dev.gnupg.org/T4255) which can manifest itself with errors such as:

gpg: signing failed: Cannot allocate memory

(as usual with gpg, this error is very confusing: the system didn't run out of memory at all). This isn't just a theoretical concern, this is currently blocking the release of Dotty 0.19.0-RC1: https://dotty-ci.epfl.ch/lampepfl/dotty/1487/1/8

To fix this, an option auto-expand-secmem was added to gpg-agent.conf (https://dev.gnupg.org/T3530), according to that PR:

The --auto-expand-secmem option is available in 2.2. and master for quite some time. It works if libgcrypt 1.8.2 or newer is used.

Unfortunately, Ubuntu 18.04 uses libgcrypt 1.8.1, so this fix isn't usable on a lot of CI setups.

Therefore it seems like the only sane thing to do for sbt-pgp is to wrap calls to gpg into a synchronized block to make sure they're never executed in parallel.

eed3si9n commented 5 years ago

Would this go away if you set

useGpgAgent := false

given .gnupg/gpg.conf doesn't include use-agent?

smarter commented 5 years ago

From the gpg manpage on my system:

       --use-agent
       --no-use-agent
              This is dummy option. gpg always requires the agent.

So I'm going to guess no.

eed3si9n commented 4 years ago

Do you know what is the degree of parallelism here that causes the issue? Is it anything more than 1 or 16?

smarter commented 4 years ago

It happened when publishing dotty which is maybe 8 modules, but it's going to depend on a bunch of things like what is being signed and what parameters gpg-agent is using. So the most conservative thing to do is to be serial.

eed3si9n commented 4 years ago

ok.

@debasishg said he's hitting it too with a 16 subproject project.

debasishg commented 4 years ago

@smarter Regarding parameters of gpg-agent, here's my gpg-agent.conf 

$ cat gpg-agent.conf 
pinentry-program /usr/local/bin/pinentry-tty
default-cache-ttl 600
max-cache-ttl 7200

Do you have any suggestion that may improve the situation ? /cc @eed3si9n

smarter commented 4 years ago

My suggestion is to fix sbt-pgp, or to switch back to making it use bouncycastle (beware, the README instructions for doing that are incorrect, see https://github.com/sbt/sbt-pgp/issues/169).

eed3si9n commented 4 years ago

Here's my PR - https://github.com/sbt/sbt-pgp/pull/171