Signed commit failure

[almi33]

Version

6.8.0.202311291450-r

Operating System

Windows

Eclipse version

2023-12 (4.30.0)

Bug description

Failing to perform signed commit using external gpg. See log in "Relevant log output" section.

Actual behavior

Failed to perform signed commit

Expected behavior

Expected to succeed. Command line git performs signed commit using the same external gpg, user name and mail successfully

Relevant log output

!ENTRY org.eclipse.egit.ui 4 0 2024-01-10 18:31:17.230
!MESSAGE An internal error occurred
!STACK 0
org.eclipse.jgit.api.errors.JGitInternalException: External program failed ([GNUPG:] KEY_CONSIDERED A1427A7A21F2545AAC19BFB1661C4889FE30F3D1 2
[GNUPG:] BEGIN_SIGNING H8
[GNUPG:] PINENTRY_LAUNCHED 11288 w32 1.2.1 - - - - 0/0 -
[GNUPG:] SIG_CREATED D 1 8 00 1704907867 A1427A7A21F2545AAC19BFB1661C4889FE30F3D1
): C:\Users\some_user\AppData\Local\Programs\gnupg\bin\gpg.exe -bsau <some_user@mail.org> --batch --no-tty --status-fd 2 --output -
    at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.signObject(ExternalGpgSigner.java:191)
    at org.eclipse.jgit.api.CommitCommand.sign(CommitCommand.java:336)
    at org.eclipse.jgit.api.CommitCommand.call(CommitCommand.java:283)
    at org.eclipse.egit.core.op.CommitOperation.commit(CommitOperation.java:259)
    at org.eclipse.egit.core.op.CommitOperation$1.run(CommitOperation.java:211)
    at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2453)
    at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2478)
    at org.eclipse.egit.core.op.CommitOperation.execute(CommitOperation.java:220)
    at org.eclipse.egit.ui.internal.commit.CommitJob.run(CommitJob.java:135)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: java.io.IOException: External program failed ([GNUPG:] KEY_CONSIDERED A1427A7A21F2545AAC19BFB1661C4889FE30F3D1 2
[GNUPG:] BEGIN_SIGNING H8
[GNUPG:] PINENTRY_LAUNCHED 11288 w32 1.2.1 - - - - 0/0 -
[GNUPG:] SIG_CREATED D 1 8 00 1704907867 A1427A7A21F2545AAC19BFB1661C4889FE30F3D1
): C:\Users\some_user\AppData\Local\Programs\gnupg\bin\gpg.exe -bsau <some_user@mail.org> --batch --no-tty --status-fd 2 --output -
    at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.runProcess(ExternalGpgSigner.java:135)
    at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.signWithGpg(ExternalGpgSigner.java:297)
    at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.signObject(ExternalGpgSigner.java:189)
    ... 9 more
Caused by: java.io.IOException: External GPG did not return a valid signature; got: -----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoUJ6eiHyVFqsGb+xZhxIif4w89EFAmWe1FsACgkQZhxIif4w
89H/xA/8Dbq4pr+wWSMVzcsHwY+gii1FDgbg5zHut6WX+hK8d11mrJTlPbqbyUPq
QFouwB9y3wtIluroF6Ics89wNAlRCBGqGLuHJCpjVHGhyMoyhp5l7X8gF7TORf+a
X/MH3h0pAyKzprRD7sZ3A9RmPL9MBX25KwieuTBbZdhBvWRC2j5dj3RsCBSwn4vn
ZbSB0JawNWSX6qsttH2+8FSgWkZFred2juIUkg0NmJCPF3Lg8GBTiTo366X27/TN
1+hjGl6yUl5pwXKpl2Z4A0fcEbu2A6osmLI64hB09reD7JdlHAtsLUty7Y9mTRyB
qnEsmw5/p7nogFsrAvAmLapD/fm9K6hiu3rzGOpywvFPv45EJzL2HQgzYKyU70fr
ODMVZJET0HKx25LO20rURmuZR+PcSJUE7d/9mjYbpaF3k+5it5kzfRQF2vcV0mGA
dWpnbYFRA8HcvOn6nZIKzNth1roP8lluFvT2AXtLdvLJ27SfUG6JCjgcBe1H5dDY
q9xARe1S78x+uwSrZzTjplit+LHcP6odHqybMJdSL9RiBq4iaYkyf7m8cydKE5Nl
ls9xBBc0ZcNG5m53UtzTU2CVsMzW36VOZUsIUJ5Q1hWhlt5jfBBc9wleXCRCvD1q
FOD5Nul9mMWgCqsouO3nVb0Xa5j64caOiTFD2FL0qoT5TrzMqBg=
=hrDq
-----END PGP SIGNATURE-----

    at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.lambda$1(ExternalGpgSigner.java:326)
    at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.runProcess(ExternalGpgSigner.java:110)
    ... 11 more

Other information

No response

[tomaswolf]

Earlier report at Eclipse bugzilla 581545.

We still use Bouncy Castle to verify the result of the external GPG, and apparently the JGit Bouncy Castle verifier cannot find the public key.

[tomaswolf]

The issue might be caused by different GPG installations using different directories being present. Gpg4Win uses %APPDATA%\gnupg, but AFAIK git-for-windows also bundles a GPG, which might be using something else (probably .gnupg in the user's home directory). We use %APPDATA%\gnupg if it exists, otherwise .gnupg in the home directory. Probably we should use both if both exist.

[tomaswolf]

@almi33 : could you please install EGit nightly and try again? It won't have fixed the problem, but it should give a slightly more informative error message.

To install EGit nightly, go in Eclipse to "Help->Install New Software..." and use the p2 repository URL https://download.eclipse.org/egit/updates-nightly .

If the error messages indicates "no public key found", figure out which directory is used by gpg for storing and finding its keys. Then try launching Eclipse with environment variable GNUPGHOME set to that directory. If it then works we know at least that the problem is indeed that EGit/JGit is looking in the wrong directory for the public key.

[almi33]

I am using Gpg4Win. It is the only gpg in the path. Yes I do have Git installed also with gpg bundled, but bundled gpg is not in the path and when I used Git for signed commit I have explicitly pointed to gpg from Gpg4Win using gpg.program key in configuration file. On my PC, which is running Windows 11, gpg itself is located in "C:\Users\some_user\AppData\Local\Programs\gnupg" while keys are located in "C:\Users\some_user\AppData\Local\gnupg". And just for clarity "APPDATA=C:\Users\some_user\AppData\Roaming".

Nightly build shows "no public key found".

Attempt to perform signed commit with GNUPGHOME=C:\Users\some_user\AppData\Local\gnupg has failed with message

External program failed (2: gpg: keybox 'C:\Users\some_user\AppData\Local\gnupg\pubring.kbx' created gpg: C:\Users\some_user\AppData\Local\gnupg\trustdb.gpg: trustdb created gpg: error retrieving 'some_user@mail.org' via WKD: No data gpg: error reading key: No data ): C:\Users\some_user\AppData\Local\Programs\gnupg\bin\gpg.exe --locate-keys --with-colons --batch --no-tty some_user@mail.org

and stack trace

java.io.IOException: External program failed (2: gpg: keybox 'C:\Users\some_user\AppData\Local\gnupg\pubring.kbx' created gpg: C:\Users\some_user\AppData\Local\gnupg\trustdb.gpg: trustdb created gpg: error retrieving 'some_user@mail.org' via WKD: No data gpg: error reading key: No data ): C:\Users\some_user\AppData\Local\Programs\gnupg\bin\gpg.exe --locate-keys --with-colons --batch --no-tty some_user@mail.org at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.runProcess(ExternalGpgSigner.java:104) at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.canLocateSigningKey(ExternalGpgSigner.java:229) at org.eclipse.egit.ui.internal.credentials.SignatureUtils.checkSigningKey(SignatureUtils.java:93) at org.eclipse.egit.ui.internal.credentials.SignatureUtils.checkSigningKey(SignatureUtils.java:53) at org.eclipse.egit.ui.internal.dialogs.CommitMessageComponent.checkCommitInfo(CommitMessageComponent.java:678) at org.eclipse.egit.ui.internal.staging.StagingView.internalCommit(StagingView.java:4817) at org.eclipse.egit.ui.internal.staging.StagingView.commit(StagingView.java:4780) at org.eclipse.egit.ui.internal.staging.StagingView$23.widgetSelected(StagingView.java:1279) at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:252) at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:89) at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4273) at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1066) at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4071) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3659) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$5.run(PartRenderingEngine.java:1155) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:342) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1046) at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:155) at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:648) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:342) at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:555) at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:173) at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:152) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:208) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:651) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:588) at org.eclipse.equinox.launcher.Main.run(Main.java:1459)

Also after that a lot of odd files have appeared in C:\Users\some_user\AppData\Local\gnupg.

[tomaswolf]

keys are located in "C:\Users\some_user\AppData\Local\gnupg".

Doesn't look like that was the case, judging from the exceptions you got when you tried to use that directory. Without that setting, you got further: gpg.exe did produce a signature, just when EGit tried to verify that signature, it could not find the key.

And just for clarity "APPDATA=C:\Users\some_user\AppData\Roaming".

The latest Gpg4Win documentation still claims %APPDATA%\gnupg was the default directory: https://www.gpg4win.org/doc/en/gpg4win-compendium_28.html

Nightly build shows "no public key found".

So at least we know now for sure that the problem is that somehow JGit cannot find the public key.

I have no Windows 11. I might be able to try in a WIndows 10 VM. What Gpg4Win version do you use? Perhaps some way of storing keys has changed, and JGit should try a new way to find them.

In particular, I see in the change log that Gpg4Win 4.2.0 now apparently uses an sqlite database to store public keys. That of course is not handled at all by JGit. If that's the root cause of the problem, then JGit has a fundamental problem, and all this signing or verifying signatures with Bouncy Castle won't work anymore at all with Gpg4Win. (Unless we'd include the 12MB sqlite-jdbc driver and read that database from Java. I don't want to go there... it'd be very dependent on the database schema.) For EGit, we might also use the external gpg for verifying signatures, and perhaps not verify the created signature at all (except checking that it starts with the expected "-----BEGIN PGP SIGNATURE-----" line).

[almi33]

The latest Gpg4Win documentation still claims %APPDATA%\gnupg was the default directory

My bad, haven't noticed this folder. So, I have run Eclipse again with GNUPGHOME=%APPDATA%\gnupg. This time it failed similar to the original one with log:

!ENTRY org.eclipse.egit.ui 4 0 2024-01-14 18:46:19.839 !MESSAGE An internal error occurred !STACK 0 org.eclipse.jgit.api.errors.JGitInternalException: External program failed ([GNUPG:] KEY_CONSIDERED A1427A7A21F2545AAC19BFB1661C4889FE30F3D1 2 [GNUPG:] BEGIN_SIGNING H8 [GNUPG:] PINENTRY_LAUNCHED 34856 w32 1.2.1 - - - - 0/0 - [GNUPG:] SIG_CREATED D 1 8 00 1705254372 A1427A7A21F2545AAC19BFB1661C4889FE30F3D1 ): C:\Users\some_user\AppData\Local\Programs\gnupg\bin\gpg.exe -bsau some_user@mail.org --batch --no-tty --status-fd 2 --output - at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.signObject(ExternalGpgSigner.java:191) at org.eclipse.jgit.api.CommitCommand.sign(CommitCommand.java:336) at org.eclipse.jgit.api.CommitCommand.call(CommitCommand.java:283) at org.eclipse.egit.core.op.CommitOperation.commit(CommitOperation.java:259) at org.eclipse.egit.core.op.CommitOperation$1.run(CommitOperation.java:211) at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2453) at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2478) at org.eclipse.egit.core.op.CommitOperation.execute(CommitOperation.java:220) at org.eclipse.egit.ui.internal.commit.CommitJob.run(CommitJob.java:135) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63) Caused by: java.io.IOException: External program failed ([GNUPG:] KEY_CONSIDERED A1427A7A21F2545AAC19BFB1661C4889FE30F3D1 2 [GNUPG:] BEGIN_SIGNING H8 [GNUPG:] PINENTRY_LAUNCHED 34856 w32 1.2.1 - - - - 0/0 - [GNUPG:] SIG_CREATED D 1 8 00 1705254372 A1427A7A21F2545AAC19BFB1661C4889FE30F3D1 ): C:\Users\some_user\AppData\Local\Programs\gnupg\bin\gpg.exe -bsau some_user@mail.org --batch --no-tty --status-fd 2 --output - at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.runProcess(ExternalGpgSigner.java:135) at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.signWithGpg(ExternalGpgSigner.java:297) at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.signObject(ExternalGpgSigner.java:189) ... 9 more Caused by: java.io.IOException: Cannot validate signature returned by external GPG; got message "No public key found to verify the signature" for: -----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoUJ6eiHyVFqsGb+xZhxIif4w89EFAmWkHeQACgkQZhxIif4w 89FbSw/+ItTPhClHkPiwwUBiAzFtqj0zlcaeSvIMseDmKywItj135jWVMRmshPi7 G77zRJ+CYt126URqfdwrlDsyTnnzqMtwlQ1XUtnrxT/qNM2E4QTrccO0mLqDEUSU ClpiaT/V4Zt0FVrMy2RV5YMq7FAJmxtk1rXxejnNbQhqzQxEBiL2IHo6PPqJDLzV eRB8wtIqS6VJhCiSQNpuPiSP8zGiMBnhrhbFHYL5jofBTnIl9PUBSD1bPoUpo267 JgfTVOV9199Mmihb7p4oOt9EWdTWv0Bej+VABIA/MoAl7zDkqTb5WhXgJCY8SjgJ VOUFt5eAKZf4jf1wJu5A4PNRrnpLo4xocb9U6zLOxe54oLj8IiORo0sZLxKm/K4G PAUiXtVZOVe+TZZAqVHJ89VqogqA0ArnRbwsdqYlVOPCMov3vwAnptbeVH1tPmL3 E2sYSoJmqx3vIt1hJ0siq/lmKZ4rsHtzzqtAEecVb7LNyEiSIoWQcDaa6HuxWRm5 b1b8+YQ70RBfyC+g1umhvrx0CjI/zTbA5mCiNaAqTknIFXn7PGfSHuPVMTcpVt0G VuuImSqCtVayaEmST9DQ2FHe3qO+IuhW22uneBvb9k4NitNf9VlKLYoqyLoVTSaB cMN2BaO0REnK+NSlBtheSPcE3sH6qU3gAhmdyAUda+5i1P1c1rs= =51Lg -----END PGP SIGNATURE-----

at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.lambda$1(ExternalGpgSigner.java:334)
at org.eclipse.egit.core.internal.signing.ExternalGpgSigner.runProcess(ExternalGpgSigner.java:110)
... 11 more

What Gpg4Win version do you use?

Here is the output from "gpg --version": gpg (GnuPG) 2.4.3 libgcrypt 1.10.2 Copyright (C) 2023 g10 Code GmbH License GNU GPL-3.0-or-later https://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Home: C:\Users\some_user\AppData\Roaming\gnupg Supported algorithms: Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH, CAMELLIA128, CAMELLIA192, CAMELLIA256 Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224 Compression: Uncompressed, ZIP, ZLIB, BZIP2

[tomaswolf]

What files are inside %APPDATA%\gnupg? In particular, is there a pubring.db file? (And no pubring.kbx?)

If there are any *.conf files, do any of them contain a line use-keyboxd?

[almi33]

What files are inside %APPDATA%\gnupg?

Here is what DIR shows: 2023-12-17 14:39 13 common.conf 2024-01-06 14:16 \

crls.d 2024-01-06 18:30 0 gnupg_spawn_agent_sentinel.lock 2024-01-06 14:15 0 gnupg_spawn_dirmngr_sentinel.lock 2024-01-06 14:15 0 gnupg_spawn_keyboxd_sentinel.lock 2024-01-06 19:29 \ openpgp-revocs.d 2024-01-06 19:35 \ private-keys-v1.d 2024-01-06 19:37 \ public-keys.d 2024-01-06 19:35 687 sshcontrol 2024-01-06 19:37 1.360 trustdb.gpg 2024-01-06 14:15 0 trustdb.gpg.lock

In particular, is there a pubring.db file? (And no pubring.kbx?)

File pubring.db is located in public-keys.d folder. File pubring.kbx is missing.

If there are any *.conf files, do any of them contain a line use-keyboxd?

File common.conf contains 'use-keyboxd' line. It is the only line in the file.

[tomaswolf]

Thank you!

So there we have it: this Gpg4Win installation uses the new sqlite database to store the public keys. JGit doesn't know about that way of storing public keys, so it cannot find the public key for verifying the signature.

EGit will need to use the external GPG executable also for verifying signatures. JGit will need at least a caveat in some documentation (perhaps in the readme of org.eclipse.jgit.gpg.bc) that the Bouncy Castle GPG signing does not work if GPG uses the keybox daemon.

[ronnypolley]

Hi @tomaswolf,

I face the same issue with gpg set to be used:

So this is already the external program, which is running well, when using the command from the stacktrace itself.

gpg.exe -bsau <KEY> --batch --no-tty --status-fd 2 --output -<somefile>

[tomaswolf]

As I wrote before: EGit verifies that the result of calling the external GPG is a valid signature. It uses Bouncy Castle to verify the signature, and that code cannot find the public key because when GPG uses the keyboxd daemon, it stores public keys in a place and format that JGit doesn't know about. As I also wrote the fix will be to verify signatures via the external GPG instead of via Bouncy Castle.

[tomaswolf]

Gerrit change 1176329 should fix this.

[almi33]

Great. When to expect fix at any update site?

[tomaswolf]

If all goes well with the EGit 6.9 release on 2024-03-13. EGit nightly may have it earlier. But there are two JGit changes that need to go in first.

[tomaswolf]

Gerrit change 1176329 is merged.