search

opendevstack / ods-jenkins-shared-library

Shared Jenkins library which all ODS projects & components use - provisioning, SonarQube code scanning, Nexus publishing, OpenShift template based deployments and repository orchestration
Apache License 2.0
73 stars 57 forks source link

Jenkins sometimes fails to decrypt helm secrets (in the roll out stage in the orchestration pipeline #1028

Open matzehecht opened 1 year ago

matzehecht commented 1 year ago

Describe the bug Since we switched to helm our pipeline sometimes fails to decrypt our helm secrets as it can not find the key to decrypt it (added the error log under "log output").

I know that everything is configured as it should, because this happens only now and than. Also I can see in the logs, that it decrypts the secrets successfully in the helm secrets diff call but than fails to do so in the helm secrets upgrade. This is not urgent as rerunning the pipeline solves it. But at least the development team should be aware of it!

To Reproduce Steps to reproduce the behavior:

  1. Configure a ODS project (by adding helm secrets to a project, store the key in an openshift secret and sync it to jenkins).
  2. Start an orchestration pipeline and hope you are lucky enough to encounter this bug.

Expected behavior decrypting the secrets should work all the time.

Affected version (please complete the following information):

Log Output (ensure to remove any confidential information like tokens, project names, etc.

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  <PRIVATE_KEY_FINGERPRINT>: FAILED
    - | could not decrypt data key with PGP key:
      | github.com/ProtonMail/go-crypto/openpgp error: Could not
      | load secring: open /home/jenkins/.gnupg/secring.gpg: no such
      | file or directory; GPG binary error: exit status 2

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.
matzehecht commented 1 year ago

@serverhorror This is the issue for the bug we previously talked about.

serverhorror commented 1 year ago

rerunning the pipeline solves it

Tough one, sounds like some race condition. I'll try and take a look.

serverhorror commented 1 year ago

@matzehecht, @braisvq1996 sorry for not getting back earlier, did opendevstack/ods-quickstarters#946 fix this issue?

matzehecht commented 1 year ago

@matzehecht, @braisvq1996 sorry for not getting back earlier, did opendevstack/ods-quickstarters#946 fix this issue?

@serverhorror As mentioned in my comment on the mentioned PR those issues are not related. You can find all information in the other comment or the description of the respective issues. In short: The other issue/PR affected the import of gpg keys. This issue happens while reading the successfully imported gpg key. Also this issue happens only some times and the other issue happened every time. So no: The other issue/PR does not fix the issue.

serverhorror commented 10 months ago

Can someone, with the technical permissions, label this with help-wanted? I have no idea how to fix this :(