Git Fatal Error: PRNG is not seeded

[j-wozniack]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Upgrading from 2.11.5 to 2.12.3 I keep getting the same git error, that causes the repo server to fail to clone. When I go to check the repo in the list it shows as connected and healthy. However, when I roll back to 2.11.5 the applications sync and there is no issue with git.

I have checked all the recent issues for 2.12.x and not seen any specific to this.

To Reproduce

Upgrade from 2.11.5 to 2.12.3 using the argocd-helm chart, with a repository secret.

Expected behavior

I am able to upgrade from 2.11.5 to 2.12.3 without getting any git errors

Screenshots

Version

argocd: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T11:57:48Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: linux/amd64

Logs

Repo Server:

time="2024-09-10T14:13:44Z" level=debug msg="Checking out revision dd260c2b386674d0067a09c2ff94976b7c1bd5d7" skipFetch=false
time="2024-09-10T14:13:44Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/_argocd-repo/8dc5008e-d688-4d5a-a55a-5075b180d400 execID=7d05a
time="2024-09-10T14:13:44Z" level=debug duration=4.641649ms execID=7d05a
time="2024-09-10T14:13:44Z" level=error msg="`git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=7d05a
time="2024-09-10T14:13:44Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/_argocd-repo/8dc5008e-d688-4d5a-a55a-5075b180d400 operation_name="exec git" time_ms=4.723298
time="2024-09-10T14:13:44Z" level=error msg="finished unary call with code Unknown" error="failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-09-10T14:13:44Z" grpc.time_ms=229.871 span.kind=server system=grpc

Server:

time="2024-09-10T14:11:06Z" level=debug msg="ssh://<user>@<repo> has credentials"
time="2024-09-10T14:11:06Z" level=error msg="finished unary call with code Unknown" error="rpc error: code = Unknown desc = error acquiring repo lock: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." grpc.code=Unknown grpc.method=RevisionMetadata grpc.service=application.ApplicationService grpc.start_time="2024-09-10T14:11:06Z" grpc.time_ms=12.147 span.kind=server system=grpc

[nitishfy]

I'd like to take this up.

[blakepettersson]

Could it be a permutation of #19587?

[j-wozniack]

Could it be a permutation of #19587?

I saw that issue before posting this. I verified our credential for the repo is not scoped to a project. We have multiple projects that all share the same repo. But I doubled check it is not scoped incorrectly.

[j-wozniack]

For more context (not sure if it will help). The repo is an AWS code commit repo. We are using ssh keys in order to clone.

[nitishfy]

I'm already working on couple of issues in Argo CD right now, so if anyone else would like to take this up, feel free to do that. Thanks!

[pzhen01]

We got the same error when we did upgrade from 2.9.17 to 2.12.3 All applications failed to get the "status" with the following error

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

After we rollback, everything is back to normal. Is there any workaround?

[j-wozniack]

We got the same error when we did upgrade from 2.9.17 to 2.12.3 All applications failed to get the "status" with the following error

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

After we rollback, everything is back to normal. Is there any workaround?

We have yet to find a work around, we simply rolled back and waiting to hear back on this issue. It isn't strictly urgent but we would like to be able to upgrade eventually.

[blakepettersson]

@j-wozniack can you post the application spec that's failing? along with the secret it's supposed to use? redacted where applicable

[j-wozniack]

@j-wozniack can you post the application spec that's failing? along with the secret it's supposed to use? redacted where applicable

For more reference, we are using the helmfile plugin: https://github.com/travisghansen/argo-cd-helmfile

Here is the repo secret we are using:

enableLfs: false
insecure: true
name: argo-cd-istio-ssh-repo
sshPrivateKey: |
-----BEGIN OPENSSH PRIVATE KEY-----
<ssh key>
-----END OPENSSH PRIVATE KEY-----

url: ssh://<user>@git-codecommit.us-west-1.amazonaws.com/v1/repos/helmfile

Application Spec:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: "2024-09-24T09:40:00Z"
  generation: 5
  name: istio-base
  namespace: argocd-system
  resourceVersion: "6074450"
  uid: b159c700-0861-4611-ba82-de2730cc1a64
spec:
  destination:
    namespace: istio-system
    server: https://kubernetes.default.svc
  project: default
  source:
    path: apps/istio-base/
    plugin:
      env:
      - name: HELMFILE_GLOBAL_OPTIONS
        value: -e personal
      - name: HELMFILE_TEMPLATE_OPTIONS
        value: --include-crds
      name: helmfile
    repoURL: ssh://<user>@git-codecommit.us-west-1.amazonaws.com/v1/repos/helmfile
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
status:
  conditions:
  - lastTransitionTime: "2024-09-24T09:48:48Z"
    message: "Failed to load target state: failed to generate manifest for source
      1 of 1: rpc error: code = Unknown desc = failed to initialize repository resources:
      rpc error: code = Internal desc = Failed to fetch default: `git fetch origin
      --tags --force --prune` failed exit status 128: PRNG is not seeded\r\nfatal:
      Could not read from remote repository.\n\nPlease make sure you have the correct
      access rights\nand the repository exists."
    type: ComparisonError
  controllerNamespace: argocd-system
  health:
    status: Healthy
  reconciledAt: "2024-09-24T09:48:48Z"
  summary: {}
  sync:
    comparedTo:
      destination:
        namespace: istio-system
        server: https://kubernetes.default.svc
      source:
        path: apps/istio-base/
        plugin:
          env:
          - name: HELMFILE_GLOBAL_OPTIONS
            value: -e personal
          - name: HELMFILE_TEMPLATE_OPTIONS
            value: --include-crds
          name: helmfile
        repoURL: ssh://<user>@git-codecommit.us-west-1.amazonaws.com/v1/repos/helmfile
        targetRevision: main
    status: Unknown

[neiljain]

the node is running the following if that helps with the investigation with a 5.4.0 aws fips kernel

  OS Image:                   Ubuntu 20.04.6 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.16
  Kubelet Version:            v1.28.10
  Kube-Proxy Version:         v1.28.10

argocd version

Argo CD v2.12.3+6b9cd82
Build Date 2024-08-27T11:57:48Z
Go Version go1.22.4
Go Compiler gc
Platform linux/amd64
jsonnet v0.20.0
kustomize v5.4.2 2024-05-22T15:19:38Z
Helm v3.15.2+g1a500d5
kubectl v0.29.6

argocd-repo-server logs

{"level":"info","msg":"manifest cache miss: \u0026ApplicationSource{RepoURL:ssh://git@gitlab/**argo-repo.git,Path:.,TargetRevision:master,Helm:nil,Kustomize:nil,Directory:nil,Plugin:\u0026ApplicationSourcePlugin{Name:custom-plugin,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,}/foo","time":"2024-10-03T21:08:16Z"}
{"dir":"/tmp/_argocd-repo/68a348dd-6af1-43e6-ac38-9e976047b861","execID":"f2f80","level":"info","msg":"git cat-file -t 0c09beac9f63c902d18114060b942d07bb6b71c4","time":"2024-10-03T21:08:16Z"}
{"args":"[git cat-file -t 0c09beac9f63c902d18114060b942d07bb6b71c4]","dir":"/tmp/_argocd-repo/68a348dd-6af1-43e6-ac38-9e976047b861","level":"info","msg":"Trace","operation_name":"exec git","time":"2024-10-03T21:08:16Z","time_ms":1.014093}
{"dir":"/tmp/_argocd-repo/68a348dd-6af1-43e6-ac38-9e976047b861","execID":"cf5be","level":"info","msg":"git fetch origin --tags --force --prune","time":"2024-10-03T21:08:16Z"}
{"execID":"2429a","level":"error","msg":"`git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.","time":"2024-10-03T21:08:16Z"}
{"args":"[git fetch origin --tags --force --prune]","dir":"/tmp/_argocd-repo/4065359c-5588-41ec-98c8-5098c8789d6d","level":"info","msg":"Trace","operation_name":"exec git","time":"2024-10-03T21:08:16Z","time_ms":4.067582}
{"error":"failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.","grpc.code":"Unknown","grpc.method":"GenerateManifest","grpc.service":"repository.RepoServerService","grpc.start_time":"2024-10-03T21:08:16Z","grpc.time_ms":7.571,"level":"error","msg":"finished unary call with code Unknown","span.kind":"server","system":"grpc","time":"2024-10-03T21:08:16Z"}
{"execID":"cf5be","level":"error","msg":"`git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.","time":"2024-10-03T21:08:16Z"}
{"args":"[git fetch origin --tags --force --prune]","dir":"/tmp/_argocd-repo/68a348dd-6af1-43e6-ac38-9e976047b861","level":"info","msg":"Trace","operation_name":"exec git","time":"2024-10-03T21:08:16Z","time_ms":3.893785}
{"error":"failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.","grpc.code":"Unknown","grpc.method":"GenerateManifest","grpc.service":"repository.RepoServerService","grpc.start_time":"2024-10-03T21:08:16Z","grpc.time_ms":9.518,"level":"error","msg":"finished unary call with code Unknown","span.kind":"server","system":"grpc","time":"2024-10-03T21:08:16Z"}

works fine after rolling back to

Argo CD v2.11.7+e4a0246
Build Date 2024-07-24T09:33:49Z
Go Version go1.21.10
Go Compiler gc
Platform linux/amd64
jsonnet v0.20.0
kustomize v5.2.1 2023-10-19T20:13:51Z
Helm v3.14.4+g81c902a
kubectl v0.26.11

[noskovao]

We are experiencing the same issue on v2.12.3: failed exit status 128: PRNG is not seeded Is there are a way to WA it?

[showalter]

It seems like this might only occur when running ArgoCD on FIPS-enabled hosts, and may be caused by the switch to Ubuntu 24.04 as a base image which was done in #18093. I built v2.12.4 with Ubuntu 22.04 as the base image, and that appears to have worked.

[reegnz]

We are experiencing this on FIPS-enabled hosts with ArgoCD v2.12.4. We didn't experience the issue on non-FIPS-ed hosts:

ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: PRNG is not seeded fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

[neiljain]

tried upgrading to 2.12.6 and still have the same issue on fips-enabled hosts.

[blakepettersson]

It seems like OpenSSL 3 (which is bundled with Ubuntu 24.04) will not run with FIPS-enabled kernels unless OpenSSL has the FIPS provider library bundled with it. Canonical is nice enough to have that locked away in Ubuntu Pro (where we would need to mount a secret in order to install a FIPS-enabled OpenSSL).

[nkalscheuer]

On Ubuntu 24, you can compile and install the fips module and install it into your openssl installation. Video on it here: https://www.youtube.com/watch?v=geAtEXbHaFg One step missing there is to move the fips.so file to the same folder as the other OpenSSL libs (usually here: /usr/lib/x86_64-linux-gnu/ossl-modules/) Also make sure you set up the openssl.cnf file correctly from the video. It's a little confusing.

[andrii-korotkov-verkada]

@crenshaw-dev, how big are the downsides of reverting to Ubuntu 22? @blakepettersson, does it effectively mean that Ubuntu won't have necessary support for this in future versions, unless using pro version?

[blakepettersson]

@andrii-korotkov-verkada

@blakepettersson, does it effectively mean that Ubuntu won't have necessary support for this in future versions, unless using pro version?

There are basically two* options with Ubuntu 24

1. Pay Canonical to get access to the pro repos, which has a precompiled version of openssl with fips support
2. Try to compile the FIPS-module yourself, as @nkalscheuer suggests

Both of those options implies FIPS-users would need to create a custom Dockerfile and build their own Argo CD Docker images.

*there's a third option, which is that FIPS-users would pay some third-party vendor (there are a couple out there) that distributes pre-built "hardened Argo" images with FIPS-support

[andrii-korotkov-verkada]

How much do they charge for the pro version?

[blakepettersson]

TBH the subscribe page is really confusing. What I think is the case is that the subscription is free if the intent is to have <= 5 Argo instances running, otherwise there's a myriad of options. I guess the idea is to force potential users to contact sales.

[andrii-korotkov-verkada]

Can we create this custom Dockerfile for them and start distributing FIPS-compatible image as well?

[blakepettersson]

Can we create this custom Dockerfile for them and start distributing FIPS-compatible image as well?

🤷, if there is a way to build the FIPS-module that still works for non-FIPS users, then I guess someone can take a stab at following the guide @nkalscheuer posted and submit a PR for that.

If this is somehow not compatible with non-FIPS usages, I guess the other option would be to build a FIPS-version of Argo CD along with our normal images.