Increasing timeout values for creating an application does not work (Canceled desc = context canceled)

[tomisboy]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I am trying to create a simple application via the ArgoCD API (/api/v1/applications).

Creating the application fails with “Canceled desc = context canceled” after 30 seconds. It is always canceled when grpc.time_ms=29999.738 is about to reach 30 secounds

Although I have set the timeouts to 600s in all possible places, this threshold is ignored.

The timeouts occur randomly and affect the “generate manifests” as well as the repository pull: “repository not accessible”

I have installed ArgoCD over a Openshift Operator, where i configured the ENV:

  server:
    env:
      - name: ARGOCD_SERVER_REPO_SERVER_TIMEOUT_SECONDS
        value: '600'
      - name: ARGOCD_EXEC_TIMEOUT
        value: 600s
---
  controller:
    env:
      - name: ARGOCD_APPLICATION_CONTROLLER_REPO_SERVER_TIMEOUT_SECONDS
        value: '600'
      - name: ARGOCD_EXEC_TIMEOUT
        value: 600s
---
  repo:
    env:
      - name: ARGOCD_GIT_REQUEST_TIMEOUT
        value: 600s
      - name: ARGOCD_REPO_SERVER_PARALLELISM_LIMIT
        value: '0'
      - name: ARGOCD_EXEC_TIMEOUT
        value: 600s

---
  applicationSet:
    enabled: true
    env:
      - name: ARGOCD_APPLICATIONSET_CONTROLLER_REPO_SERVER_TIMEOUT_SECONDS
        value: '600'
      - name: ARGOCD_EXEC_TIMEOUT
        value: 600s

Version

sh-4.4$ argocd version
argocd: v2.12.3+6b9cd82
  BuildDate: 2024-09-13T16:43:33Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.22.5 (Red Hat 1.22.5-1.module+el8.10.0+22070+9237f38b)
  Compiler: gc
  Platform: linux/amd64
  ExtraBuildInfo: {Vendor Information: Red Hat OpenShift GitOps version: v1.14.0}

Logs

time="2024-10-29T14:07:03Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for frontend-bode-gnnesf5vtluphdzps2txx8 is invalid: InvalidSpecError: Unable to generate manifests in backstage_frontend: rpc error: code = Canceled desc = context canceled" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2024-10-29T14:06:33Z" grpc.time_ms=30002.523 span.kind=server system=grpc

time="2024-10-29T14:07:03Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for frontend-bode-jug6hxoia4yybhjolazujk is invalid: InvalidSpecError: repository not accessible: repositories not accessible: &Repository{Repo: \"https://***.git\", Type: \"git\", Name: \"\", Project: \"\"}: repo client error while testing repository: rpc error: code = Canceled desc = context canceled" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2024-10-29T14:06:33Z" grpc.time_ms=29999.738 span.kind=server system=grpc

[tomisboy]

I got my collection of the env from: https://argo-cd.readthedocs.io/en/stable/operator-manual/argocd-cmd-params-cm-yaml/ https://github.com/argoproj/argo-cd/blob/master/manifests/install.yaml#L23770C1-L23771C1 https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#argocd-repo-server:~:text=by%20using%20the-,ARGOCD_EXEC_TIMEOUT,-env%20variable.%20The

[andrii-korotkov-verkada]

Sorry, which container are the logs from? Is it app controller or something else?

Based on the code, one of ARGOCD_SERVER_REPO_SERVER_TIMEOUT_SECONDS, ARGOCD_APPLICATIONSET_CONTROLLER_REPO_SERVER_TIMEOUT_SECONDS, ARGOCD_APPLICATION_CONTROLLER_REPO_SERVER_TIMEOUT_SECONDS may be relevant. For notifications controller it's hardcoded as 5 seconds. But seems like all of them have a default 60 seconds.

One lead could be a part of the message grpc.code=InvalidArgument. So maybe there's a mistake in the request.

Can you walk me through the process of how you call the endpoint, e.g. what's the entry point and what are the params?