Argo doesn't reload repo-creds secrets

[maartengo]

Checklist:

• [X] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• [X] I've included steps to reproduce the bug.
• [X] I've pasted the output of argocd version.

Describe the bug Changing the repository URL in the repository secret isn't recognized by ArgoCD until a reboot. We've only tested this with Repository Credentials.

Reproduction: (in my case)

1. Create an Azure DevOps project with a space in the name Prep Project.2
2. Create a git repository with, or without, a space in the name SomeGitRepo
3. Setup a 'repo-creds' credential with the following url https://dev.azure.com/<yourorgname>/ (orgname without spaces)
4. Setup a specification for the Argo project and application (see template blow, replace the projectname and giturl)
   1. For the git url, use something like https://dev.azure.com/<yourorgname>/Prep Project.2/_git/SomeGitRepo
   2. Notice the space in the URL
5. Apply the repo-creds, apply the template
   1. An application should now be created that produces the following error
   2. rpc error: code = Internal desc = Failed to fetch 3f884e25c735231ed489a8efdd1e041d36ec62b0: `git fetch origin --tags --force` failed exit status 128: fatal: unable to access 'https://dev.azure.com/<yourorgname>/Prep Project.2/_git/SomeGitRepo/': The requested URL returned error: 400
6. If you now change and apply the template giturl to have %20 instead of spaces, you will get the same error

Template used:

# search + replace
# https://dev.azure.com/YourOrgName/Prep Project.2/_git/SomeGitRepo/ (gitUrl), Prep Project.2 (project Name)
apiVersion: 'argoproj.io/v1alpha1'
kind: 'Application'
metadata:
  name: 'Prep Project.2'
  namespace: 'argocd'
spec:
  project: 'Prep Project.2'
  source:
    directory:
      recurse: true
    repoURL: 'https://dev.azure.com/YourOrgName/Prep Project.2/_git/SomeGitRepo/'
    targetRevision: 'HEAD'
    path: '.' # replace this if you want to only sync a specific folder
  destination:
    namespace: 'Prep Project.2'
    server: 'https://kubernetes.default.svc'
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
---
apiVersion: 'argoproj.io/v1alpha1'
kind: 'AppProject'
metadata:
  name: 'Prep Project.2'
  namespace: 'argocd'
  # Finalizer that ensures that project is not deleted until it is not referenced by any application
  finalizers:
    - 'resources-finalizer.argocd.argoproj.io'
spec:
  # Project description
  description: 'Project filter for Prep Project.2'
  # Allow manifests to deploy from any repos
  sourceRepos:
    - '*'
  # Allow all resources to be deployed
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  # Allow applications referencing this project to deploy to all namespaces in the local server
  destinations:
    - namespace: '*'
      server: 'https://kubernetes.default.svc'
---
apiVersion: 'v1'
kind: 'Secret'
metadata:
  name: 'Prep Project.2-repo'
  namespace: 'argocd'
  labels:
    argocd.argoproj.io/secret-type: 'repository'
stringData:
  url: 'https://dev.azure.com/YourOrgName/Prep Project.2/_git/SomeGitRepo/'

Expected behavior

You'd expect that, after updating the git url to replace https://dev.azure.com/YourOrgName/Prep Project.2/_git/SomeGitRepo/ (contains a space) with https://dev.azure.com/YourOrgName/Prep%20Project.2/_git/SomeGitRepo/ (space is now %20) that the application will successfully pull the git repo.

A git URL that contains a space converted to %20 always works, so long as this is the first time you configure that URL. I'd expect it to also work if you've first configured the wrong URl in the repository secret.

The issue is that Argo doesn't load the new repository secret, which makes it so that there is a mismatch between the application and the secret. I'd expect Argo to be able to see that the secret is updated and using that instead of the original secret.

A workaround we currently use:

• Configure the template with the correct git URL:
• Apply the template
• Reboot Argo
• Delete the application
• Apply the template again (or have it sync back)
• It will work as if you did it correct the first time

Version

argocd: v2.1.6+a346cf9
  BuildDate: 2021-10-28T19:59:40Z
  GitCommit: a346cf933e10d872eae26bff8e58c5e7ac40db25
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc       
  Platform: linux/amd64

Running version 3.26.5 of helm chart at https://argoproj.github.io/argo-helm

[jannfis]

I think we do cache the repository secrets (like we do with the argocd-secret), but don't setup watches for the cache updates. This effectively never updates the credentials we initially loaded from the secret, until a restart of argocd-server or argocd-application-controller happens.

[sbose78]

Hi, Is this a regression or did this never work?

[jannfis]

Is this a regression or did this never work

It's dependent on how you define regression in this case :) The previous way of configuring the repositories, e.g. adding them into argocd-secret was auto-reloading. The new method is not. So in a way, it is a regression, yes.

[sbose78]

Makes sense, we have an unbounded list of secrets to watch now :)

[alexmt]

I'm pretty sure we still have auto-reloading. Probably there is an edge case related to whitespace. Trying to reproduce

[alexmt]

I was able to reproduce the issue. It is a day zero bug related to the whitespace in git repo URL. Repo server creates temp directory for "normalized" URL that already HTTP encodes whitespace characters but uses original URL for the remote

https://github.com/argoproj/argo-cd/blob/3c874ae065c14102003d041d76d4a337abd72f1e/util/git/client.go#L139

As a result after switching URL to https://dev.azure.com/YourOrgName/Prep%20Project.2/_git/SomeGitRepo/ repo server keep using previously initialized git repo with https://dev.azure.com/YourOrgName/Prep Project.2/_git/SomeGitRepo/ . The workaround is to restart repo-servers.

I think fix is to use "normalized" URL for all git operations. E.g git client should be initialized with normalized URL: https://github.com/argoproj/argo-cd/blob/3c874ae065c14102003d041d76d4a337abd72f1e/util/git/client.go#L148

[alexmt]

But good thing it is not a regression and affects only repo with whitespace in the URL :)

[domeales-paloit]

Actually this issue isn't about spaces per se, rather about the URL encoding. We are having the same issue with AWS CodeCommit repo urls where we have to use this format

https://SOME_CREDENTIAL_ID@git-codecommit.REGION.amazonaws.com/v1/repos/super-repo-name

There is also a username and password.

The @ symbol may be causing the same issue as the space character in the OP.

(I am trying to understand how I can do the workaround via Terraform.)

UPDATE: Actually scratch that, this is not affecting me, please ignore this message.

[crenshaw-dev]

I encountered an issue yesterday where deleting a repo cred secret 1) did not update the list of repos in the UI and 2) did not allow an App using that repo to fall back to a credentials template (it errored on refresh with "secret not found).

I think a watch would solve this issue.

[nicolasmafraintelipost]

Use hard refresh

[maartengo]

This is no longer a problem for me. My current workaround is to just avoid spaces in repository credentials or replace them with %20, which works OK.

[blakepettersson]

I encountered an issue yesterday where deleting a repo cred secret 1) did not update the list of repos in the UI and 2) did not allow an App using that repo to fall back to a credentials template (it errored on refresh with "secret not found).

I think a watch would solve this issue.

I bumped into this issue myself in #10218, adding a watch on the secrets sounds good to me.

[blakepettersson]

Scratch that, I could not reproduce this locally, where this actually works. Will dig a bit further to see what the actual issue is.

[raghureddycloud]

@nicolasmafraintelipost

Use hard refresh

It worked for me.