search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.47k stars 5.31k forks source link

Setting proxy in argocd server fails to start #2954

Open ksaito1125 opened 4 years ago

ksaito1125 commented 4 years ago

Checklist:

Describe the bug

I run Argo CD behind a proxy server.

I set proxy to environment variable of argocd-repo-server and installed ArgoCD. It worked fine, but argocd-server could not communicate with github.

After configuring proxy in the same way for argocd-server and installing ArgoCD, argocd-server failed to start.

To Reproduce

Set proxy in argocd-repo-server and start.

$ helm repo add argo https://argoproj.github.io/argo-helm
$ kubectl create namespace argocd
$ kubectl ns argocd
$ diff -u <(helm inspect values argo/argo-cd) values-proxy.yaml
...
@@ -494,7 +494,13 @@

   ## Environment variables to pass to argocd-repo-server
   ##
-  env: []
+  env:
+  - name: http_proxy
+    value: http://proxy:8080/
+  - name: https_proxy
+    value: http://proxy:8080/
+  - name: no_proxy
+    value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,10.0.0.0/8
...
$ helm install --values=values-proxy.yaml argocd argo/argo-cd --namespace=argocd
$ kubectl get pod
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-application-controller-8466f9bd5b-mkwhd   1/1     Running   0          57s
argocd-dex-server-865d7bc898-nmldc               1/1     Running   0          57s
argocd-redis-59dd8cb6f6-2nvqq                    1/1     Running   0          57s
argocd-repo-server-5784d4dcf5-srmtz              1/1     Running   0          57s
argocd-server-7f4dc4cd64-kclcp                   1/1     Running   0          57s

Once, uninstall argocd.

$ helm uninstall argocd --namespace=argocd

Set proxy in argocd-server and install again.

$ git diff
diff --git a/values-proxy.yaml b/values-proxy.yaml
index 306ae14..2a54a15 100644
--- a/values-proxy.yaml
+++ b/values-proxy.yaml
@@ -260,7 +260,13 @@ server:

   ## Environment variables to pass to argocd-server
   ##
-  env: []
+  env:
+  - name: http_proxy
+    value: http://proxy:8080/
+  - name: https_proxy
+    value: http://proxy:8080/
+  - name: no_proxy
+    value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,10.0.0.0/8

   ## Argo server log level
   logLevel: info
$ helm install --values=values-proxy.yaml argocd argo/argo-cd --namespace=argocd
$ kubectl get pod
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-application-controller-8466f9bd5b-zt6s5   1/1     Running   0          62s
argocd-dex-server-865d7bc898-z8c8n               0/1     Error     3          62s
argocd-redis-59dd8cb6f6-qgg5x                    1/1     Running   0          62s
argocd-repo-server-5784d4dcf5-f5d5x              1/1     Running   0          62s

argocd-server-7f5cf69ff6-skqnd                   0/1     Running   1          62s

Expected behavior

I expected that github.com repository could be used by setting proxy in argocd-server Looking at the log, it seems that argocd-server could not get the configmap required for startup. I think no_proxy needs additional settings, but I didn't know what to set.

Version

$ argocd version
argocd: v1.3.6+89be1c9
  BuildDate: 2019-12-10T22:46:45Z
  GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.3.6+89be1c9
  BuildDate: 2019-12-10T22:47:48Z
  GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: Version: {Version:kustomize/v3.2.1 GitCommit:d89b448c745937f0cf1936162f26a5aac688f840 BuildDate:2019-09-27T00:10:52Z GoOs:linux GoArch:amd64}
  Helm Version: v2.15.2
  Kubectl Version: v1.14.0

Logs

Log at normal startup with proxy setting only for argocd-repo-server.

$ stern argocd-server
+ argocd-server-7f4dc4cd64-kclcp › server
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Starting configmap/secret informers"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Configmap/secret informer synced"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Initialized server signature"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Initialized admin password"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Initialized TLS certificate"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="configmap informer cancelled"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Starting configmap/secret informers"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="Configmap/secret informer synced"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="secrets informer cancelled"
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [Model:]
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 r.r: sub, res, act, obj
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 p.p: sub, res, act, obj, eft
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 m.m: g(r_sub, p_sub) && keyMatch(r_res, p_res) && keyMatch(r_act, p_act) && keyMatch(r_obj, p_obj)
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 g.g: _, _
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [Policy:]
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [p :  sub, res, act, obj, eft :  []]
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [g :  _, _ :  []]
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [Role links for: g]
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="argocd v1.3.6+89be1c9 serving on port 8080 (url: https://argocd.example.com, tls: true, namespace: argocd, sso: false)"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="0xc00081faa0 subscribed to settings updates"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="Starting rbac config informer"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"

Failure log.

$ stern argocd-server
+ argocd-server-7f5cf69ff6-skqnd › server
argocd-server-7f5cf69ff6-skqnd server time="2020-01-09T05:00:50Z" level=info msg="Starting configmap/secret informers"
argocd-server-7f5cf69ff6-skqnd server I0109 05:01:09.661074       1 trace.go:82] Trace[692801166]: "Reflector github.com/argoproj/argo-cd/util/settings/settings.go:544 ListAndWatch" (started: 2020-01-09 05:00:50.169021218 +0000 UTC m=+0.147664721) (total time: 19.491985533s):
argocd-server-7f5cf69ff6-skqnd server Trace[692801166]: [19.491985533s] [19.491985533s] END
argocd-server-7f5cf69ff6-skqnd server E0109 05:01:09.661099       1 reflector.go:126] github.com/argoproj/argo-cd/util/settings/settings.go:544: Failed to list *v1.Secret: an error on the server ("") has prevented the request from succeeding (get secrets)
argocd-server-7f5cf69ff6-skqnd server I0109 05:01:10.701287       1 trace.go:82] Trace[1087694162]: "Reflector github.com/argoproj/argo-cd/util/settings/settings.go:540 ListAndWatch" (started: 2020-01-09 05:00:50.168314313 +0000 UTC m=+0.146957813) (total time: 20.532916909s):
argocd-server-7f5cf69ff6-skqnd server Trace[1087694162]: [20.532916909s] [20.532916909s] END
argocd-server-7f5cf69ff6-skqnd server E0109 05:01:10.701308       1 reflector.go:126] github.com/argoproj/argo-cd/util/settings/settings.go:540: Failed to list *v1.ConfigMap: an error on the server ("") has prevented the request from succeeding (get configmaps)
Kyrklund commented 4 years ago

Hi I seem to have stumbled into the same rabbit hole with a proxy. I can't connect to Github repos and have added the proxy information in env for the "argocd-repo-server" but no difference. I then added the same config to the "argocd-server" and in the logs i got the following errors:

reflector.go:123] github.com/argoproj/argo-cd/util/settings/settings.go:600: Failed to list *v1.Secret: Get "https://172.17.0.1:443/api/v1/namespaces/argocd/secrets?limit=500&resourceVersion=0": Forbidden

So i then went ahead and added the 172.17.0.1 address. The "argocd-server" pod now starts, but i get denied by our proxy instead

Failed to query provider "https://argocd.example.com/api/dex": 403 Forbidden: And even if I add this host to no_proxy list I still get denied by the proxy.

I will continue to test and see if i can figure out a working config.

toVersus commented 4 years ago

@Kyrklund Could you check if argocd-dex-server is addded in NO_PROXY env value. I think the 403 status code would be returned when argocd-server is trying to connect to dex server (http://argocd-dex-server:5556) for SSO. The argocd-dex-server value is missing in https://github.com/argoproj/argo-cd/issues/2954#issue-547270068 's suggested proxy setting.

  env:
  - name: http_proxy
    value: http://proxy:8080/
  - name: https_proxy
    value: http://proxy:8080/
  - name: NO_PROXY
     value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,10.0.0.0/8
Kyrklund commented 4 years ago

@toVersus That could absolutely be something missing in the config. At the moment we settled for just letting our developers use our in-house private repositories. But if we change our mind and go back trying to implement external access i'll be sure to check if the Dex server is present.

Thank you so much for the tip

matthewhembree commented 3 years ago

Additional information for people that stumble upon this issue thread. I use GitHub OIDC via Dex. I had to also add the proxy envars to the argocd-dex-server deployment.

Additionally, I have an HA ArgoCD deployment, so I needed to add argocd-redis-ha-haproxy to the NO_PROXY list.

Below is what I have added for the envars:

env:
    - name: HTTP_PROXY
      value: http://proxy.example.com:3128/
    - name: HTTPS_PROXY
      value: http://proxy.example.com:3128/
    - name: NO_PROXY
      value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-redis-ha-haproxy,argocd-dex-server,localhost,10.0.0.0/8

localhost,10.0.0.0/8 <- You'll need to add your Kubernetes service address range in the NO_PROXY list. Otherwise, you'll get certificate errors with argocd-server when it tries to pull objects from the Kubernetes API server. I don't understand it exactly, but it looks like the pod will resolve the address of kubernetes.default.svc.cluster.local and then connect directly to that IP address.

If you're using kustomize to deploy ArgoCD, use the following JSON6902 patches: For argocd-server (this envar should already be present: ARGOCD_API_SERVER_REPLICAS):

[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/env/-",
    "value": {
      "name": "HTTP_PROXY",
      "value": "http://proxy.example.com:3128/"
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/env/-",
    "value": {
      "name": "HTTPS_PROXY",
      "value": "http://proxy.example.com:3128/"
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/env/-",
    "value": {
      "name": "NO_PROXY",
      "value": "argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-redis-ha-haproxy,argocd-dex-server,localhost,10.0.0.0/8"
    }
  }
]

For argocd-repo-server and argocd-dex-server:

[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/env",
    "value": [
      {
        "name": "HTTP_PROXY",
        "value": "http://proxy.example.com:3128/"
      },
      {
        "name": "HTTPS_PROXY",
        "value": "http://proxy.example.com:3128/"
      },
      {
        "name": "NO_PROXY",
        "value": "argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-redis-ha-haproxy,argocd-dex-server,localhost,10.0.0.0/8"
      }
    ]
  }
]
migka2012 commented 3 years ago

@matthewhembree I am brand new to argoCD. I created values.yaml file as your comment above

env:
  - name: HTTP_PROXY
    value: http://ip_address:3128
  - name: HTTPS_PROXY
    value: http://ip_address:3128
  - name: NO_PROXY
    value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,10.0.0.0/8

and then installed argoCD helm install --values=values.yaml argocd argo/argo-cd --namespace=argocd

All servers started, however, I could not connect github on web UI. Please help me to configure proxy settings for argoCD.

migka2012 commented 3 years ago

Never mind. I now can set proxy settings for argoCD by using values.yaml below

server:
  env:
    - name: HTTP_PROXY
      value: http://xxx:3128
    - name: HTTPS_PROXY
      value: http://xxx:3128
    - name: NO_PROXY
      value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,10.0.0.0/8 

repoServer:
  env:
    - name: HTTP_PROXY
      value: http://xxx:3128
    - name: HTTPS_PROXY
      value: http://xxx:3128
    - name: NO_PROXY
      value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,10.0.0.0/8

dex:
  env:
    - name: HTTP_PROXY
      value: http://xxx:3128
    - name: HTTPS_PROXY
      value: http://xxx:3128
    - name: NO_PROXY
      value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,10.0.0.0/8
magf commented 1 year ago

maybe that's better?

extraObjects:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      labels:
        app.kubernetes.io/instance: argocd
        app.kubernetes.io/name: my-proxy
        app.kubernetes.io/part-of: argocd
      name: my-proxy
      namespace: argocd
    data:
      HTTP_PROXY: http://my-proxy:3128
      HTTPS_PROXY: http://my-proxy:3128
      NO_PROXY: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,my-network,10.0.0.0/8

server:
  env:
  envFrom:
  - configMapRef:
      name: my-proxy

repoServer:
  env:
  envFrom:
  - configMapRef:
      name: my-proxy

dex:
  env:
  envFrom:
  - configMapRef:
      name: my-proxy
c-p-b commented 8 months ago

In my case I wanted to use Tailscale as a sidecar via ExtraContainers to enable private access to remote clusters in different clouds. It took a bit of finagling to get right, but here is my configuration:

server:
  env:  
    - name: ALL_PROXY                                                                                                   
      value: "socks5://localhost:1055"
    - name: HTTP_PROXY                                                                                                   
      value: "http://localhost:1055"
    - name: HTTPS_PROXY                                                                                                   
      value: "http://localhost:1055"
    - name: NO_PROXY
      value: |
        argo-cd-argocd-repo-server,
        argo-cd-argocd-application-controller,
        argo-cd-argocd-applicationset-controller,
        argo-cd-argocd-metrics,argo-cd-argocd-server,
        argo-cd-argocd-server-metrics,
        argo-cd-argocd-redis,
        argo-cd-argocd-dex-server,
        localhost,
        127.0.0.1,
        kubernetes.default.svc,
        .svc.cluster.local,
        172.29.0.0/16,
  extraContainers:                                                                                                     
    - name: tailscale                                                                                                   
      image: tailscale/tailscale                                                                                        
      command: ["/bin/sh", "-c"]                                                                                        
      args:                                                                                                             
        - |                                                                                                             
          tailscaled --tun=userspace-networking --socks5-server=0.0.0.0:1055 --outbound-http-proxy-listen=0.0.0.0:1055 &
          sleep 5                                                                                                             
          until tailscale up --authkey $TS_AUTHKEY --accept-routes; do                                                                        
            echo "Tailscale up failed, retrying in 5 seconds"                                                                 
            sleep 5                                                                                                           
          done                                                                                                                
          echo "Tailscale up succeeded"                                                                                       
          tail -f /dev/null                                                                                                       
      env:  
        - name: ALL_PROXY                                                                                                   
          value: "socks5://localhost:1055"                                                                                                            
        - name: TS_AUTHKEY                                                                                              
          valueFrom:                                                                                                    
            secretKeyRef:                                                                                               
              name: tailscale-authkey                                                                                   
              key: TS_AUTHKEY

Where 172.29.0.0/16 is the cidr that corresponds to my internal vpc/cluster network.

Note that argo does not seem to support ALL_PROXY, only HTTP_PROXY and HTTPS_PROXY env variables seemed to be respected.

In particular, as noted elsewhere, it is important to make the local network and services excluded from proxy configuration. Also verify that the services referenced in NO_PROXY match up with services listed at kubectl get svc -n argocd

mnasty commented 3 months ago

So stumbled on this issue trying to add corporate proxy to a deployKF managed argocd deployment with a special plugin that is heavily customized and has no option to install with helm.

I discovered that I can add these variables quite simply using kubectl post-deployment in a way that can be easily understood and reproduced in an ansible playbook. Might also be useful for debugging:

kubectl -n argocd set env deployment/argocd-server HTTP_PROXY=0.0.0.0:3128 HTTPS_PROXY=0.0.0.0:3128 NO_PROXY=argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,my-network,10.0.0.0/8

kubectl -n argocd set env deployment/argocd-repo-server HTTP_PROXY=0.0.0.0:3128 HTTPS_PROXY=0.0.0.0:3128 NO_PROXY=argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,my-network,10.0.0.0/8

kubectl -n argocd set env deployment/argocd-dex-server HTTP_PROXY=0.0.0.0:3128 HTTPS_PROXY=0.0.0.0:3128 NO_PROXY=argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,my-network,10.0.0.0/8

Rather than reverse engineering their code and rebuilding a customized custom repo from scratch or installing it separately and integrating their special plugin manually (huge headache), this approach seems to be just as good as helm values for my purposes.

You can exec into the pods and confirm the variables have been applied successfully. Hope this is helpful to someone.

endreszabo commented 2 months ago

You can exec into the pods and confirm the variables have been applied successfully. Hope this is helpful to someone.

While this is true and when executed manually helm itself now works in an argocd-server pod, it will not work when argocd server binary itself launches helm as it does not pass the proxy variables.