dashboard `/api/v1/rollouts/default/info/watch` timeout

Sn0rt commented 4 months ago

Checklist:

[x] I've included steps to reproduce the bug.
[x] I've included the version of argo rollouts.

Describe the bug

/api/v1/rollouts/default/info/watch Interface timeout

To Reproduce

install argo-rollouts follow the guide

➜  argo-rollouts git:(master) kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
customresourcedefinition.apiextensions.k8s.io/analysisruns.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/analysistemplates.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/clusteranalysistemplates.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/experiments.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/rollouts.argoproj.io created
serviceaccount/argo-rollouts created
clusterrole.rbac.authorization.k8s.io/argo-rollouts created
clusterrole.rbac.authorization.k8s.io/argo-rollouts-aggregate-to-admin created
clusterrole.rbac.authorization.k8s.io/argo-rollouts-aggregate-to-edit created
clusterrole.rbac.authorization.k8s.io/argo-rollouts-aggregate-to-view created
clusterrolebinding.rbac.authorization.k8s.io/argo-rollouts created
configmap/argo-rollouts-config created
secret/argo-rollouts-notification-secret created
service/argo-rollouts-metrics created
deployment.apps/argo-rollouts created

start dahsboard

➜  argo-rollouts git:(master) kubectl argo rollouts dashboard
INFO[0000] Argo Rollouts Dashboard is now available at http://localhost:3100/rollouts

copy the request from the borswer

➜  argo-cd git:(master) curl 'http://localhost:3100/api/v1/rollouts/default/info/watch' \
                              -H 'Accept: text/event-stream' \
                              -H 'Accept-Language: en-US,en;q=0.9,zh;q=0.8' \
                              -H 'Cache-Control: no-cache' \
                              -H 'Connection: keep-alive' \
                              -H 'Cookie: _ga=GA1.1.1839896636.1717124486; _ga_0XNCHBGG81=GS1.1.1717124485.1.1.1717124879.0.0.0; argocd.token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcmdvY2QiLCJzdWIiOiJhZG1pbjpsb2dpbiIsImV4cCI6MTcxODY5NTAxNywibmJmIjoxNzE4NjA4NjE3LCJpYXQiOjE3MTg2MDg2MTcsImp0aSI6IjQ3ZGM1NDkzLWU3YTktNDBjMy1hZmQ1LWRjMjNlMDFiNDE3OCJ9.poucdwt1MN4q_hQKaHNcvyZcVojYsBmEe8YJpGyJ_1w' \
                              -H 'DNT: 1' \
                              -H 'Pragma: no-cache' \
                              -H 'Referer: http://localhost:3100/rollouts/default' \
                              -H 'Sec-Fetch-Dest: empty' \
                              -H 'Sec-Fetch-Mode: cors' \
                              -H 'Sec-Fetch-Site: same-origin' \
                              -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36' \
                              -H 'sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"' \
                              -H 'sec-ch-ua-mobile: ?0' \
                              -H 'sec-ch-ua-platform: "macOS"' -v
* Host localhost:3100 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:3100...
* Connected to localhost (::1) port 3100
> GET /api/v1/rollouts/default/info/watch HTTP/1.1
> Host: localhost:3100
> Accept: text/event-stream
> Accept-Language: en-US,en;q=0.9,zh;q=0.8
> Cache-Control: no-cache
> Connection: keep-alive
> Cookie: _ga=GA1.1.1839896636.1717124486; _ga_0XNCHBGG81=GS1.1.1717124485.1.1.1717124879.0.0.0; argocd.token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcmdvY2QiLCJzdWIiOiJhZG1pbjpsb2dpbiIsImV4cCI6MTcxODY5NTAxNywibmJmIjoxNzE4NjA4NjE3LCJpYXQiOjE3MTg2MDg2MTcsImp0aSI6IjQ3ZGM1NDkzLWU3YTktNDBjMy1hZmQ1LWRjMjNlMDFiNDE3OCJ9.poucdwt1MN4q_hQKaHNcvyZcVojYsBmEe8YJpGyJ_1w
> DNT: 1
> Pragma: no-cache
> Referer: http://localhost:3100/rollouts/default
> Sec-Fetch-Dest: empty
> Sec-Fetch-Mode: cors
> Sec-Fetch-Site: same-origin
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
> sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"
> sec-ch-ua-mobile: ?0
> sec-ch-ua-platform: "macOS"
>

* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

Expected behavior

no timeout request

Screenshots

Version

➜  argo-rollouts git:(master) kubectl argo rollouts version

kubectl-argo-rollouts: v1.6.0+7eae71e
  BuildDate: 2023-09-06T18:41:21Z
  GitCommit: 7eae71ed89f1a3769864435bddebe3ca05384df3
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: darwin/amd64

➜  argo-rollouts git:(master) kubectl -n argo-rollouts get pod argo-rollouts-7867c998f9-vk995 -o yaml | grep image
  - image: quay.io/argoproj/argo-rollouts:v1.7.0
    imagePullPolicy: Always
    image: quay.io/argoproj/argo-rollouts:v1.7.0
    imageID: quay.io/argoproj/argo-rollouts@sha256:8ce9e8227043f0aa1d559e356e2494d21a501169454481f16d1c485be2792c87

Logs

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

EnriqueTejeda commented 4 months ago

Hi @Sn0rt, I tried to reproduce the issue, but I was unable to do so. Could you please provide more details? It might also help if you activate the debug logging level.

Sn0rt commented 4 months ago

Hi @Sn0rt, I tried to reproduce the issue, but I was unable to do so. Could you please provide more details? It might also help if you activate the debug logging level.

thx you EnriqueTejeda , what detail need append under this issues ?

I have provider the version info and install step.

caspar-ds commented 3 months ago

Also seeing the same issue. Occasionally see this in the logs too:

time="2024-08-05T20:49:48Z" level=info msg="[core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: \"0.0.0.0:3100\", ServerName: \"0.0.0.0:3100\", }. Err: connection error: desc = \"transport: Error while dialing: dial tcp 0.0.0.0:3100: connect: connection refused\""

mzupan commented 1 month ago

I'm seeing this also.. any chance you are using cilium and/or bottlerocket in kubernetes?

mzupan commented 1 month ago

So i was able to re-create this..

created a stock EKS cluster
installed rollouts and was able to view a dashboard when i deployed an example rollout app
removed aws-node and kube-proxy addons
installed cilium with the helm chart overrides below
deleted rollouts/crds and deleted example app
installed rollouts and example app
dashboard not working

eni:
  enabled: true

ipam:
  mode: eni

egressMasqueradeInterfaces: eth0

kubeProxyReplacement: true
routingMode: native

tunnel: disabled

k8sServiceHost: D5E44.....
k8sServicePort: 443

encryption:
  enabled: true
  type: wireguard
  nodeEncryption: true

argoproj / argo-rollouts

dashboard `/api/v1/rollouts/default/info/watch` timeout #3660