[Bug][devlake-ui] Cannot Connect from devlake-ui to devlake-lake

[ohjongsung]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

Did I miss something?

error log

[error] 24#24: *31991 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.42.2.46, server: localhost, request: "GET /api/plugins HTTP/1.1", upstream: "http://10.43.48.226:8080/plugins", host: "devlake-ui.devlake.svc.cluster.local:4000"

svc

nginx conf

  location /api/ {
    resolver 10.43.0.10  valid=300s;
    resolver_timeout 3s;
    set $target "devlake-lake.devlake.svc.cluster.local:8080";
    rewrite /api/(.*) /$1  break;
    proxy_send_timeout 60s;
    proxy_read_timeout 60s;
    proxy_pass http://$target;
    proxy_http_version 1.1;
    proxy_set_header   "Connection" "";
  }

Local k3s (v1.28.6) is OK

Use nodeport

curl test

kubectl exec devlake-ui-7dc97b47cb-zg8p5 -n devlake -- curl devlake-lake.devlake.svc.cluster.local:8080/plugins % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2974 0 2974 0 0 1219k 0 --:--:-- --:--:-- --:--:-- 1452k [{"plugin":"slack","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"starrocks","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"teambition","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"zentao","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"bitbucket","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"dora","metric":{"requiredDataEntities":[{"model":"cicd_tasks","requiredFields":{"column":"type","execptedValue":"Deployment"}}],"runAfter":[],"isProjectMetric":true}},{"plugin":"github","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"icla","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"jenkins","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"refdiff","metric":{"requiredDataEntities":[],"runAfter":[],"isProjectMetric":false}},{"plugin":"tapd","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"trello","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"webhook","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"circleci","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"jira","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"sonarqube","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"azuredevops","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"dbt","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"org","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"customize","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitee","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitextractor","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"feishu","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"github_graphql","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"ae","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"bamboo","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitlab","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"opsgenie","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"pagerduty","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}}]%

But, Dev Cluster(v1.26.11) is not working

Use ingress

curl test in devlake-ui

kubectl exec devlake-ui-9d5bf46f4-2lw7d -n devlake -- curl devlake-lake.devlake.svc.cluster.local:8080/plugins % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:02:07 --:--:-- 0 curl: (28) Failed to connect to devlake-lake.devlake.svc.cluster.local port 8080 after 127268 ms: Couldn't connect to server command terminated with exit code 28

curl test in busybox

kubectl exec busybox-6b95744666-7rxg7 -n devlake -- ./curl-amd64 devlake-lake.devlake.svc.cluster.local:8080/plugins % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2974 0 2974 0 0 996k 0 --:--:-- --:--:-- -[{"plugin":"ae","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"bamboo","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"github","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"slack","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"zentao","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"bitbucket","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"icla","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"opsgenie","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"starrocks","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"trello","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"customize","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"dbt","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"dora","metric":{"requiredDataEntities":[{"model":"cicd_tasks","requiredFields":{"column":"type","execptedValue":"Deployment"}}],"runAfter":[],"isProjectMetric":true}},{"plugin":"jenkins","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"pagerduty","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"tapd","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"feishu","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"circleci","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitextractor","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"org","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"github_graphql","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"sonarqube","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"azuredevops","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitee","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitlab","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"jira","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"refdiff","metric":{"requiredDataEntities":[],"runAfter":[],"isProjectMetric":false}},{"plugin":"teambition","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"webhook","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}}]-:--:-- 1452k

So it is in pending state

What do you expect to happen

Correctly Connect from devlake-ui to devlake-lake

How to reproduce

values.yaml

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# replica count
replicaCount: 1
imageTag: v0.21.0-beta7

# image pull secrets
imagePullSecrets: []

#the common environments for all pods except grafana, grafana needs to be set in grafana section seperately
commonEnvs:
  TZ: "Asia/Seoul"

mysql:
  # if use external mysql server, please set true
  # by default using false, chart will create a single mysql instance
  useExternal: false

  # the external mysql server address
  externalServer: 127.0.0.1

  # external mysql port
  externalPort: 3306

  # the username for devlake database
  username: merico

  # the password for devlake database
  password: merico

  # the database for devlake
  database: lake

  # root password for mysql, only used when use_external=false
  rootPassword: admin

  # storage for mysql
  storage:
    # pvc or hostpath
    type: pvc
    # the storage class for pv, leave empty will using default
    class: "local-path"
    size: 50Gi
    hostPath: /devlake/mysql/data

  # image for mysql
  image:
    repository: mysql
    tag: 8
    pullPolicy: IfNotPresent

  # init containers for mysql if have
  initContainers: []

  # resources config for mysql if have
  resources: {}

  # nodeSelector config for mysql if have
  nodeSelector: {node: worker}

  # tolerations config for mysql if have
  tolerations: []

  # affinity config for mysql if have
  affinity: {}

  extraLabels: {}

  securityContext: {}

  containerSecurityContext: {}

  podAnnotations: {}

  service:
    type: "ClusterIP"
    nodePort: ""

# pgsql:
#   # if use external pgsql server, please set true
#   #   by default using false, chart will create a single pgsql instance
#   useExternal: false

#   # the external pgsql server address
#   externalServer: 127.0.0.1

#   # external pgsql port
#   externalPort: 5432
#   # the username for devlake database
#   username: merico

#   # the password for devlake database
#   password: merico

#   # the database for devlake
#   database: lake

#   # storage for pgsql
#   storage:
#     # the storage class for pv, leave empty will using default
#     class: ""
#     size: 5Gi

#   # image for pgsql
#   image:
#     repository: postgres
#     tag: 14.5
#     pullPolicy: IfNotPresent

#   # resources config for pgsql if have
#   resources: {}

#   # nodeSelector config for pgsql if have
#   nodeSelector: {}

#   # tolerations config for pgsql if have
#   tolerations: []

#   # affinity config for pgsql if have
#   affinity: {}

#   extraLabels: {}

#   securityContext: {}

#   containerSecurityContext: {}

#   annotations: {}

# dependency chart values
grafana:
  enabled: true
  #if grafana enabled is false, then external url should be provided
  external:
    url: ""
  image:
    repository: devlake.docker.scarf.sh/apache/devlake-dashboard
    tag: v0.21.0-beta7
  adminPassword: "admin"
  grafana.ini:
    server:
      root_url: "%(protocol)s://%(domain)s/grafana"
  #the secret name should be as same as .Values.option.connectionSecretName
  envFromSecrets:
    - name: "devlake-mysql-auth"
  #keep grafana timezone same as other pods, which is set by .Values.commonEnvs.TZ
  env:
    TZ: "Asia/Seoul"
  persistence:
    enabled: true
    size: 4Gi
  ingressServiceName: ""
  ingressServicePort: ""
  nodeSelector: {node: worker}

lake:
  image:
    repository: devlake.docker.scarf.sh/apache/devlake
    pullPolicy: Always
    # defaults to imageTag; if set, lake.image.tag will override imageTag
    # tag:
  # storage for config
  port: 8080
  envs:
    API_TIMEOUT: "120s"
    API_RETRY: "3"
    API_REQUESTS_PER_HOUR: "10000"
    PIPELINE_MAX_PARALLEL: "1"
    IN_SECURE_SKIP_VERIFY: "false"
    LOGGING_DIR: "/app/logs"
    # debug, info, warn, error
    LOGGING_LEVEL: "info"
  #extra envs from an existing secret
  extraEnvsFromSecret: ""
  encryptionSecret:
    # The name of secret which contains keys named ENCRYPTION_SECRET
    secretName: ""
    # if secretName is empty, secret should be set
    # you can generate the encryption secret via cmd `openssl rand -base64 2000 | tr -dc 'A-Z' | fold -w 128 | head -n 1`
    secret: "YRQRONVNYMRFXDYANUHRTUDDNKRJPKBCKDUSRYCVZDRTRCGVDXDVLWTBJZADCUJUSFKAKOOUAXERAEIXZKQUWYNMYNABMFOMZASGBUBPOIDOOWRJMPDQJLYRAOQAJLNI"
    autoCreateSecret: true

  # If hostNetwork is true, then dnsPolicy is set to ClusterFirstWithHostNet
  hostNetwork: false

  resources: {}

  strategy:
    type: Recreate

  nodeSelector: {node: worker}

  tolerations: []

  affinity: {}

  extraLabels: {}

  securityContext: {}

  containerSecurityContext: {}

  podAnnotations: {}

  livenessProbe:
    httpGet:
      path: /ping
      port: 8080
      scheme: HTTP
    failureThreshold: 5
    initialDelaySeconds: 30
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5

  readinessProbe:
    httpGet:
      path: /ping
      port: 8080
      scheme: HTTP
    failureThreshold: 3
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5

  deployment:
    extraLabels: {}

ui:
  image:
    repository: devlake.docker.scarf.sh/apache/devlake-config-ui
    pullPolicy: Always
    # defaults to imageTag; if set, lake.image.tag will override imageTag
    # tag:
  resources: {}

  strategy: {}

  nodeSelector: {node: worker}

  tolerations: []

  affinity: {}

  livenessProbe:
    httpGet:
      path: /health/
      port: 4000
      scheme: HTTP
    failureThreshold: 5
    initialDelaySeconds: 15
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5

  readinessProbe:
    httpGet:
      path: /health/
      port: 4000
      scheme: HTTP
    failureThreshold: 3
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5

  basicAuth:
    enabled: false
    user: admin
    password: admin
    autoCreateSecret: true
    secretName: ""

  extraLabels: {}

  podAnnotations: {}

  ## SecurityContext holds pod-level security attributes and common container settings.
  ## This defaults to non root user with uid 101 and gid 1000. *v1.PodSecurityContext  false
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  securityContext:
    {}
    # fsGroup: 101
    # runAsGroup: 1000
    # runAsNonRoot: true
  # runAsUser: 101

  ## K8s containers' Security Context
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
  containerSecurityContext:
    {}
    # allowPrivilegeEscalation: false
    # capabilities:
    #   drop:
  #       - all

  deployment:
    extraLabels: {}

# alpine image for some init containers
alpine:
  image:
    repository: alpine
    tag: 3.16
    pullPolicy: IfNotPresent

service:
  # service type: NodePort/ClusterIP
  type: "ClusterIP"
  # node port for devlake-ui if NodePort is enabled
  uiPort: 32001

ingress:
  enabled: true
  enableHttps: false
  # Set to false if you want to use a different ingress controller
  useDefaultNginx: true
  # ingress class name, example: alb for AWS load balancer controller
  className:
  # domain name for hosting devlake, must be set if ingress is enabled
  hostname: dev-devlake.sample.com
  # annotations required for your ingress controller; see the examples below
  # for nginx, use the first two lines of annotations
  # for alb (w/ external-dns), use the last 5 (6) lines of annotations
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  #
  # alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
  # alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:xxx:certificate/xxx-xxx-xxx
  # alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
  # alb.ingress.kubernetes.io/scheme: internet-facing
  # alb.ingress.kubernetes.io/target-type: ip
  # external-dns.alpha.kubernetes.io/hostname: www.example.com

  # url prefix, not works right now, keep "/"
  prefix: /
  # if using https provides the certificates secret name
  tlsSecretName: ""
  # ingress http port
  httpPort: 80
  # ingress https port
  httpsPort: 443

  extraPaths: []
#  extraPaths:
#    - path: /*
#      pathType: ImplementationSpecific
#      backend:
#        service:
#          name: ssl-redirect
#          port:
#            name: use-annotation

option:
  # database type, supported: [mysql]
  database: mysql
  # the existing k8s secret name of db connection auth. The secret name should be as same as .Values.grafana.envFromSecret
  connectionSecretName: "devlake-mysql-auth"
  autoCreateSecret: true

Anything else

No response

Version

devlake-0.21.0-beta7

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[ohjongsung]

It was determined that the node network settings were the problem.