search

apache / incubator-devlake-helm-chart

Apache DevLake Helm Chart
https://devlake.apache.org/
Apache License 2.0
32 stars 45 forks source link

Cannot Connect from devlake-ui to devlake-lake #261

Closed ohjongsung closed 5 months ago

ohjongsung commented 5 months ago

error log

[error] 24#24: *31991 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.42.2.46, server: localhost, request: "GET /api/plugins HTTP/1.1", upstream: "http://10.43.48.226:8080/plugins", host: "devlake-ui.devlake.svc.cluster.local:4000"

Local k3s (v1.28.6) is OK

Use nodeport

image

curl test

kubectl exec devlake-ui-7dc97b47cb-zg8p5 -n devlake -- curl devlake-lake.devlake.svc.cluster.local:8080/plugins % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2974 0 2974 0 0 1219k 0 --:--:-- --:--:-- --:--:-- 1452k [{"plugin":"slack","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"starrocks","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"teambition","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"zentao","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"bitbucket","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"dora","metric":{"requiredDataEntities":[{"model":"cicd_tasks","requiredFields":{"column":"type","execptedValue":"Deployment"}}],"runAfter":[],"isProjectMetric":true}},{"plugin":"github","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"icla","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"jenkins","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"refdiff","metric":{"requiredDataEntities":[],"runAfter":[],"isProjectMetric":false}},{"plugin":"tapd","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"trello","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"webhook","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"circleci","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"jira","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"sonarqube","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"azuredevops","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"dbt","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"org","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"customize","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitee","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitextractor","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"feishu","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"github_graphql","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"ae","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"bamboo","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitlab","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"opsgenie","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"pagerduty","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}}]%

But, Dev Cluster(v1.26.11) is not working

Use ingress

image

curl test in devlake-ui

kubectl exec devlake-ui-9d5bf46f4-2lw7d -n devlake -- curl devlake-lake.devlake.svc.cluster.local:8080/plugins % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:02:07 --:--:-- 0 curl: (28) Failed to connect to devlake-lake.devlake.svc.cluster.local port 8080 after 127268 ms: Couldn't connect to server command terminated with exit code 28

curl test in busybox

kubectl exec busybox-6b95744666-7rxg7 -n devlake -- ./curl-amd64 devlake-lake.devlake.svc.cluster.local:8080/plugins % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2974 0 2974 0 0 996k 0 --:--:-- --:--:-- -[{"plugin":"ae","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"bamboo","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"github","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"slack","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"zentao","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"bitbucket","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"icla","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"opsgenie","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"starrocks","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"trello","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"customize","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"dbt","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"dora","metric":{"requiredDataEntities":[{"model":"cicd_tasks","requiredFields":{"column":"type","execptedValue":"Deployment"}}],"runAfter":[],"isProjectMetric":true}},{"plugin":"jenkins","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"pagerduty","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"tapd","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"feishu","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"circleci","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitextractor","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"org","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"github_graphql","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"sonarqube","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"azuredevops","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitee","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"gitlab","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"jira","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"refdiff","metric":{"requiredDataEntities":[],"runAfter":[],"isProjectMetric":false}},{"plugin":"teambition","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}},{"plugin":"webhook","metric":{"requiredDataEntities":null,"runAfter":null,"isProjectMetric":false}}]-:--:-- 1452k

So it is in pending state

image

values.yaml

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# replica count
replicaCount: 1
imageTag: v0.21.0-beta7

# image pull secrets
imagePullSecrets: []

#the common environments for all pods except grafana, grafana needs to be set in grafana section seperately
commonEnvs:
  TZ: "Asia/Seoul"

mysql:
  # if use external mysql server, please set true
  # by default using false, chart will create a single mysql instance
  useExternal: false

  # the external mysql server address
  externalServer: 127.0.0.1

  # external mysql port
  externalPort: 3306

  # the username for devlake database
  username: merico

  # the password for devlake database
  password: merico

  # the database for devlake
  database: lake

  # root password for mysql, only used when use_external=false
  rootPassword: admin

  # storage for mysql
  storage:
    # pvc or hostpath
    type: pvc
    # the storage class for pv, leave empty will using default
    class: "local-path"
    size: 50Gi
    hostPath: /devlake/mysql/data

  # image for mysql
  image:
    repository: mysql
    tag: 8
    pullPolicy: IfNotPresent

  # init containers for mysql if have
  initContainers: []

  # resources config for mysql if have
  resources: {}

  # nodeSelector config for mysql if have
  nodeSelector: {node: worker}

  # tolerations config for mysql if have
  tolerations: []

  # affinity config for mysql if have
  affinity: {}

  extraLabels: {}

  securityContext: {}

  containerSecurityContext: {}

  podAnnotations: {}

  service:
    type: "ClusterIP"
    nodePort: ""

# pgsql:
#   # if use external pgsql server, please set true
#   #   by default using false, chart will create a single pgsql instance
#   useExternal: false

#   # the external pgsql server address
#   externalServer: 127.0.0.1

#   # external pgsql port
#   externalPort: 5432
#   # the username for devlake database
#   username: merico

#   # the password for devlake database
#   password: merico

#   # the database for devlake
#   database: lake

#   # storage for pgsql
#   storage:
#     # the storage class for pv, leave empty will using default
#     class: ""
#     size: 5Gi

#   # image for pgsql
#   image:
#     repository: postgres
#     tag: 14.5
#     pullPolicy: IfNotPresent

#   # resources config for pgsql if have
#   resources: {}

#   # nodeSelector config for pgsql if have
#   nodeSelector: {}

#   # tolerations config for pgsql if have
#   tolerations: []

#   # affinity config for pgsql if have
#   affinity: {}

#   extraLabels: {}

#   securityContext: {}

#   containerSecurityContext: {}

#   annotations: {}

# dependency chart values
grafana:
  enabled: true
  #if grafana enabled is false, then external url should be provided
  external:
    url: ""
  image:
    repository: devlake.docker.scarf.sh/apache/devlake-dashboard
    tag: v0.21.0-beta7
  adminPassword: "admin"
  grafana.ini:
    server:
      root_url: "%(protocol)s://%(domain)s/grafana"
  #the secret name should be as same as .Values.option.connectionSecretName
  envFromSecrets:
    - name: "devlake-mysql-auth"
  #keep grafana timezone same as other pods, which is set by .Values.commonEnvs.TZ
  env:
    TZ: "Asia/Seoul"
  persistence:
    enabled: true
    size: 4Gi
  ingressServiceName: ""
  ingressServicePort: ""
  nodeSelector: {node: worker}

lake:
  image:
    repository: devlake.docker.scarf.sh/apache/devlake
    pullPolicy: Always
    # defaults to imageTag; if set, lake.image.tag will override imageTag
    # tag:
  # storage for config
  port: 8080
  envs:
    API_TIMEOUT: "120s"
    API_RETRY: "3"
    API_REQUESTS_PER_HOUR: "10000"
    PIPELINE_MAX_PARALLEL: "1"
    IN_SECURE_SKIP_VERIFY: "false"
    LOGGING_DIR: "/app/logs"
    # debug, info, warn, error
    LOGGING_LEVEL: "info"
  #extra envs from an existing secret
  extraEnvsFromSecret: ""
  encryptionSecret:
    # The name of secret which contains keys named ENCRYPTION_SECRET
    secretName: ""
    # if secretName is empty, secret should be set
    # you can generate the encryption secret via cmd `openssl rand -base64 2000 | tr -dc 'A-Z' | fold -w 128 | head -n 1`
    secret: "YRQRONVNYMRFXDYANUHRTUDDNKRJPKBCKDUSRYCVZDRTRCGVDXDVLWTBJZADCUJUSFKAKOOUAXERAEIXZKQUWYNMYNABMFOMZASGBUBPOIDOOWRJMPDQJLYRAOQAJLNI"
    autoCreateSecret: true

  # If hostNetwork is true, then dnsPolicy is set to ClusterFirstWithHostNet
  hostNetwork: false

  resources: {}

  strategy:
    type: Recreate

  nodeSelector: {node: worker}

  tolerations: []

  affinity: {}

  extraLabels: {}

  securityContext: {}

  containerSecurityContext: {}

  podAnnotations: {}

  livenessProbe:
    httpGet:
      path: /ping
      port: 8080
      scheme: HTTP
    failureThreshold: 5
    initialDelaySeconds: 30
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5

  readinessProbe:
    httpGet:
      path: /ping
      port: 8080
      scheme: HTTP
    failureThreshold: 3
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5

  deployment:
    extraLabels: {}

ui:
  image:
    repository: devlake.docker.scarf.sh/apache/devlake-config-ui
    pullPolicy: Always
    # defaults to imageTag; if set, lake.image.tag will override imageTag
    # tag:
  resources: {}

  strategy: {}

  nodeSelector: {node: worker}

  tolerations: []

  affinity: {}

  livenessProbe:
    httpGet:
      path: /health/
      port: 4000
      scheme: HTTP
    failureThreshold: 5
    initialDelaySeconds: 15
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5

  readinessProbe:
    httpGet:
      path: /health/
      port: 4000
      scheme: HTTP
    failureThreshold: 3
    initialDelaySeconds: 5
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 5

  basicAuth:
    enabled: false
    user: admin
    password: admin
    autoCreateSecret: true
    secretName: ""

  extraLabels: {}

  podAnnotations: {}

  ## SecurityContext holds pod-level security attributes and common container settings.
  ## This defaults to non root user with uid 101 and gid 1000. *v1.PodSecurityContext  false
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  securityContext:
    {}
    # fsGroup: 101
    # runAsGroup: 1000
    # runAsNonRoot: true
  # runAsUser: 101

  ## K8s containers' Security Context
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
  containerSecurityContext:
    {}
    # allowPrivilegeEscalation: false
    # capabilities:
    #   drop:
  #       - all

  deployment:
    extraLabels: {}

# alpine image for some init containers
alpine:
  image:
    repository: alpine
    tag: 3.16
    pullPolicy: IfNotPresent

service:
  # service type: NodePort/ClusterIP
  type: "ClusterIP"
  # node port for devlake-ui if NodePort is enabled
  uiPort: 32001

ingress:
  enabled: true
  enableHttps: false
  # Set to false if you want to use a different ingress controller
  useDefaultNginx: true
  # ingress class name, example: alb for AWS load balancer controller
  className: nginx
  # domain name for hosting devlake, must be set if ingress is enabled
  hostname: dev-devlake.sample.com
  # annotations required for your ingress controller; see the examples below
  # for nginx, use the first two lines of annotations
  # for alb (w/ external-dns), use the last 5 (6) lines of annotations
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  #
  # alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
  # alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:xxx:certificate/xxx-xxx-xxx
  # alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
  # alb.ingress.kubernetes.io/scheme: internet-facing
  # alb.ingress.kubernetes.io/target-type: ip
  # external-dns.alpha.kubernetes.io/hostname: www.example.com

  # url prefix, not works right now, keep "/"
  prefix: /
  # if using https provides the certificates secret name
  tlsSecretName: ""
  # ingress http port
  httpPort: 80
  # ingress https port
  httpsPort: 443

  extraPaths: []
#  extraPaths:
#    - path: /*
#      pathType: ImplementationSpecific
#      backend:
#        service:
#          name: ssl-redirect
#          port:
#            name: use-annotation

option:
  # database type, supported: [mysql]
  database: mysql
  # the existing k8s secret name of db connection auth. The secret name should be as same as .Values.grafana.envFromSecret
  connectionSecretName: "devlake-mysql-auth"
  autoCreateSecret: true
ZhangNing10 commented 5 months ago

hi @ohjongsung , Dev Cluster(v1.26.11) works for the busybox pod but doesn't work for the devlake-ui pod? I think it's an network issue of the cluster, kindly check whether it's ok to connect the lake backend pod ip from ui

ohjongsung commented 5 months ago

hi @ohjongsung , Dev Cluster(v1.26.11) works for the busybox pod but doesn't work for the devlake-ui pod? I think it's an network issue of the cluster, kindly check whether it's ok to connect the lake backend pod ip from ui

hi @ZhangNing10 , Yes How to check to connect the lake backend pod ip from ui? The error log was created in the ui pod. I tested it with curl in the ui pod shell, and nslookup also accurately pointed to the lake backend pod. If no one has the same problem as me, the problem is with the dev cluster... but it connects fine with the busybox pod, so it's frustrating. Is there anything else I should check?

ohjongsung commented 5 months ago

It was determined that the node network settings were the problem.