ml-pipeline unable to connect to mysql in kubefloq 1.6

psheorangithub commented 2 years ago

Environment

How do you deploy Kubeflow Pipelines (KFP)? Deployed pipelines as part of kubeflow 1.6
KFP version: 2.0.0-alpha.3 To find the version number, See version number shows on bottom of KFP UI left sidenav. -->
KFP SDK version:

Steps to reproduce

Deploy kubeflow manifest 1.6

Expected result

ml-pipeline pod should come up

Materials and reference

Below is ml-pipeline anc mysql deployment manifest.

ml-pipeline:

apiVersion: apps/v1 kind: Deployment metadata: labels: app: ml-pipeline app.kubernetes.io/component: ml-pipeline app.kubernetes.io/name: kubeflow-pipelines application-crd-id: kubeflow-pipelines {{- toYaml .Values.labels | nindent 4 }} name: ml-pipeline namespace: kubeflow annotations: {{- toYaml .Values.annotations | nindent 4 }} spec: selector: matchLabels: app: ml-pipeline app.kubernetes.io/component: ml-pipeline app.kubernetes.io/name: kubeflow-pipelines application-crd-id: kubeflow-pipelines template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: 'true' {{- toYaml .Values.annotations | nindent 8 }} labels: app: ml-pipeline app.kubernetes.io/component: ml-pipeline app.kubernetes.io/name: kubeflow-pipelines application-crd-id: kubeflow-pipelines {{- toYaml .Values.labels | nindent 8 }} spec: containers:

env:
- name: KUBEFLOW_USERID_HEADER value: kubeflow-userid
- name: KUBEFLOW_USERID_PREFIX value: ''
- name: AUTO_UPDATE_PIPELINE_DEFAULT_VERSION valueFrom: configMapKeyRef: key: autoUpdatePipelineDefaultVersion name: pipeline-install-config
- name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace
- name: OBJECTSTORECONFIG_SECURE value: "true"
- name: MINIO_SERVICE_SERVICE_HOST value: '{{ .Values.s3.host }}'
- name: MINIO_SERVICE_SERVICE_PORT value: '{{ .Values.s3.port | quote }}'
- name: OBJECTSTORECONFIG_BUCKETNAME valueFrom: configMapKeyRef: key: bucketName name: pipeline-install-config
- name: DBCONFIG_USER valueFrom: secretKeyRef: key: username name: mysql-secret
- name: DBCONFIG_PASSWORD valueFrom: secretKeyRef: key: password name: mysql-secret
- name: DBCONFIG_DBNAME valueFrom: configMapKeyRef: key: pipelineDb name: pipeline-install-config
- name: DBCONFIG_HOST valueFrom: configMapKeyRef: key: dbHost name: pipeline-install-config
- name: DBCONFIG_PORT valueFrom: configMapKeyRef: key: dbPort name: pipeline-install-config
- name: DBCONFIG_CONMAXLIFETIME valueFrom: configMapKeyRef: key: ConMaxLifeTime name: pipeline-install-config
- name: OBJECTSTORECONFIG_ACCESSKEY valueFrom: secretKeyRef: key: accesskey name: mlpipeline-ceph-s3-artifact
- name: OBJECTSTORECONFIG_SECRETACCESSKEY valueFrom: secretKeyRef: key: secretkey name: mlpipeline-ceph-s3-artifact envFrom:
- configMapRef: name: pipeline-api-server-config-f4t72426kt image: gcr.io/ml-pipeline/api-server:2.0.0-alpha.3 imagePullPolicy: IfNotPresent livenessProbe: exec: command:
  - wget
  - -q
  - -S
  - -O
  - '-'
  - http://localhost:8888/apis/v1beta1/healthz initialDelaySeconds: 3 periodSeconds: 5 timeoutSeconds: 2 name: ml-pipeline-api-server ports:
- containerPort: 8888 name: http
- containerPort: 8887 name: grpc readinessProbe: exec: command:
  - wget
  - -q
  - -S
  - -O
  - '-'
  - http://localhost:8888/apis/v1beta1/healthz initialDelaySeconds: 3 periodSeconds: 5 timeoutSeconds: 2 resources: requests: cpu: 250m memory: 500Mi startupProbe: exec: command:
    - wget
    - -q
    - -S
    - -O
    - '-'
    - http://localhost:8888/apis/v1beta1/healthz failureThreshold: 12 periodSeconds: 5 timeoutSeconds: 2 volumeMounts:
      - name: ca-bundles mountPath: /etc/ssl/certs/ca-certificates.crt subPath: ca-certificates.crt serviceAccountName: ml-pipeline volumes:
- name: ca-bundles configMap: name: ca-bundles defaultMode: 420

mysql:

apiVersion: apps/v1 kind: Deployment metadata: labels: app: mysql application-crd-id: kubeflow-pipelines name: mysql namespace: kubeflow spec: selector: matchLabels: app: mysql application-crd-id: kubeflow-pipelines strategy: type: Recreate template: metadata: labels: app: mysql application-crd-id: kubeflow-pipelines spec: containers:

args:
- --ignore-db-dir=lost+found
- --datadir
- /var/lib/mysql env:
  - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: key: password name: mysql-secret image: gcr.io/ml-pipeline/mysql:5.7-debian name: mysql ports:
- containerPort: 3306 name: mysql resources: requests: cpu: 100m memory: 800Mi volumeMounts:
- mountPath: /var/lib/mysql name: mysql-persistent-storage serviceAccountName: mysql volumes:
name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pv-claim

Labels

No errors in ml-pipeline logs

ml-pipeline-5fc565dfc4-2zljp 1/2 Running 55 3h27m

bash-3.2$ k logs ml-pipeline-5fc565dfc4-2zljp -n kubeflow I1005 16:29:47.455100 8 client_manager.go:160] Initializing client manager I1005 16:29:47.455255 8 config.go:57] Config DBConfig.ExtraParams not specified, skipping bash-3.2$ k logs ml-pipeline-5fc565dfc4-2zljp -n kubeflow -p I1005 16:28:17.581634 7 client_manager.go:160] Initializing client manager I1005 16:28:17.581844 7 config.go:57] Config DBConfig.ExtraParams not specified, skipping bash-3.2$

mysql shows below errors:

2022-10-05T12:53:21.615125Z 0 [Note] - '::' resolves to '::'; 2022-10-05T12:53:21.615171Z 0 [Note] Server socket created on IP: '::'. 2022-10-05T12:53:21.617003Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory. 2022-10-05T12:53:21.635095Z 0 [Note] Event Scheduler: Loaded 0 events 2022-10-05T12:53:21.635645Z 0 [Note] mysqld: ready for connections. Version: '5.7.38' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL) 2022-10-05T12:56:07.554978Z 7 [Note] Aborted connection 7 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 2022-10-05T12:57:37.555598Z 9 [Note] Aborted connection 9 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 2022-10-05T12:59:07.557053Z 11 [Note] Aborted connection 11 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 2022-10-05T13:00:37.551263Z 13 [Note] Aborted connection 13 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 2022-10-05T13:02:07.552976Z 15 [Note] Aborted connection 15 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 2022-10-05T13:03:37.547383Z 17 [Note] Aborted connection 17 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 2022-10-05T13:07:57.543144Z 19 [Note] Aborted connection 19 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 2022-10-05T13:14:37.538061Z 21 [Note] Aborted connection 21 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 2022-10-05T13:16:07.554355Z 23 [Note] Aborted connection 23 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets)

Impacted by this bug? Give it a 👍.

gkcalat commented 1 year ago

Hi @psheorangithub!

Could you please clarify where and how did you deploy Kubeflow 1.6? I see some placeholders in the yaml manifests. Were they replaced before applying?

asahnovskiy-deloitte commented 1 year ago

@psheorangithub Did you resolve the issue? it seems I have the same

asahnovskiy-deloitte commented 1 year ago

@gkcalat I have the same issue, when I change minio to use gsc instead pvc. But I don't understand how it related to mysql connections from ml-pipeline

asahnovskiy-deloitte commented 1 year ago

@gkcalat logs from ml-pipeline I0201 13:28:57.675273 7 client_manager.go:160] Initializing client manager 10 I0201 13:28:57.675379 7 config.go:57] Config DBConfig.ExtraParams not specified, skipping 9 [mysql] 2023/02/01 13:28:57 packets.go:37: unexpected EOF 8 [mysql] 2023/02/01 13:28:58 packets.go:37: unexpected EOF 7 [mysql] 2023/02/01 13:28:58 packets.go:37: unexpected EOF 6 [mysql] 2023/02/01 13:29:00 packets.go:37: unexpected EOF 5 [mysql] 2023/02/01 13:29:02 packets.go:37: unexpected EOF 4 [mysql] 2023/02/01 13:29:05 packets.go:37: unexpected EOF 3 [mysql] 2023/02/01 13:29:07 packets.go:37: unexpected EOF 2 [mysql] 2023/02/01 13:29:12 packets.go:37: unexpected EOF 1 [mysql] 2023/02/01 13:29:16 packets.go:37: unexpected EOF

asahnovskiy-deloitte commented 1 year ago

logs from mysql 2023-02-01T09:24:18.770139Z 278 [Note] Aborted connection 278 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 299 2023-02-01T11:10:37.355390Z 711 [Note] Aborted connection 711 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 298 2023-02-01T11:10:38.763233Z 713 [Note] Aborted connection 713 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 297 2023-02-01T11:10:57.104936Z 717 [Note] Aborted connection 717 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 296 2023-02-01T11:11:26.189905Z 721 [Note] Aborted connection 721 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 295 2023-02-01T11:12:09.084285Z 725 [Note] Aborted connection 725 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 294 2023-02-01T11:13:33.124578Z 733 [Note] Aborted connection 733 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 293 2023-02-01T11:16:18.158538Z 747 [Note] Aborted connection 747 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 292 2023-02-01T11:21:21.536725Z 769 [Note] Aborted connection 769 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 291 2023-02-01T11:26:28.264881Z 791 [Note] Aborted connection 791 to db: 'mlpipeline' user: 'root' host: '127.0.0.6' (Got an error reading communication packets) 290 2023-02-01T11:31:36.117026Z 813 [Note] Aborted connection 813 to db: 'mlpipeline' user: 'ro

gkcalat commented 1 year ago

Hi @asahnovskiy-deloitte,

Which version of KFP are you using? What type of deployment is it (standalone, full Kubeflow, etc.)?

@gkcalat I have the same issue, when I change minio to use gsc instead pvc. But I don't understand how it related to mysql connections from ml-pipeline

How did you change the storage? Did you deploy a new instance or used kubectl edit on an existing cluster? Any chances you removed or changed MySQL's PVC? MySQL expects to find local files in /var/lib/mysql (mysql-pv-claim).

Can you also provide logs from minio?

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

rimolive commented 6 months ago

Closing this issue. No activity for more than a year.

/close

google-oss-prow[bot] commented 6 months ago

@rimolive: Closing this issue.

In response to [this](https://github.com/kubeflow/pipelines/issues/8332#issuecomment-2016970638): >Closing this issue. No activity for more than a year. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

kubeflow / pipelines