Check failed: absl::OkStatus() == status (OK vs. INTERNAL: mysql_real_connect failed: errno: , error: [mysql-error-info='']) MetadataStore cannot be created with the given connection config.

922tech commented 2 days ago

Validation Checklist

[X] Is this a Kubeflow issue?
[X] Are you posting in the right repository ?
[X] Did you follow the Kubeflow installation guideline ?
[X] Is the issue report properly structured and detailed with version numbers?
[X] Is this for Kubeflow development ?
[X] Would you like to work on this issue?
[X] You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.9

Describe your issue

I followed the instruction for installing kubeflow pipeline on version 1.9.1 After creation of the namespaces and pods:

NAME                                               READY   STATUS             RESTARTS         AGE
cache-deployer-deployment-55458bb9bf-d4vxf         1/1     Running            1 (16h ago)      21h
cache-server-568d98d56f-mxgck                      1/1     Running            34 (58s ago)     21h
metadata-envoy-deployment-b4bc46768-mmx8j          1/1     Running            1 (16h ago)      21h
metadata-grpc-deployment-65fc97b764-6h7k4          0/1     CrashLoopBackOff   99 (3m35s ago)   21h
metadata-writer-86b4945bd5-78tjx                   0/1     CrashLoopBackOff   57 (25s ago)     21h
minio-7c77bc59b8-sks9f                             1/1     Running            1 (16h ago)      21h
ml-pipeline-7f6bcb764-b9d4n                        0/1     Running            120 (6m8s ago)   21h
ml-pipeline-persistenceagent-d9c785cdb-hvxpj       0/1     CrashLoopBackOff  67 (4m56s ago)   21h
ml-pipeline-scheduledworkflow-6d5fc7cb6d-phftf     1/1     Running            1 (16h ago)      21h
ml-pipeline-ui-5785c9d69d-gf2mk                    1/1     Running            1 (107m ago)     21h
ml-pipeline-viewer-crd-5869ccf789-np2wx            1/1     Running            1 (16h ago)      21h
ml-pipeline-visualizationserver-65df54899f-9zzph   1/1     Running            1 (107m ago)     21h
mysql-758cd66576-4q56v                             1/1     Running            1 (16h ago)      21h
workflow-controller-5584bbc656-dlgmr               1/1     Running            2 (106m ago)     21h

As you see some pods do not run. And I think the main cause of the problem is that metadata-grpc-deployment cannot start working and it crashes with this error log:

WARNING: Logging before InitGoogleLogging() is written to STDERR
F1105 08:28:41.229691     1 metadata_store_server_main.cc:236] Check failed: absl::OkStatus() == status (OK vs. INTERNAL: mysql_real_connect failed: errno: , error:  [mysql-error-info='']) MetadataStore cannot be created with the given connection config.
*** Check failure stack trace: ***

I don't know why on earth it cannot connect to mysql!

Steps to reproduce the issue

Follow the instructions for installation of pipeline.

Put here any screenshots or videos (optional)

No response

juliusvonkohout commented 2 days ago

Please try again with version 1.9.1 or the master branch and report back.

922tech commented 1 day ago

Please try again with version 1.9.1 or the master branch and report back.

I have already tried these several times. Also deployed cert-manager but did not solve the problem

juliusvonkohout commented 1 day ago

Please follow the instructions and install according to the readme a full kubeflow platform on a kind cluster. If not I have to close or transfer the issue to Kubeflow/pipelines.

LehmBook commented 16 hours ago

Is your pv and pvc bond correctly and whats your config of your sql pv?

kubeflow / manifests