W0911 03:12:55.601024 1 metadata_store_server_main.cc:550] Connection Aborted with error: ABORTED: In the given db, MLMDEnv table exists but no schema_version can be found. This may be due to concurrent connection to the empty database. Please retry connection. I0911 03:12:55.601061 1 metadata_store_server_main.cc:551] Retry attempt 4

vak890 commented 6 days ago

Validation Checklist

[X] Is this a Kubeflow issue?
[X] Are you posting in the right repository ?
[X] Did you follow the Kubeflow installation guideline ?
[X] Is the issue report properly structured and detailed with version numbers?
[ ] Is this for Kubeflow development ?
[X] Would you like to work on this issue?
[ ] You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.9

Describe your issue

When I followed the kubeflow installation guide, metadata grpc deployment couldn't run successfully kubectl logs -f metadata-grpc-deployment-786f94dc8b-kfrgc -n kubeflow W0911 03:12:55.545574 1 metadata_store_server_main.cc:550] Connection Aborted with error: ABORTED: In the given db, MLMDEnv table exists but no schema_version can be found. This may be due to concurrent connection to the empty database. Please retry connection. I0911 03:12:55.545694 1 metadata_store_server_main.cc:551] Retry attempt 0 W0911 03:12:55.559099 1 metadata_store_server_main.cc:550] Connection Aborted with error: ABORTED: In the given db, MLMDEnv table exists but no schema_version can be found. This may be due to concurrent connection to the empty database. Please retry connection. I0911 03:12:55.559130 1 metadata_store_server_main.cc:551] Retry attempt 1 W0911 03:12:55.572381 1 metadata_store_server_main.cc:550] Connection Aborted with error: ABORTED: In the given db, MLMDEnv table exists but no schema_version can be found. This may be due to concurrent connection to the empty database. Please retry connection. I0911 03:12:55.572420 1 metadata_store_server_main.cc:551] Retry attempt 2 W0911 03:12:55.586738 1 metadata_store_server_main.cc:550] Connection Aborted with error: ABORTED: In the given db, MLMDEnv table exists but no schema_version can be found. This may be due to concurrent connection to the empty database. Please retry connection. I0911 03:12:55.586771 1 metadata_store_server_main.cc:551] Retry attempt 3 W0911 03:12:55.601024 1 metadata_store_server_main.cc:550] Connection Aborted with error: ABORTED: In the given db, MLMDEnv table exists but no schema_version can be found. This may be due to concurrent connection to the empty database. Please retry connection. I0911 03:12:55.601061 1 metadata_store_server_main.cc:551] Retry attempt 4 F0911 03:12:55.614584 1 metadata_store_server_main.cc:555] Check failed: absl::OkStatus() == status (OK vs. ABORTED: In the given db, MLMDEnv table exists but no schema_version can be found. This may be due to concurrent connection to the empty database. Please retry connection.) MetadataStore cannot be created with the given connection config. Check failure stack trace:

Steps to reproduce the issue

kustomize build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | kubectl apply -f -

Put here any screenshots or videos (optional)

No response

vak890 commented 6 days ago

I found that the schema_version field is present in MySQL mysql log:

juliusvonkohout commented 6 days ago

It might be related to https://github.com/kubeflow/manifests/issues/2800#issuecomment-2340570754 and the merged PRs, although I did not check the details.

vak890 commented 5 days ago

@juliusvonkohout The solution is whether the kubeflow platform needs to be deployed in a new Kubernetes cluster, but our current Kubernetes cluster is native without any changes, and only some pods have been deployed in the cluster

juliusvonkohout commented 5 days ago