search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.6k stars 1.62k forks source link

[backend] mysql 8.0 support (upstream ml-metadata issue) #9549

Open thesuperzapper opened 1 year ago

thesuperzapper commented 1 year ago

While we updated our clients for MySQL 8.0 in https://github.com/kubeflow/pipelines/pull/8351, we still depend on google/ml-metadata, which does not fully support MySQL 8.0. In MySQL 8.0, the default-authentication-plugin was changed from mysql_native_password to caching_sha2_password, but ml-metadata does not support caching_sha2_password auth:

Furthermore, we are running a very old version of ml-metadata, specifically version 1.5.0 from 2021, but 1.13.1 exists, as raised here: https://github.com/kubeflow/pipelines/issues/5439#issuecomment-1572728997

Impacted by this bug? Give it a 👍.

zijianjoy commented 1 year ago

We are hoping to be able to upgrade MLMD when its next minor version is available (1.14.0)

davidspek commented 1 year ago

@zijianjoy Just wanted to ping you since v1.14.0 has now been released.

davidspek commented 1 year ago

It looks like this was implemented in https://github.com/kubeflow/pipelines/pull/9856, but that hasn't made it into a pipelines release yet.

thesuperzapper commented 1 year ago

@chensun just want to confirm if this important fix will be in the Kubeflow 1.8 version of Kubeflow Pipelines?

chensun commented 1 year ago

@chensun just want to confirm if this important fix will be in the Kubeflow 1.8 version of Kubeflow Pipelines?

https://github.com/kubeflow/pipelines/pull/9856 is not included in any KFP release yet, so it won't be in Kubeflow 1.8 either.

nithin8702 commented 10 months ago

We have upgraded mysql db from 5 to 8 and now its causing issue related to mlmd authentication plugin. i think kfp uses mysql_native_password type whereas mlmd requires caching_sha2_password. could anyone please how to overcome this? Is it possible to change mlmd to kfp like mysql_native_password ?

https://github.com/google/ml-metadata/issues/178 https://github.com/kubeflow/pipelines/issues/9549

@chensun @thesuperzapper

papagala commented 8 months ago

I'm confused by these two things at the same time, from the title of this issue

I don't think they can both be true at the same time 🙃

Any help would be appreciated 😄

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

thesuperzapper commented 6 months ago

This is absolutely critical, and should not be closed until it's resolved.

/lifecycle frozen

nithin8702 commented 5 months ago

Any solution for this issue?

thesuperzapper commented 5 months ago

@nithin8702 you can continue to use mysql_native_password for your Kubeflow Pipelines user on your MySQL 8 database.

You just need to specify mysql_native_password when creating the user, as MySQL 8.0.4+ will use caching_sha2_password by default, which is not supported by Kubeflow Pipelines (specifically ml-metadata https://github.com/google/ml-metadata/issues/178):

CREATE USER 'kubeflow'@'%' IDENTIFIED WITH mysql_native_password BY 'MY_PASSWORD';