Closed natalytvinova closed 2 weeks ago
Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6213.
This message was autogenerated
@natalytvinova could you give some more context on what is included in the mlflow-overlay.yaml
?
We can't reproduce the issue without the context of what's in those bundles.
Also, what do you mean with clustered mysql?
@kimwnasptd I'm sorry, I didn't get your question about the context. Can you see the bundle attached correctly or are there any issues? Clustered mysql I mean is 3 mysql units
Hm, I'm actually able to reproduce this even with 1 mlflow-mysql unit
But maybe some resources didn't clean up like this time: https://github.com/canonical/istio-operators/issues/549
I have synced with @natalytvinova the problem was that in the original deployment we were using the same minio as for kfp ... After we have deployed separate minio and related it to mlflow the issue got resolved.
@natalytvinova will let me know if we need to use the same mino for kfp and mlflow.
@misohu what is the issue with relating KFP MinIO with MLflow and how is it related to mysql relations?
We will need to ensure that users could also use KFP MinIO if needed. Should we have a dedicated issue with logs around this to track the KFP MinIO integration in this case?
I dont think this issue has anything to do with mysql ... also the error above is from botocore which is lib used in mlflow to talk to minio (thats how I knew its minio).
The original design for mlflow was that it has its own minio so we never tested that situation. I did not dig deeper for now. so @kimwnasptd if we think we should support this one mino solution I think we can track it in this issue. WDYT ?
If we have two minio, which one will be added as a PodDefault to the notebooks?
so mlflow has its own poddefaults called
"mlflow-server-access-minio",
"mlflow-server-minio",
you can enable those when creating notebook in config
here is the content for each of those.
SO if you pick that mlflow config you will use AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
pointing to mlflow's minio in the notbook
I can see a problem where user may want to use mlflow from inside the notebook and talk to kfp minio at the same time ... the env variables will be set to MLflow minio. But there is no other poddefault for the kfp minio. Only if you create one manually. So if you have notebook where you want to talk to kfp minio not to mlflow then dont select the mlflow poddefault.
@Barteus So we distinguish between 3 usecases for minio:
"mlflow-server-access-minio",
"mlflow-server-minio",
Which mounts credentials for mlflow's minio to user notebook under AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
.
So if you want to have a notebook which can talk to mlflow and kfp minio you need to:
KFP_MINIO_ACCESS_KEY_ID
and KFP_MINIO_SECRET_ACCESS_KEY
and the variables for URL).# Initialize a MinIO client
mc = Minio(
endpoint=KFP_MINIO_HOST,
access_key=os.environ["KFP_MINIO_ACCESS_KEY_ID"],
secret_key=os.environ["KFP_MINIO_SECRET_ACCESS_KEY"],
secure=False,
)
Closing this issue, if you have a similar issue please refer to this comment for context.
Bug Description
Hi team,
I deployed mlflow-server and related it to a clustered mysql. This leads to relational-db-relation-changed hooks failure.
I also tried the same bundle, but without clustering mysql and the charm works fine.
Bundles: auth-overlay.yaml.txt storage-overlay.yaml.txt mlflow-overlay.yaml.txt kubeflow.yaml.txt cos-integration-overlay.yaml.txt
To Reproduce
Environment
Juju 3.5.3 Kubernetes: AKS version 1.29
Relevant Log Output
Additional Context
No response