Open rxm7706 opened 8 months ago
This is a duplicate of https://github.com/Galileo-Galilei/kedro-mlflow/issues/344. It is quite old, so I will reassess it to see if we can make it work!
Sorry I didn't see that issue earlier, I see your response explaining the associated effort it would take to make this change. Thank you & Regards.
Some good & bad news after testing:
kedro-mlflow
works 99% correctly with mlflow-skinny. The only issues arise from MlflowModelRegistryDataset
and this can be fixed by installing sqlalchemy
and alembic
. This is not even a real problem because I don't think people use the registry only locally without a remote mlflow server (with the whole mlflow install) so they very likely have what it needs install. kedro-mlflow
locally and want to open the ui with kedro mlflow ui
. You need to install flask
and waitress
(but this is straightforward), but you also need to have a couple of .js
files, which are not bundled with mlflow skinny. This means this is not really possible to get these files just by installing an extra dependency (apart from downloading them directly from the mlflow source, but this is complicated to make it reliable across versions). I don't think people would accept dropping the local in in favor of faster builds. Decision : I think the only way to make it work is to publish a kedro-mlflow-skinny
separately on PyPI whose only difference with standard kedro-mlflow would be the replacement of the mlflow
dependency by mlflow-skinny.
TODO :
setup.py
to handle packaging the package in two different ways. mlflow
can be a [good source for inspiration https://github.com/mlflow/mlflow/blob/5d13d6ec620a02de9a5e31201bf1becdb9722ea5/setup.py#L126]kedro-mlflow-skinny
syncrhoneaously with kedro-mlflow
PR's are welcome!
If you like the repo, please give it a :star:
Description
A clear and concise description of what you want to achieve. An image or a code example is worth thousand words!
With the introduction of ML-FLOW AI Gateway; ML-FLOW has become quite large in the number of dependencies. To manage the growth of the ML-FLOW ecosystem, MLFLOW-Skinny was introduced.
conda install mlflow-skinny vs conda install mlflow is over 100 packages additional.
Context
Why is this change important to you? How would you use it? How can it benefit other users?
Currently an Open CVE on Pyarrow https://nvd.nist.gov/vuln/detail/CVE-2023-47248 is flagged on kedro-mlflow because it depends on ML-Flow ; because one of the additional dependencies uses pyarrow
Possible Implementation
(Optional) Suggest an idea for implementing the addition or change.
If its possible for kedro-mlflow to use mlflow-skinny, it might be a good idea to change the dependency from mlflow to mlflow-skinny and let users manage their dependencies with more granularity.
Possible Alternatives
(Optional) Describe any alternative solutions or features you've considered.
do nothing - leave things the way they are , but kedro-mlflow becomes bloated as mlflow full grows.