Closed PriyanshBhardwaj closed 1 week ago
@PriyanshBhardwaj
The inconsistent type naming between the model deployer registration and the MLFlowDeploymentService
definition could also be contributing to this issue.
One thing you try is to register the MLFLow model deployer as
zenml model-deployer register mlflow --type=model-serving --flavor=mlflow
As it's defined in ServiceType as:
SERVICE_TYPE = ServiceType(
name="mlflow-deployment",
type="model-serving",
flavor="mlflow",
description="MLflow prediction service",
)
Or we can change the type in src\zenml\integrations\mlflow\services\mlflow_deployment.py on line 128 to just mlflow
rather than mlflow-serving
.
So that the service type name matches what is defined in the MLFlowDeploymentService
class.
This can contribute to the issue you are facing.
@Vishal-Padia thanks for your response.
zenml model-deployer register mlflow --type=model-serving --flavor=mlflow
this will not work as --type
is an extra field in model deployer registration as you can see in the error below. I tried it because I saw it in some old docs.
ValidationError: 1 validation error for MLFlowModelDeployerConfig
type
extra fields not permitted (type=value_error.extra)
for your 2nd solution, I tried it but it didn't work.
@PriyanshBhardwaj okay, I got it. I guess Alex will be able to zero-down on the issue!
Hello @PriyanshBhardwaj , sorry about the delay!
As I learned from the info provided you install mlflow
using pip directly, which might not work well with zenml due to version mismatch. Moreover mlflow
integration also pulls important components for model deployment, the full list is (for 0.53.1
version you used): 'mlflow>=2.1.1,<=2.9.2', 'mlserver>=1.3.3', 'mlserver-mlflow>=1.3.3'
.
Can you do the following and retest?
pip3 uninstall mlflow
zenml integration install mlflow -y
Moreover, forking on MacOS might not be working always smoothly, if OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
is not set, so also ensure that this ENV is properly set before rerunning.
Did someone came up with any solutions? Because I tried everything mentioned over here and still getting that error. @avishniakov @PriyanshBhardwaj
@jatinpreeet Thank you for messaging here. Some questions so we can narrow down exactly what's going on in your case:
TypeError
you're seeing in the logs? That will help identify exactly where the unregistered service type error is originating.mlflow>=2.1.1,<=2.9.2
?OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
environment variable, but the error still persists after that, correct?zenml info -a -s
would also be really useful. Feel free to message me on Slack with this information, or paste it into a Gist if it's too long etc.Please let me know the answers to the above when you can. I know this has been frustrating to troubleshoot, but I appreciate you taking the time to provide all these details. We'll get to the bottom of it! Let me know if any other questions or issues come up in the meantime.
Hello there! Seems that I also face the same issue. Code to reproduce can be found here: https://github.com/michalziobro/mlops-projects-course it is based on: https://github.com/ayush714/mlops-projects-course/tree/main
mlflow version is 2.15.1
, installed with zenml integration install mlflow -y
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
is set (otherwise zenml up
fails)
zenml version is 0.64.0
, more details here: https://gist.github.com/michalziobro/2bf39c55e8f696e8f4b20215a1319586
Hello there! Seems that I also face the same issue. Code to reproduce can be found here: https://github.com/michalziobro/mlops-projects-course it is based on: https://github.com/ayush714/mlops-projects-course/tree/main
mlflow version is
2.15.1
, installed withzenml integration install mlflow -y
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
is set (otherwisezenml up
fails)zenml version is
0.64.0
, more details here: https://gist.github.com/michalziobro/2bf39c55e8f696e8f4b20215a1319586
Hello @michalziobro , thank you for reporting this.
I've tried to reproduce this issue using your code repository, but I ran into several issues unrelated to what you reported here:
requirements.txt
file you provided cannot be installed due to conflicts like these:INFO: pip is looking at multiple versions of streamlit to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements.txt (line 168), -r requirements.txt (line 28), -r requirements.txt (line 48), -r requirements.txt (line 90), -r requirements.txt (line 91) and click==8.1.7 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested click==8.1.7
click-params 0.3.0 depends on click<9.0 and >=7.0
flask 3.0.3 depends on click>=8.1.3
mlflow-skinny 2.15.1 depends on click<9 and >=7.0
mlserver 1.4.0 depends on click
streamlit 1.8.1 depends on click<8.1 and >=7.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict
I had to manually adjust your requirements.txt
several times to get everything to install correctly. Among other things, I removed backports.zoneinfo
and streamlit
.
the code repository makes a lot of references to a dataset stored in path that is only available on your machine: /Users/michalziobro/Documents/Private/customer-satisfaction-mlops-main/data/olist_customers_dataset.csv
. I had to manually fix this so it could run on my PC.
running python run_deployment.py
doesn't actually deploy a model because the deployment_trigger
always returns False, because the accuracy of the model is really really low (0.0834 compared to the trigger threshold set to 0.92). I had to manually bypass the deployment_trigger
implementation to always return True.
Finally, after making several manual changes to the code you provided, I was able to successfully run the continuous deployment and inference pipelines.
In order for me to be able to help you, I would kindly ask you to revise your code and then provide me with a more accurate way to reproduce this on my machine.
System Information
python = 3.9 zenml version = 0.53.1 os = macos integration = mlflow (downloaded separately by pip install mlflow)
What happened?
Unable to deploy a ml model locally in mlflow. The problem lies in class
MLFlowDeploymentService
in filemlflow_deployment.py
.Please check the reproduction steps to understand the issue clearly.
Reproduction steps
I followed all steps correctly, set the experiment tracker, model deployer and created the stack:
i created a pipeline which will ingest data, train the model, evaluate performance and then deploy model after passing the trigger:
I debugged it, everything is working good and the model is also passing the deployment trigger but the model deployer is not working properly. the problem is with this step:
when pipeline calls it, the log which prints is:
this gets logged from function
deploy_model()
which is in the file/zenml/integrations/mlflow/model_deployers/mlflow_model_deployer.py
. In the same function at line 210 it callsservice.start()
which is in the file/zenml/services/local/local_service.py
and in the same under the start function it logsStarting service 'MLFlowDeploymentService[2ade1153-7fd3-45d1-8ecd-412f86b264b5] (type: model-serving, flavor: mlflow)'.
from line 387 and then when it callsif not self.poll_service_status(timeout):
at line 391 it logs error:when i visited the log file it says:
it raises this issue from here:
/zenml/services/service_registry.py:193 in load_service_from_dict
but in the file
mlflow_model_deployer.py
at line 187 service gets its value from hereservice = cast(MLFlowDeploymentService, existing_service)
which isMLFlowDeploymentService[2ade1153-7fd3-45d1-8ecd-412f86b264b5] (type: model-serving, flavor: mlflow)
.but in class
MLFlowDeploymentService
in filemlflow_deployment.py
at line 128 type is already defined as "model-serving" which i think cant be changed:It is getting timed out due to this because
MLFlowDeploymentService[2ade1153-7fd3-45d1-8ecd-412f86b264b5] (type: model-serving, flavor: mlflow)
is not starting and it will always give timeout error doesn't matter what will be the value of timeout.so why it is giving this error in log file
I tried everything to solve it: created new stack from scratch created new pipeline in different stacks initialize zenml again by deleting .zen folder and again calling
zenml init
in terminal in same directoryi also tried to use
--type=mlflow
while creating model deployer as mentioned somewhere in your old docs in this command:It obviously didnt work.
Nothing worked and also your docs doesnt have a solution for such problems.
My issue is I tried everything to debug this but not able to deploy my model locally to mlflow bcz i cant change type in your internal class which is the root cause. Please resolve the issue and please update your logs and make them more clear for the users.
P.S: model size 1000 bytes only, timeout 60, 120 didnt work for both values I'm using a separate env for this. latest version of zenml and mlflow
Relevant log output
Code of Conduct