[FR] Is someone working with LangGraph integration, or Interesting to develop it?

daniel-maturana commented 1 month ago

Willingness to contribute

Yes. I can contribute this feature independently.

Proposal Summary

LangGraph use langchain expression language and graph to generate Agents, It is a good solution to multi agent development and for sure it going to increase the use. I currently developed several Agents using Datbaricks data, SQL, retrievals, etc.. But I need to deploy them on Databricks using mlflow.

Motivation

What is the use case for this feature?

Why is this use case valuable to support for MLflow users in general?

Why is this use case valuable to support for your project(s) or organization?

Why is it currently difficult to achieve this use case?

Details

No response

What component(s) does this bug affect?

[X] area/artifacts: Artifact stores and artifact logging
[X] area/build: Build and test infrastructure for MLflow
[X] area/deployments: MLflow Deployments client APIs, server, and third-party Deployments integrations
[ ] area/docs: MLflow documentation pages
[ ] area/examples: Example code
[X] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
[X] area/models: MLmodel format, model serialization/deserialization, flavors
[ ] area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
[ ] area/projects: MLproject format, project running backends
[ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
[ ] area/server-infra: MLflow Tracking server backend
[ ] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

[ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
[ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
[ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
[ ] area/windows: Windows support

What language(s) does this bug affect?

[ ] language/r: R APIs and clients
[ ] language/java: Java APIs and clients
[ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

[ ] integrations/azure: Azure and Azure ML integrations
[ ] integrations/sagemaker: SageMaker integrations
[ ] integrations/databricks: Databricks integrations

harupy commented 1 month ago

@daniel-maturana Thanks for the FR! It looks like LangGraph compiles the graph into a LangChain Runnable (which mlflow.langchain supports). Can you use mlflow.langchain.log_model to log your graph?

daniel-maturana commented 1 month ago

Hi Harutaka, yes I’m can log the calls, and trace the model, thats working perfect. But I was actually interested in saving, loading, and deploying models using mlflow.

Maybe is not exactly here, I am trying to use the integration of mlflow in Databricks, to deploy the Agent.

Best,

Daniel

harupy commented 1 month ago

@daniel-maturana can you register the logged model and deploy it as an endpoint? Here's a tutorial that demonstrates how to create custom model serving endpoints:

https://docs.databricks.com/en/machine-learning/model-serving/create-manage-serving-endpoints.html

daniel-maturana commented 1 month ago

Hi @harupy , Well yes, it is possible and it iw what I did. I tried to deploy it similarly than langchain, registering the model with

mlflow.models.set_model

But it was not possible because it is not a langchain type.

I finally registered the model as a pyfunc

Is there any big difference between both saving, registering the model?

Thanks

github-actions[bot] commented 1 month ago

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

morenoj11 commented 1 month ago

Hi @harupy , Well yes, it is possible and it iw what I did. I tried to deploy it similarly than langchain, registering the model with
mlflow.models.set_model
But it was not possible because it is not a langchain type.

I finally registered the model as a pyfunc

Is there any big difference between both saving, registering the model?

Thanks

@daniel-maturana did you get it to work for the tracing as well?

daniel-maturana commented 1 month ago

@morenoj11 I used the pyfunc model with code to do the deployment in databricks. In my .py file with all the logic, I create a class with the predict and predict_stream methods as well as some accesory classes to do the input to state and state to output for langraph.| Then I registered and deployed in databricks, no problem. When I run it in the notebook, it is working perfect and tracing all the nodes, chains or functions where I added the tracer decorator. I also modified the output to return the SQL queries as well as code, so everything is going.

However now I am stopped because my langraph, which is a SQL and coder agent is taking to long and the serving is giving worker timeout.

I will tryi with another RAG langraph that I alradey have to check if the timeout is langraph problem or is really a timeout. I tried to change several TIMEOUT env variables from mlflow but is not working yet.

Best,

morenoj11 commented 1 month ago

@daniel-maturana thanks for the elaborate response! Would you mind keeping me/this discussion updated on the progress? Also if you could share some sample code it would be great. I'm in a similar situation, and I think I'll go with Langfuse/Langsmith until MLflow catches up, but if there's a workaround I'd be down for it.

Thanks!

mindful-time commented 1 week ago

Langsmith can capture feedback for experiments. Is there a similar capability in mlflow like langgraph/langchain?

B-Step62 commented 1 day ago

Hi folks! MLflow now support saving LangGraph in native langchain flavor. Please check it out and let us know if there are any bugs or feedbacks! https://mlflow.org/docs/latest/llms/langchain/index.html#how-can-i-log-an-agent-built-with-langgraph-to-mlflow

B-Step62 commented 21 hours ago

Langsmith can capture feedback for experiments. Is there a similar capability in mlflow like langgraph/langchain?

@mindful-time We haven't supported this but we are planning to add a feature to record human feedback. Please feel free to open a new FR if you have specific use case in your mind. Thanks!

mlflow / mlflow