Deploying langchain pipeline in MLFlow not working

[lenehavehansen]

I have followed the "Build your Chat Bot with Dolly" demo and successfully logged an MLFlow model.

I am facing issues when loading the model again, following the demo I use:

def load_model_and_answer(similar_docs, question): 
  # Note: this will load the model once more in memory
  # Load the langchain pipeline & run inferences
  model_uri = 'runs:/2dc17c4b80994a3b9d300d199f6978a1/model'

  chain = mlflow.pyfunc.load_model(model_uri)
  chain.predict({"input_documents": similar_docs, "human_input": question})

To call the logged model I do the following:

hf_embed = HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2")

chroma_db = Chroma(collection_name="documentation_docs", embedding_function=hf_embed, persist_directory= gardening_vector_db_path)

question = "Why does this not work?"
similar_docs = chroma_db.similarity_search(question, k=1)

load_model_and_answer(similar_docs = similar_docs, question = question) 

Failing with:

MlflowException: Incompatible input types for column input_documents. Can not safely convert float64 to <U0.

Any suggestions for solving this?

Requirements: mlflow==2.3 configparser==5.2.0 gunicorn==20.1.0 langchain==0.0.199 numpy==1.24.3 pyyaml==6.0 requests==2.28.1 sqlalchemy==2.0.16 tornado==6.1

[QuentinAmbard]

Hi @lenehavehansen , sorry for the late reply. It looks like you have a type issue, like it's trying to convert float to string. How did you save your model ?

[lenehavehansen]

Now I had a chance to test it. It was due to saving the model in the wrong units. Adding the following when saving the model solved it:

input_schema = Schema([
            ColSpec("double", "input_documents")
        ])

      output_schema = Schema([
            ColSpec("double", "output_text")
        ])

Thanks!!

[QuentinAmbard]

ok I got it, I'll update the demo to include GPU model serving and have a complete solution and make sure this works too.

[srikanthgr]

@lenehavehansen Can you please explain how did you resolve this issue? I'm currently trying to deploy and create a serving point.

[nigelhussain]

@lenehavehansen @QuentinAmbard If possible do you think you can upload the full code for this. I have been having similar troubles loading the model after deploying.

[nigelhussain]

Anybody? @lenehavehansen @QuentinAmbard

[QuentinAmbard]

hzy @nigelhussain , sorry I've been busy with the new LLM rag demo content. Do you want to load the model as serverless serving endpoint at the end? I'm asking because it likely won't work out of the box, I need to make some other changes. I'm trying to release a full, updated demo with that in addition of the new LLM+rag demo.

For now did you try the new one which is using the mosaicML endpoint + the vector search?

[nigelhussain]

Hi @QuentinAmbard. Thank you for your response. Most of the relevant info is on here. https://github.com/databricks-demos/dbdemos/issues/61

But at this stage I don't believe I'm trying to load the model as a serverless serving endpoint yet, but I look forward to viewing that demo. It's mostly just saving the model into mlflow and loading it via mlflow.pyfunc.load_model that I'm interested in at this point. Unfortunately it's returning a bizzare error documented in the issue above when I try to load and feed data into the mlflow model.

I also did try that new demo and it looks quite good, but I'm more interested in the Open Source huggingface dolly demo at the moment and just trying to debug the issue I have been having.

[nigelhussain]

@QuentinAmbard did you get a chance to review/see this?