Replicate Version Numbers and issues with running Llama 3 using Langchain's Replicate class

[abhirupghosh]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain_community.llms import Replicate

model = Replicate(
    model="meta/meta-llama-3-70b-instruct:" + version,
    model_kwargs={"temperature": 0.2, "max_length": 1024, "top_p": 1},
)

This when compared to directly using Replicate's API within Python:

import replicate

replicate.run(
        "meta/meta-llama-3-70b-instruct",
        input={
            "top_p": 0.9,
            "prompt": prompt,
            "max_tokens": 512,
            "min_tokens": 0,
            "temperature": 0.6,
            "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
            "presence_penalty": 1.15,
            "frequency_penalty": 0.2
        }
    )

Error Message and Stack Trace (if applicable)

replicate.exceptions.ReplicateError: ReplicateError Details: title: Invalid version or not permitted status: 422 detail: The specified version does not exist (or perhaps you don't have permission to use it?)

Description

I am trying to use Langchain to use Llama 3 - however, there are no version numbers that are required when I am using Replicate's API directly. There are also no direct ways on the Replicate website to find which specific version number we are using when trying to use replicate.

To identify the version number, I queried https://api.replicate.com/v1/models/meta/meta-llama-3-70b-instruct as a GET request, and received the latest_version in the response. Upon feeding this latest_version into the 'version' variable, I still get the above error message.

Two questions:

1. Am I doing something wrong here when invoking the Replicate model using Langchain?
2. Can we get rid of the version number requirement, when Replicate's own API does not require a version number in most scenarios? It could be an optional parameter perhaps.

System Info

System Information

OS: Darwin OS Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:25 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6030 Python Version: 3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:35:41) [Clang 16.0.6 ]

Package Information

langchain_core: 0.1.42 langchain: 0.1.16 langchain_community: 0.0.32 langsmith: 0.1.46 langchain_text_splitters: 0.0.1

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph langserve

[ebakken]

This is also an issue with langchainjs.

Setting the Replicate model version was made optional some time ago, but the Langchain wrappers has not been updated to reflect this change. Public Replicate models should now be run without setting version.

[abhirupghosh]

Any update on correcting this? A lot of newer models (such as Llama 3) do not explicitly mention the version numbers on Replicate's end. To reiterate, to identify the version number, I queried https://api.replicate.com/v1/models/meta/meta-llama-3-70b-instruct as a GET request, and received the latest_version in the response. Upon feeding this latest_version into the 'version' variable, I still get the same error message.

I am temporarily being forced to use Replicate's API directly, which makes it more difficult to utilize inbuilt output parsers, amongst various other thing.

[Jonathan-Ddn]

I have the exact same issue, no idea how to fix it...

[rdewolff]

same proble, would love to have this fixed

[jondoescoding]

Same problem here as well. Unable to use Replicate models on python as well

[rdewolff]

Anyone taking care of this? The issue is still here. Updating models with version is a PITA.

[adibenmati]

the solution is to use langchain community, https://api.python.langchain.com/en/latest/community_api_reference.html https://pypi.org/project/langchain-community/

[rohititaliaya]

also facing same issue. simple curl request also not working

[me1nna]

has anyone solved this problem?

[BeastDevMoney]

Has any one know a way to use prompt templates(including variables) with replicate models? There is a way to use langchain with replicate?

[geido]

The information from the author seems incorrect or outdated. There is a way to know the version on Replicate by clicking on "versions" and on the specific version.

You will get a URL like this in the navbar:

https://replicate.com/lucataco/phi-3-mini-128k-instruct/versions/45ba1bd0a3cf3d5254becd00d937c4ba0c01b13fa1830818f483a76aa844205e

Where the version is 45ba1bd0a3cf3d5254becd00d937c4ba0c01b13fa1830818f483a76aa844205e.

To use that with langchain_community:

from langchain_community.llms import Replicate

Replicate(
    model=lucataco/phi-3-mini-128k-instruct:45ba1bd0a3cf3d5254becd00d937c4ba0c01b13fa1830818f483a76aa844205e,
    api_key=llm_api_key,
)

[cicheck]

@geido not all models have a version tab, e.g. https://replicate.com/meta/meta-llama-3-70b-instruct

[cicheck]

@adibenmati I am not sure what you mean, OP did use langchain community. Can you provide a functioning snippet that uses meta/meta-llama-3-70b-instruct without specifying the model version?

[adibenmati]

@cicheck First of all the most important is to use the following import code:

from langchain_community.llms import Replicate

make sure you are using "langchain_community" recommended version is langchain-community > 0.2.2

1. def load_model(self): replicate_api_token = ConfigManager.get(ConfigKeys.replicate_token) os.environ["REPLICATE_API_TOKEN"] = replicate_api_token

   llama_model_name = "meta/meta-llama-3.1-405b-instruct"
   llm = Replicate(
       model=llama_model_name,
       model_kwargs={"temperature": 0.1, "top_p": 1, "max_new_tokens": 100},
       api_token=replicate_api_token
   )

2. def query_model(prompt, temperature=0.01, max_tokens=1000, attempts=2):
   print(f"Querying model with prompt: {prompt[:50]}...")  # Log the first 50 characters of the prompt
   input = {
       "prompt": prompt,
       "temperature": temperature,
       "max_tokens": max_tokens
   }
   responses = []
   for attempt in range(attempts):
       print(f"Query attempt {attempt + 1}")
       response = ""
       for event in replicate.stream(
               "meta/meta-llama-3.1-405b-instruct",
               input=input
       ):
           response += event.data
       responses.append(response.strip())
       print(f"Response from attempt {attempt + 1}: {response[:50]}...")  # Log the first 50 characters of the response
   
   return responses

   Yes it's not currently supported in the official langchain library, until then please use the community version. That should work hope it helps and everyone stop spamming my email :)

[cicheck]

2) Worked for me, thanks, and enjoy your peace:P

[ivan-wald]

This doesn't work for me. Getting this error, ValidationError: 1 validation error for Replicate model Field required [type=missing, input_value={'model_kwargs': {'model'...ing': False, 'stop': []}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing

[geido]

Any update on correcting this? A lot of newer models (such as Llama 3) do not explicitly mention the version numbers on Replicate's end. To reiterate, to identify the version number, I queried https://api.replicate.com/v1/models/meta/meta-llama-3-70b-instruct as a GET request, and received the latest_version in the response. Upon feeding this latest_version into the 'version' variable, I still get the same error message.

I am temporarily being forced to use Replicate's API directly, which makes it more difficult to utilize inbuilt output parsers, amongst various other thing.

This also works but the version needs to be specified like this

from langchain_community.llms import Replicate

Replicate(
    model=lucataco/phi-3-mini-128k-instruct:45ba1bd0a3cf3d5254becd00d937c4ba0c01b13fa1830818f483a76aa844205e,
    api_key=llm_api_key,
)