Initial plugin

[simonw]

Description:

LLM plugin for models hosted on Replicate

Will use the replicate PyPI package: https://replicate.com/docs/get-started/python

[simonw]

I prototyped this in a Jupyter notebook:

import replicate
client = replicate.Client(api_token=token)
output = client.run(
    "joehoover/falcon-40b-instruct:7eb0f4b1ff770ab4f68c3a309dd4984469749b7323a3d47fd2d5e09d58836d3c",
    input={"prompt": "ten names for a pet tiger"}
)
print("".join(output))

[simonw]

There's a collection of LLMs here: https://replicate.com/collections/language-models

Also available via API:

import requests
r = requests.get(
    "https://api.replicate.com/v1/collections/language-models",
    headers={"Authorization": "Token {}".format(token)
})
llm_models = r.json()['models']
version = llm_models[0]['latest_version']
version['openapi_schema']['components']['schemas']['Input']

Which produces the things that should be turned into options:

{'type': 'object',
 'title': 'Input',
 'required': ['prompt'],
 'properties': {'debug': {'type': 'boolean',
   'title': 'Debug',
   'default': False,
   'x-order': 5,
   'description': 'provide debugging output in logs'},
  'top_p': {'type': 'number',
   'title': 'Top P',
   'default': 1,
   'maximum': 1,
   'minimum': 0.01,
   'x-order': 3,
   'description': 'When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens'},
  'prompt': {'type': 'string',
   'title': 'Prompt',
   'x-order': 0,
   'description': 'Prompt to send to FLAN-T5.'},
  'max_length': {'type': 'integer',
   'title': 'Max Length',
   'default': 50,
   'minimum': 1,
   'x-order': 1,
   'description': 'Maximum number of tokens to generate. A word is generally 2-3 tokens'},
  'temperature': {'type': 'number',
   'title': 'Temperature',
   'default': 0.75,
   'maximum': 5,
   'minimum': 0.01,
   'x-order': 2,
   'description': 'Adjusts randomness of outputs, greater than 1 is random and 0 is deterministic, 0.75 is a good starting value.'},
  'repetition_penalty': {'type': 'number',
   'title': 'Repetition Penalty',
   'default': 1,
   'maximum': 5,
   'minimum': 0.01,
   'x-order': 4,
   'description': 'Penalty for repeated words in generated text; 1 is no penalty, values greater than 1 discourage repetition, less than 1 encourage it.'}}}

[simonw]

Maybe have this command:

llm replicate fetch-models

Which fetches and caches details of the language models in that collection.

Separately, this command:

llm replicate add \
  joehoover/falcon-40b-instruct:7eb0f4b1ff770ab4f68c3a309dd4984469749b7323a3d47fd2d5e09d58836d3c

Which adds that language model. It could take --alias x options for setting aliases too.

So I think Replicate ends up with its own .../io.datasette.llm/replicate folder where it stores a language-models.json cache and a extra-models.yaml file with models added by the user.

[simonw]

I'll get this working without options first.

[simonw]

Ran into a problem: replicate-python still uses Pydantic 1.0 but LLM uses 2.0.

Opened a PR with a fix for that here:

• https://github.com/replicate/replicate-python/pull/121

[simonw]

In the meantime I can depend on my branch to get a temporary fix:

dependencies = [
    "llm",
    # Fork of replicate-python that works with Pydantic 2.0:
    "replicate @ https://github.com/simonw/replicate-python/archive/6c8edb74dce8492095de029602d37816db55fde8.zip"
]

[simonw]

llm -m replicate-llama-7b '3 great names for a pet alpaca'

Whether you want a cute nickname or a name that reflects their alpaca heritage, these names are great for the friendly animal that gives you warmth, comfort, and cuddles. Whether an alpaca is a pet or a farm animal, the alpaca is a great friend. They are playful and have a gentle disposition, which makes them excellent family pets. In fact, if you’re looking for a new member for your family, alpacas are great for families with young children. For some families, they even help with raising the kids. If you’re looking for a great name for your alpaca, then you’re in luck! Here are some great names for male or female alpacas. A name for a male alpaca If you’re looking for a name for a male alpaca, then you should consider the name of a Greek god. These names are strong and will reflect the mood of your alpaca: Aeolus: This is one of the names of the god of wind. When given this name, your alpaca will have a great sense of adventure and will be able to wander from place to place. The name will also make you think about the wind, which is a great way to show your appreciation for your alpaca’s strength and resilience. Ares: This name is an excellent fit for an alpaca, because they are a strong and dominant animal. They are also known to be a bit unpredictable, which is why their namesake was the god of war. If you’re looking for a name that will make people think about the power of your alpaca, then this name is perfect for you. Poseidon: This name is also a great fit for an alpaca, because they are a strong animal. They also often live at sea or in an area that can become flooded, which is why they are named after the god of the sea. If you want a name that will show that your alpaca is a strong animal, then this is the name for you. A name for a female alpaca If you want to give your female alpaca an appropriate name, then you should consider these names:

[simonw]

After running:

llm replicate fetch-models

This command:

llm models list

Includes these:

ReplicateModel: replicate-flan-t5-xl
ReplicateModel: replicate-llama-7b
ReplicateModel: replicate-gpt-j-6b
ReplicateModel: replicate-dolly-v2-12b
ReplicateModel: replicate-oasst-sft-1-pythia-12b
ReplicateModel: replicate-stablelm-tuned-alpha-7b
ReplicateModel: replicate-vicuna-13b
ReplicateModel: replicate-replit-code-v1-3b
ReplicateModel: replicate-replit-code-v1-3b