llm hangs forever when running Replicate models

[zeke]

Works for OpenAI:

$ llm "Five cute names for a pet penguin"
1. Pippin
2. Snowball
3. Waddles
4. Flippers
5. Chilly

Doesn't work for Replicate:

$ llm --version
llm, version 0.13.1

$ llm install llm-replicate
Requirement already satisfied: llm-replicate in /opt/homebrew/Cellar/llm/0.13.1_1/libexec/lib/python3.12/site-packages (0.3.1)
Requirement already satisfied: llm in /opt/homebrew/Cellar/llm/0.13.1_1/libexec/lib/python3.12/site-packages (from llm-replicate) (0.13.1)
...etc

$ llm keys set replicate
Enter key: 

$ llm replicate fetch-models

$ llm models list | grep replicate                 
Replicate: replicate-flan-t5-xl
Replicate: replicate-llama-7b
...etc
Replicate (chat): replicate-meta-meta-llama-3-70b-instruct (aliases: llama3)
Replicate (chat): replicate-a16z-infra-llama13b-v2-chat (aliases: llama2)

$ llm -m llama2 "Ten great names for a pet pelican"
(HANGS FORVER)

[zeke]

I thought maybe this was because of a redirect (https://replicate.com/a16z-infra/llama-2-13b-chat was renamed to https://replicate.com/meta/llama-2-13b-chat) but it hangs with every Replicate model I try.

[simonw]

Thanks - I need to upgrade this plugin to the latest Replicate library version and make a bunch of changes.