Open simonw opened 6 hours ago
With llm-claude-3==0.9a0
installed:
curl http://localhost:8002/-/llm/chat -XPOST -d '{"prompt": "two names for a duck", "model": "claude-3-haiku"}' | jq
Outputs:
{
"prompt": "two names for a duck",
"response": "Here are two common names for a duck:\n\n1. Mallard\n2. Quack\n\nMallard is a specific species of duck, the most common and widespread type of wild duck. Quack is a generic term used to refer to ducks in general, based on the distinctive quacking sound they make.\n\nSome other names for ducks include:\n\n- Drake (male duck)\n- Hen (female duck)\n- Duckling (baby duck)\n- Waterfowl\n- Dabbler\n- Diving duck\n- Puddle duck\n\nBut \"mallard\" and \"quack\" are two of the most recognizable and commonly used names when referring to ducks.",
"details": {
"id": "msg_01QTfyra7jGXU8PFxaFT4SZB",
"content": [
{
"text": "Here are two common names for a duck:\n\n1. Mallard\n2. Quack\n\nMallard is a specific species of duck, the most common and widespread type of wild duck. Quack is a generic term used to refer to ducks in general, based on the distinctive quacking sound they make.\n\nSome other names for ducks include:\n\n- Drake (male duck)\n- Hen (female duck)\n- Duckling (baby duck)\n- Waterfowl\n- Dabbler\n- Diving duck\n- Puddle duck\n\nBut \"mallard\" and \"quack\" are two of the most recognizable and commonly used names when referring to ducks.",
"type": "text"
}
],
"model": "claude-3-haiku-20240307",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 12,
"output_tokens": 155
}
}
}
Without "model": "..."
in the request JSON it defaults to gpt-4o-mini
.
API keys are currently picked up from environment variables or keys.json
. I should provide an option to configure those using traditional Datasette plugin settings too.
That streaming API seems to work:
curl http://localhost:8002/-/llm/stream -XPOST \
-d '{"prompt": "two names for a duck with lengthy explanations", "model": "claude-3.5-sonnet"}' \
--no-buffer
data: {"text": "1. Dabbler -"}
data: {"text": " This name refers to the duck's"}
data: {"text": " characteristic feeding behavior of \"dabbling\" or t"}
data: {"text": "ipping forward in shallow water,"}
data: {"text": " tail-up and head-down, to"}
data: {"text": " filter food from beneath the surface. Dab"}
data: {"text": "bling ducks, also known as pu"}
data: {"text": "ddle ducks, use this technique"}
data: {"text": " to forage for aquatic plants, insects"}
data: {"text": ", and small crustaceans. The"}
data: {"text": " name captures both their playful nature"}
data: {"text": " and their practical approach to finding"}
data: {"text": " sustenance.\n\n2. Rip"}
data: {"text": "plefoot - This name reflects the duck's remarkable"}
data: {"text": " ability to navigate through water, creating"}
data: {"text": " gentle ripples as their webbed feet propel"}
data: {"text": " them gracefully across the surface. The name"}
data: {"text": " also evokes the peaceful, me"}
data: {"text": "ditative quality of watching ducks glide across a"}
data: {"text": " pond, their movements causing concentric circles to"}
data: {"text": " spread outward across the water's"}
data: {"text": " surface, connecting their presence to the broader"}
data: {"done": true}
Using new async support from this release:
I'm going to have two - one streaming and one non-streaming. Main purpose of these is to implement hidden API keys, and later to implement token accounting.