Claude-3 plugin with attachments seems to ignore prompt

[mhalle]

If llm with the llm-claude-3 plugin version 0.6 and the claude-3-5-sonnet-latest model is used with an attachment, the prompt appears to be ignored. That's not true with, say, the gemini model.

Tested with llm -m claude-3-5-sonnet-latest -a image.png "Ignore the image, describe birds"

The output is a the same description of the image, no matter what the prompt. For the latest gemini pro model, in contrast, the interpretation of the image is dependent on the prompt.

--log shows that the prompt and the attachment are at least being logged.

[simonw]

Wow that's a bad bug! Thanks for catching this.

[simonw]

llm -m claude-3.5-sonnet 'write a poem about this' -a https://static.simonwillison.net/static/2024/pelican.jpg

This is a beautiful shot of a Brown Pelican perched on some rocky shoreline, with boats visible in the background. The pelican's distinctive profile and long beak are nicely highlighted, and there's a lovely rim lighting effect around its head and neck that creates an almost glowing appearance. The bird appears to be in its adult plumage, with the characteristic gray-brown coloring. Brown Pelicans are common along coastal areas, and this one seems right at home in what looks like a marina or harbor setting. The image has nice depth of field, keeping the pelican sharp while softly blurring the boats in the background.

[simonw]

Manual testing of the fix:

% llm -m claude-3.5-sonnet 'extract text from this' -a https://static.simonwillison.net/static/2024/pelican.jpg
There is no text visible in this image. The image shows a brown pelican perched on rocks near what appears to be a marina or harbor, with a boat visible in the background. The pelican is backlit, creating a glowing effect around its head and neck feathers.
% llm -m claude-3.5-sonnet 'write a poem about this' -a https://static.simonwillison.net/static/2024/pelican.jpg
Here's a poem about this pelican:

"Marina Guardian"

Perched upon the rocky shore,
Gray feathers gleaming in morning light,
The pelican stands, watching o'er
The boats that bob in waters bright.

With regal pose and seasoned grace,
Its silhouette against the bay,
Nature's fisher takes its place,
As sentinel through the day.

That ancient beak, so long and strong,
Tells tales of dives and ocean feast,
While backlit plumes, where sunbeams throng,
Create a halo, beauty released.

At harbor's edge, twixt sea and land,
This coastal king maintains its post,
Among the rocks and shifting sand,
A quiet guardian of the coast.
% llm -c 'translate to spanish'
Here's the Spanish translation of the poem:

"Guardián de la Marina"

Posado en la orilla rocosa,
Plumas grises brillando al amanecer,
El pelícano vigila majestuoso
Los barcos que flotan al aparecer.

Con pose real y gracia curtida,
Su silueta contra la bahía,
Pescador natural toma su guarida,
Como centinela durante el día.

Ese pico antiguo, largo y fuerte,
Cuenta historias de buceos y festín marino,
Mientras plumas a contraluz revierte,
Un halo de belleza en su destino.

Al borde del puerto, entre mar y tierra,
Este rey costero mantiene su lugar,
Entre rocas y arena que encierra,
Un guardián silencioso junto al mar.

[mhalle]

Thanks for tracking this down. If you can push out a new version I would be grateful :)

[simonw]

Fix is out in 0.8 https://github.com/simonw/llm-claude-3/releases/tag/0.8