search

hegelai / prompttools

Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
http://prompttools.readthedocs.io
Apache License 2.0
2.56k stars 216 forks source link

Stable diffusion support #74

Closed steventkrawczyk closed 10 months ago

steventkrawczyk commented 11 months ago

🚀 The feature

We should look into experiments and eval functions for image models, like stable diffusion

Motivation, pitch

Many GenAI apps are multi modal, and we'll have to support more than LLMs in the long term

Alternatives

No response

Additional context

No response

HashemAlsaket commented 11 months ago

Super cool. @steventkrawczyk , @NivekT mind assigning me? Current thought process is to build the experiment to measure against basic metrics like Peak signal-to-noise ratio, Structural Similarity Index, Signal to reconstruction error ratio. I think we may need to carve out a separate experiment class for image comparison (possibly a third for video comparison as well?) since the current experiment class is built to handle text with heavy support for dataframes.

NivekT commented 11 months ago

Sure, feel free to start on that. If anyone else has a different idea, please feel free to comment here as well.

Thanks @HashemAlsaket !