furlat
commented
2 months ago Absolutely interesting! Especially to track LLM structured generations, and very interesting in the case where the llm can generate its own schemas.
Open Kludex opened 2 months ago
furlat
commented
2 months ago Absolutely interesting! Especially to track LLM structured generations, and very interesting in the case where the llm can generate its own schemas.
We want to build a catalog of Pydantic Models/Schemas as outlined in our Roadmap article with in Logfire.
The idea is that we'd use the SDK to upload the schema of Pydantic models to Logfire. Then allow you to watch how those schemas change, as well as view metrics on how validation performed by a specific model is behaving.
Please let us know if you are interested in this, so we can prioritize it.