search

evidentlyai / evidently

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
https://www.evidentlyai.com/evidently-oss
Apache License 2.0
5.46k stars 603 forks source link

Add is valid sql descriptor #1332

Open Sifr-un opened 1 month ago

Sifr-un commented 1 month ago

Implementing a SQL validator for LLM responses as a new descriptor. Resolves #1321

DimaAmega commented 1 month ago

Hi @Rayryu

You have some CI failed jobs, please check it

Sifr-un commented 1 month ago

DimaAmega

I've pushed a typo fix.

Sifr-un commented 1 month ago

I see that the minimal requirements CI step has failed. I've now added sqlvalidatorto the requirements file 👌

mike0sv commented 1 month ago

Hi! To fix mypy you need to add ignore block to setup.cfg in repo root. You can check out mypy section for examples

Sifr-un commented 1 month ago

Hi! To fix mypy you need to add ignore block to setup.cfg in repo root. You can check out mypy section for examples

Thank you @mike0sv for chiming in. I've added a sqlvalidatorblock in setup.cfg.

emeli-dral commented 1 month ago

Hi @Rayryu,

It looks like we're almost there!

Could you please add the new descriptor to the list of descriptors under the TextEvals section on the documentation page? The file that needs to be updated is located here: all-metrics.md#text-evals.

Once that’s done, we’ll be all set to proceed with the merge!

emeli-dral commented 1 month ago

Hi @Sifr-un , do you need some help with adding IsValidSQL to the list of descriptors under the TextEvals section all-metrics.md#text-evals?

vectorvp commented 1 week ago

@emeli-dral, how can I help to speed this up? Merge the changes from this branch and prepare new PR?