search

wisecubeai / pythia

Open source AI hallucination monitoring
https://askpythia.ai/
Apache License 2.0
4 stars 0 forks source link

End to end test to validate functioning of Pythia all-in-one stack #20

Open cloudronin opened 6 hours ago

cloudronin commented 6 hours ago

We need to create an end to end test that can ensure that the following flow works

  1. Client App making LLM calls instrumented and sending traces
  2. Traces are saved in Jaeger properly with the right service name
  3. Pythia is able to see the new incoming traces (once an only once)
  4. Pythia is able to run the configured validators (and only those) on the incoming traces
  5. Pythia is able to create metrics from the incoming traces
  6. Pythia is able to write the new metrics into Prometheus
  7. Grafana is able to read the Prometheus metrics