Integration with Databricks Lake Monitoring and Marketplace

[cloudronin]

Requirements

1. We need to enable Pythia to be able to used within the Databricks Lakehouse monitoring solution, More information here
   • https://docs.databricks.com/en/lakehouse-monitoring/index.html
   • The goal of this integration would be to use the Lakehouse monitoring tables as the source of the LLM requests/reponses (instead of jaeger) and then create a new Custom monitor metric table (for all the Custom metrics that Pythia produces) as defined here
   • https://docs.databricks.com/en/lakehouse-monitoring/custom-metrics.html

2.For Pythia's LLM we should use a Databricks hosted LLM for our purposes and not OpenAI

• https://docs.databricks.com/en/large-language-models/llm-serving-intro.html
• Maybe we can use the latest llama model for now as defined in the example above for now
• More information on foundational models available in Databricks here https://docs.databricks.com/en/machine-learning/foundation-models/index.html

3.This should in theory allow users to then create SQL alerts within Databricks for the custom Pythia metrics

• https://docs.databricks.com/en/lakehouse-monitoring/monitor-alerts.html

4.Finally, we also need to package this solution so it can be installed from the marketplace and also build an example notebook that can demonstrate how to use it like this listing:

• https://marketplace.databricks.com/details/a4bc6c21-0888-40e1-805e-f4c99dca41e4/Databricks_Llama-Guard-Model
• https://marketplace.databricks.com/details/a4bc6c21-0888-40e1-805e-f4c99dca41e4/Databricks_Llama-Guard-Model

Open Questions

Do we need to use the Inference tables for LLMs being served ?

• https://docs.databricks.com/en/machine-learning/model-serving/inference-tables.html
• https://docs.databricks.com/en/_extras/notebooks/source/monitoring/llm-inference-table-monitor.html

What validators do we need to enable for this effort ?

• Looks like they might have fairness and bias monitoring already
• https://docs.databricks.com/en/lakehouse-monitoring/fairness-bias.html