Deploy logfire for observability

The team behind Pydantic has made a new observability solution, logfire, and I think this would be great for us to try out. It offers both logging and metrics, so we can both view logs from Mainframe and also metrics such as packages ingested, packages failed, malicious packages, etc... It also allows us to view SQL queries and their durations and set up alerts.

Some features Logfire provides that I think would be of particular use to us:

Alerts for particular events: Logfire supports sending webhooks in Slack format, Discord also accepts Slack-formatted webhooks. See here
Metrics: Allows us to collect metrics in a variety of forms. For instance, we can collect packages ingested and display a time series graph of package ingest to determine period of high package ingest.
- We can also instrument FastAPI this way, so we can collect information on things like average request duration (and also plot this on a time series graph)
We can collect logs, and view them in Logfire's dashboards.
We can add all the core developers and give them access to logs and metrics.
Logfire integrates with pretty much everything we use: FastAPI, SQLAlchemy/Psycopg, Structlog, Pydantic, and HTTPX.

Compared to similar products like Prometheu and Loki, Logfire provides an all-in-one solution that focuses on simplicity, rather than customization options. It has all the features we need for observability, and will allow us to get it set up and maintained easily.

One potential point of contention is that their dashboard is proprietary, and the way it's marketed implies that there are very little restrictions and is completely free, but only because it's in beta, and that the Logfire team may start adding restrictions and move to a paid model. While I could argue that it may be something that's worth paying for, because Logfire also integrates with OpenTelemetry, we can use other tools and dashboards in the future if we need to.

vipyrsec / dragonfly-mainframe

Deploy logfire for observability #250