Add platform-level logging

[veekaybee]

Right now, we use logguru piecemeal, we need to have a discussion about a larger platfrom logging strategy, keeping in mind that we'll get logs from multiple services, including Ray which implements its own logging scheme. I'm recommending something like this, where we log at the app level via Uvicorn and have a log handler at the top-level of the project: https://github.com/tiangolo/fastapi/discussions/7457#discussioncomment-5141102

[binaryaaron]

@macaab26 / @chainlink - outside of sorting out an external logging host, datadog or whatever, wdyt about near-term logging?

My suggestion was going to be something akin to the linked approach - set up file handlers and we can at least start making basic logfiles and store them in S3 or something. https://github.com/tiangolo/fastapi/discussions/7457#discussioncomment-8926937 has more config info that might be useful there.

[macaab26]

@binaryaaron agree, that's a good approach. I have a story for this sprint that is "almost" related to this. I want to create a dashboard in OpenSearch, as a first attempt (I'm starting with OpenSearch because I want to do a couple fancy things for the MVP, and it's more cool and visual than CloudWatch, but it's another option too). In the long term, it will be better to have another kind of solution, probably one where we could have everything together (logs, metrics, traces, etc).