[RFC] Log aggregation + making logs readable and searchable

[apedroferreira]

What's the problem? 🤔

This issue was derived from https://github.com/mui/mui-toolpad/issues/999 because I've decided to split that issue. I'm moving the other part of the discussion here.

Basic request and response logging were added to Toolpad in https://github.com/mui/mui-toolpad/pull/1066, but right now these logs will only be visible for a short time span in Render.com, and they're not searchable or filterable, for example. They will probably be quite difficult to read, too.

We should decide on how we want to aggregate and/or store these logs so that we can easily work with them to have good visibility into how users are interacting with Toolpad, or other problems that may occur.

What are the requirements? ❓

Aggregate logs in a way that they can be searched through, they're easy to read and they are stored for an acceptable amount of time.

What are our options? 💡

Like I said in the issue this was derived from, there are third party log aggregation services such as Datadog, New Relic and Grafana Loki. We can also maybe create our own simple manual setup but I am not very familiar with that.

I'm looking to get feedback into this decision or on whether there are other ways we should tackle this problem, or if this issue is not even a problem for now.

Proposed solution 🟢

None yet, I'm open to any feedback. We can use Render's Log Streams feature though: https://render.com/docs/log-streams

Resources and benchmarks 🔗

No response

[Janpot]

For me the choice would be between

• ELK stack (either on elastic.co or on AWS) and ship logs from within the application
• Datadog and ship logs from render.com https://render.com/docs/log-streams

[apedroferreira]

I'll give ELK a try since it seems like a very solid long term solution, and move on with it if I see there's no major obstacles. Otherwise we can go with Datadog since it should be very simple and quick to implement.