Infra lead feedback on data warehousing

[bedeho]

Background

We are adding this https://github.com/Joystream/atlas/issues/3980 capability to Orion, and Orion needs some way to relay anonymized but useful summaries of this data to lead data warehousing services, both for distributors and storage providers. I think some useful requirements for these warehousing solutions are

1. can be self-hosted, does not depend on any particular service provider
2. does not require changing in any way when new data formats arrive
3. has rich suite of tools for visualizations, public dashboards, report generation, alert generation, etc.
4. is open source

Question

Obviously, we one can write a whole new solution from scratch, but that seems like a very expensive paths, so can current leads and others provide some feedback on what sort of systems they would want to use for this, and for each describe to what extent they satisfy 1-4, or any other important requirements we may want to keep in mind.

[kdembler]

I would recommend something like InfluxDB or TimescaleDB (PostgreSQL), I will use one of those for DWG monitoring

[bedeho]

• I think def. some coordination with @attemka is needed here, because he is building this.
• Why not just use ELK stack? we get so much tooling, dashboarding etc. out of the box, and its already being or going to be used other places, so limiting diversity of different setups could be a benefit as well. I am not familiar with those two, but from my quick googling they seem like databases, so a very low level of abstraction - which presumably means a lot of other stuff will need to be built to be good solutions. Do theye provide standard APIs for for adding data, is there dashboard tooling comparable to Kibana, are there plugins for making notifications that are triggrred by queries which end up delivering emails and Discord messagess, etc.