search

equinor / sumo-tableaggregator

Aggregation service for sumo. To be used with table data indexed with sumo
Apache License 2.0
0 stars 0 forks source link

Investigate Azure Data Explorer for handling tables #48

Open equinor-ruaj opened 11 months ago

equinor-ruaj commented 11 months ago

From a random conversation at EDC - Azure Data Explorer is used by some projects in EPI to handle large tables. The specific example was well logs, where they jam all logs into one huge table, and uses native functionality to query individual columns of it.

This could potentially be relevant for us (the large table use case). Nevertheless, since others are using it, it is probably good to know a little bit about it.

Suggest to do a time-boxed exploration of this topic, to check if this is something that could be useful - or not.

This flow chart perhaps gives a quick answer: https://learn.microsoft.com/en-us/azure/data-explorer/data-explorer-overview#when-should-you-use-azure-data-explorer

equinor-ruaj commented 11 months ago

Start by talking to the person? (Data fundamentals, Eirik Øverland?)

Timebox: 1 week (5d)

Outcome:

perolavsvendsen commented 11 months ago

Fo.tone team (Kjell Erik Anda et.al.) is using this heavily, and can be asked. Also, Data Fundamentals is using it for somewhat lighter use cases (?). They recommend talking to fo.tone first.

roywilly commented 9 months ago

A very quick look at documentation only, no hands-on usage:

Azure Data Explorer vs ElasticSearch: https://www.influxdata.com/comparison/azure-data-explorer-vs-elasticsearch/ https://db-engines.com/en/system/Elasticsearch%3BMicrosoft+Azure+Data+Explorer

Azure Data Explorer is available in many Azure regions, including Europe North/West, Norway East but not in Norway West. (https://azure.microsoft.com/nb-no/explore/global-infrastructure/products-by-region/)

Azure Data Explorer cost is depending on the chosen engines/hardware strenght and number, number of read operations, number of write operations, number of GB ingested per day, and data retention and so on. As usual it is hard to estimate Azure cost upfront, guess we need to experiment to know more. (https://azure.microsoft.com/en-us/pricing/calculator)

Azure Data Explorer SRA exists and it is hence allowed in Equinor. (https://docs.omnia.equinor.com/governance/security/components/v2/azuredataexplorer/)

My initial impression is that Data Explorer is a big animal and a direct competitor to ElasticSearch, and not a 'small Sumo addon for large tables'.