Reorganize directories - Githubissues

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.

https://quickwit.io

Other

7.75k stars 310 forks source link

Reorganize directories #3859

Open fulmicoton opened 11 months ago

fulmicoton commented 11 months ago

We could organize by role

ingester
indexer
searcher

Durability nature

temporary data
caches
storages

guilload commented 4 months ago

The ephemeral vs persistent dimension is interesting. For instance, you could mount a local SSD on qwdata/ephemeral to serialize splits and cache them for merges, but mount qw/persistent on an EBS to store the WAL.

Maybe this is what we want: qwdata/{ephemeral,persistent}/<role>/<purpose>:

qwdata/
    ephemeral/
        indexer/
        searcher/
    persistent/
        ingester/
            wal/
        metastore/
            indexes/
            manifest.json