search

Hurence / historian

big data timeseries historian
Apache License 2.0
4 stars 1 forks source link

improve compactor #195

Open oalam opened 3 years ago

oalam commented 3 years ago

for now we've got one compactor job with one conf


  reader:
      queryFilters: "chunk_origin:prometheus-scraper"

  chunkyfier:
      saxAlphabetSize: 7
      saxStringLength: 24
      origin: "compactor"
      dateBucketFormat: "yyyy-MM-dd.HH"

  scheduler:
      period: 10
      startNow: true

we must be able to add multiples conf in order to handle multiples workloads of compaction.

for example one metric could be done hourly for the current day and daily for the previous day

compactions:
    - id: hourly_scrapping_compaction
      reader:
          queryFilters: "chunk_origin:prometheus-scraper"
      chunkyfier:
          saxAlphabetSize: 7
          saxStringLength: 30
          origin: "prometheus-scraper-compacted-hourly"
          dateBucketFormat: "yyyy-MM-dd.HH"
      scheduler:
          period: 10
          startNow: true
    - id: daily_scrapping_compaction
      reader:
          queryFilters: "chunk_origin:prometheus-scraper-compacted-hourly"
      chunkyfier:
          saxAlphabetSize: 7
          saxStringLength: 24
          origin: "prometheus-scraper-compacted-daily"
          dateBucketFormat: "yyyy-MM-dd"
      scheduler:
          period: 86400
          startNow: true