search

go-graphite / go-carbon

Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister
MIT License
801 stars 126 forks source link

[FR] Online migration for stale files #525

Open Knud3 opened 1 year ago

Knud3 commented 1 year ago

Go-Carbon have fantastic new feature of online migration #438, but it wont affect on stale files. It is problematic in Grafana panels which now include metrics from old and new storage schemas.

It would be beneficial to have some way to trigger migration to all files in new schema.

Maybe cache a copy of storage-schemas.conf and storage-aggregation.conf and when go-carbon process starts it checks diffs to running ones.

Originally posted in https://github.com/go-graphite/go-carbon/issues/518#issuecomment-1403599698 I also did investigation where problem might be with metrics. It seems that metrics which have had writes is migrated to 1m:14d,1h:1y as you said, but Grafana panels also include metrics which haven't had writes (even out of selected time range), so it only shows 5m:1d,1h:1y.

metrics

deniszh commented 1 year ago

Hi @Knud3

Could you please elaborate what's stale files in your descriptions? FIles which stop getting updates? I'm not aware that current code not migrating them...

Knud3 commented 1 year ago

Yes. Files which are not getting updates no more.

Only files which are going through persister get updated.

online migrations are triggered when there are updates on the metric/whisper files. If there are no write to the whisper files, then migrations are not triggered. Originally posted by @bom-d-van in https://github.com/go-graphite/go-carbon/issues/518#issuecomment-1401344492

you can also manually migrate the schema of those stale whisper files with whisper-resize.py. Originally posted by @bom-d-van in https://github.com/go-graphite/go-carbon/issues/518#issuecomment-1403634907

deniszh commented 1 year ago

Hi @Knud3

Oh, I didn't know that. That could be fixed, indeed