go-graphite / buckytools

Go implementation of useful tools for dealing with Graphite's Whisper DBs and Carbon hashing
Other
19 stars 8 forks source link

Add modify command #4

Closed bom-d-van closed 6 years ago

bom-d-van commented 6 years ago

Modify command supports two operations: resize, or update aggregation policy.

Resize mode allows user to resize one archive at a time. It only change the targeting archive and does not affect other archives. Use -index to specify resized archives.

Use -retention to specify new policy (with the same format in whisper configuration). To resize to bigger time range, modify command upsample data from lower-resolution archives.

Example: bucky modify -index 1 -retention 1m:30d -f 100_olddata.wsp

Change aggregation policy. Other than changing policy, this command would also try to correct data if it's changing policy from average -> sum, or sum -> average. For other types of changes, it would only do a simple data copy.

Example: bucky modify -f small.wsp.new -agg average

By default, both tool would copy the original whisper file as a new back in the same location.

Note: the key difference between modify resize and whisper-resize.py is that modify command would manipulate the archives directly instead of through the update_many/UpdateMany call in whipser library. So it doesn't propagate to lower archives.

Civil commented 6 years ago

I'll check it on the weekend.

As about whisper library - I wanted to basically go over the difference between go-graphite/whisper and vendored one and to port whatever make sense to go-graphite/whisper. And after that to migrate to ours.

bom-d-van commented 6 years ago

@deniszh @Civil thanks for the quick reply. we are still working on some more changes. Will post it here again when I push all the changes. Sorry for this early pull request.

Civil commented 6 years ago

Please remove [WIP] From title and poke me and @deniszh once you are ok with the PR.

bom-d-van commented 6 years ago

Hi @Civil @deniszh , sorry for my long overdue final touch. I have finally managed to finish it. Also added some docs. Could you take a look whenever you have time?

Civil commented 6 years ago

LGTM

Sorry it took some time for me to review it.