graphite-project / carbonate

Utilities for managing graphite clusters
MIT License
516 stars 80 forks source link

Traceback on carbon-sync #7

Closed obfuscurity closed 10 years ago

obfuscurity commented 10 years ago

Seeing this traceback on a specific whisper file. I can make the file available if it helps with diagnosis.

Traceback (most recent call last):
  File "/usr/local/bin/carbon-sync", line 9, in <module>
    load_entry_point('carbonate==0.2.0', 'console_scripts', 'carbon-sync')()
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 190, in main
    run_batch(metrics_to_sync, remote, args.storage_dir)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 119, in run_batch
    merge_elapsed = sync_batch(metrics_to_heal)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 60, in sync_batch
    heal_metric(staging, local)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/sync.py", line 75, in heal_metric
    fill_archives(source, dest, time())
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/fill.py", line 85, in fill_archives
    fill(src, dst, gapstart - step, start)
  File "/usr/local/lib/python2.7/dist-packages/carbonate-0.2.0-py2.7.egg/carbonate/fill.py", line 50, in fill
    (timeInfo, values) = fetch(src, fromTime, untilTime)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 711, in fetch
    return file_fetch(fh, fromTime, untilTime)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 786, in file_fetch
    unpackedSeries = struct.unpack(seriesFormat, seriesString)
struct.error: unpack requires a string argument of length 1524
supine commented 10 years ago

@obfuscurity, do you still have this issue?

We've seen a similar error from whisper-dump.py when the wsp file has been truncated by a full disk.

There also appears to be an open ticket in the graphite project with identical symptoms https://github.com/graphite-project/whisper/issues/52#issuecomment-29353913

jssjr commented 10 years ago

I'm going to close this out since it seems to be a bug in whisper, not carbonate.