gnocchixyz / gnocchi

Timeseries database
Apache License 2.0
298 stars 84 forks source link

LZ4 decompression error #1348

Open mhajdu opened 8 months ago

mhajdu commented 8 months ago

Hello. We have an openstack virtualization with ceilometer on each host. Gnocchi is running on a separate VM. Almost a week ago we were moving disk of a VM running gnocchi to another storage solution, but everything looked ok from the VM perspective. However we are getting following error in /var/log/gnocchi/gnocchi-metricd.log. I came to conclusion that only memory usage data is missing for all virtual machines, CPU is working ok. We are using gnocchi version 4.4.1 from repository cloud-archive:yoga. I have no idea how to reproduce the problem, but even when I made a new VM and removed all incoming data the errors still persists.

2023-10-17 13:17:43,983 [216632] ERROR gnocchi.chef: Error processing new measures Traceback (most recent call last): File "/usr/lib/python3/dist-packages/gnocchi/chef.py", line 166, in process_new_measures_for_sack self.storage.add_measures_to_metrics({ File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 616, in add_measures_to_metrics ts = carbonara.BoundTimeSerie.unserialize( File "/usr/lib/python3/dist-packages/gnocchi/carbonara.py", line 379, in unserialize uncompressed = lz4.block.decompress(data) _block.LZ4BlockError: Decompression failed: corrupt input or insufficient space in destination buffer. Error code: 6 2023-10-17 13:17:43,988 [216629] ERROR gnocchi.chef: Error processing new measures Traceback (most recent call last): File "/usr/lib/python3/dist-packages/gnocchi/chef.py", line 166, in process_new_measures_for_sack self.storage.add_measures_to_metrics({ File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 616, in add_measures_to_metrics ts = carbonara.BoundTimeSerie.unserialize( File "/usr/lib/python3/dist-packages/gnocchi/carbonara.py", line 379, in unserialize uncompressed = lz4.block.decompress(data) _block.LZ4BlockError: Decompression failed: corrupt input or insufficient space in destination buffer. Error code: 1

Any help would be much appreciated, thank you.

ashishchandra1 commented 4 weeks ago

I am also getting pretty much the same error. Any help here would be appreciated.