search

humio / issues

Issue Tracker for Humio
4 stars 2 forks source link

Feature Request - signing of datablocks on-disk #21

Open henrikjohansen opened 6 years ago

henrikjohansen commented 6 years ago

In order to detect manipulation of log data on-disk Humio should really have the ability to checksum the blocks that are written on-disk and to compare this against a known good checksum for those data block.

This would also yield some capabilities for detection storage error conditions such as bit rot, phantom writes / reads, etc which is important for long-term storage of log data.

krestenkrab commented 6 years ago

We're working on CRC32C for all data on-disk. It should arrive soon.

mortengrouleff commented 5 years ago

The segment files now have multiple CRC32C checksums inside. For this feature we need to store a checksum of those in global for each segment. And perhaps an option to enable doing a stronger checksum (e.g. sha512) on the completed segment file and store that in global and maybe also publish the (id, sha) pairs to some stream external to Humio.