Handle additional errors when loading corrupted segments

joncrlsn / dque

dque is a fast, embedded, durable queue for Go

MIT License

767 stars 46 forks source link

Closed neilisaac closed 4 years ago

neilisaac commented 4 years ago

corrupted state files containing excess deletion records (zeros) which otherwise results in a panic
wrap gob errors

The former is surprisingly the most common form of corruption we see in production on machines that can be power cycled at will.

joncrlsn commented 4 years ago

Hi Neil, are you using turbo mode when you get those errors? It's not surprising in turbo mode, but I'd be more surprised if it happens in safe mode.

neilisaac commented 4 years ago

@joncrlsn yes, turbo mode was on. I was still surprised that we'd get invalid data; I was just expected partial truncated data.