Open honganan opened 1 month ago
Update:
After skipping the empty chunks in our version, I found that there were still Invalid chunk checksum
errors. I added logging to print the chunk's key and downloaded it. When I inspected it using the chunks-inspect
tool, it reported an error: error while reading metadata bytes: unexpected EOF
. Seems the metadata length resolved is incorrect.
The chunk cannot share here as it is from our production environment, but I can provide any other help to digest the problem(maybe provide the problem chunk privately?).
The error from the chunk-inspect is a consequence of zero byte chunk as ReadFull
method is expected to throw and error https://github.com/grafana/loki/blob/main/cmd/chunks-inspect/header.go#L42 may be we are trying to read the chunks created before your patch.
When you say skipping have you tried to handle the case in both write and read path ?
When you say skipping have you tried to handle the case in both write and read path ?
I only did in read path to make query works.
Describe the bug I have a query cause 'Invalid chunk checksum' error after upgrading write and read path to 3.0.
I added some debug log to print the invalid chunk's key in S3 and found the chunk is a 0 bytes one:
This is a sub case of #8564. But I think this case is explicitly and we can fix it by skipping 0 bytes chunks first. How do you think?
The another work need to do is to digest how the write path created 0 bytes chunks.
To Reproduce Steps to reproduce the behavior:
Expected behavior A clear and concise description of what you expected to happen.
Environment:
Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem.