We have some files which are gzip encoded at rest in GCS. Thanks to Decompressive transcoding, when getting metadata about a file the compressed byte count will be returned, but when the file itself is retrieved it will be uncompressed and so the total byte count will be higher.
This interacts poorly with this PR: https://github.com/apache/opendal/pull/4690 where the two values mentioned above are compared and we keep hitting reader got too much data.
Steps to Reproduce
I'm working on providing an actual repro but running into difficulties with permissions.
Have a file gzip encoded on GCS, with object metadata Content-Encoding set to gzip.
Now, build an operator of type opendal::types::operator::operator::Operator and call operator.read()
What currently happens is an opendal::Error as the Result that prints to:
Unexpected (permanent) at => reader got too much data
Context:
expect:
actual:
### Expected Behavior
An `Ok` `Result` with the contents of the file.
### Additional Context
_No response_
### Are you willing to submit a PR to fix this bug?
- [ ] Yes, I would like to submit a PR.
Describe the bug
We have some files which are gzip encoded at rest in GCS. Thanks to Decompressive transcoding, when getting metadata about a file the compressed byte count will be returned, but when the file itself is retrieved it will be uncompressed and so the total byte count will be higher.
This interacts poorly with this PR: https://github.com/apache/opendal/pull/4690 where the two values mentioned above are compared and we keep hitting
reader got too much data
.Steps to Reproduce
I'm working on providing an actual repro but running into difficulties with permissions.
Content-Encoding
set togzip
.operator
of typeopendal::types::operator::operator::Operator
and calloperator.read()
opendal::Error
as theResult
that prints to:Context:
expect:
actual: