search

Byron / google-apis-rs

A binding and CLI generator for all Google APIs
http://byron.github.io/google-apis-rs
Other
1.02k stars 136 forks source link

Large responses are truncated. #266

Closed harababurel closed 3 years ago

harababurel commented 3 years ago

I've encountered this bug while trying to integrate google-drive3 v2.0.1+20210322 on harababurel/gcsf.

I'm creating a FileListCall to retrieve many files from Drive. The response I get seems to be truncated to exactly 16384 bytes. Here are some logs showing the truncation.

I've pinned the root cause to client::get_body_as_string which only returns the first chunk in the hyper response:

https://github.com/Byron/google-apis-rs/blob/a3e2835a89876cd6e20a41ecce92a8a4a65cf8cc/gen/drive3/src/client.rs#L812-L816

hyper::body::aggregate returns an impl Buf which also provides remaining(). This can probably be used in conjunction with chunk() to parse the entire buffer.

I'll try to implement the logic and send a PR some time this week.

Byron commented 3 years ago

Thanks for this wonderful issue and analysis on how to fix it! I thought it's interesting that I still remember reading this .chunk() part thinking: "Aaaah, first you aggregate the body bytes which can be multiple chunks into a single chunk, makes sense" without reading docs or further questioning it, and being full aware that it would be an error prone API to have something aggregate Bytes into Bytes which then magically appear to have one chunk only.

Writing this out hopefully helps to me to listen to these glimpses of intuition next time and catch a bug, or two :D.

dvaerum commented 3 years ago

@harababurel Thanks for fixing and explaining my mistake, I had problems figuring out which one to use aggregate or to_bytes and I clearly misunderstood the documentation :sweat_smile: