16K max response size for easy::Easy.transfer.write_function()?

[asanderson]

Using _transfer.writefunction() to do a HTTP GET, it appears the default response is truncated at 16K even if setting _buffersize and/or _maxfilesize. Is there some other way to set the max response size? If not, then consider this a feature request.

[sagebind]

16k is the default buffer size in libcurl. Are you sure your write_function accounts for multiple invocations? libcurl will call your write_function multiple times, each time the response buffer is filled with bytes (or the end of the response is reached). Also, make sure your function returns Ok(data.len()) with the length of the buffer given as an argument, to signal that the entire buffer has been handled. libcurl will abort the transfer if you return a different length than the given buffer, or any error code.

If you're still seeing an issue then it might be a bug in libcurl itself -- we basically just pass in your function to libcurl unaltered (with some type conversions for convenience).

[asanderson]

16k is the default buffer size in libcurl. Are you sure your write_function accounts for multiple invocations? libcurl will call your write_function multiple times, each time the response buffer is filled with bytes (or the end of the response is reached). Also, make sure your function returns Ok(data.len()) with the length of the buffer given as an argument, to signal that the entire buffer has been handled. libcurl will abort the transfer if you return a different length than the given buffer, or any error code.

I'm rewriting my write_funciton() to handle multiple calls, since the docs weren't clear to me there was a limitation.

That said, i did set the buffer_size to 1024k, and it did not change the 16k default buffer size at least for a GET so maybe that's a bug?