EOF error parsing large JSON response

[javiorfo]

Hello, I have noticed a problem trying to deserialize a large JSON object.

In this simple example the "serde line" results in a random error because EOF I also tried to write the data bytes in a file and effectively, the "json string" was truncated. This problem occurs with large json responses (I tried short ones and everything works well). Using crate reqwest for example, the response is right. Using the standard curl cli everything is fine too. Apparently is a problem of this library

This is my example with the problematic endpoint

fn main() {
    let mut easy = Easy::new();
    easy.url("https://start.spring.io").unwrap();
    easy.accept_encoding("application/json").unwrap();
    easy.write_function(|data| {
        let value: Value = serde_json::from_slice(data).unwrap(); 
        println!("{}", value);
        stdout().write_all(data).unwrap();
        Ok(data.len())
    })
    .unwrap();
    easy.perform().unwrap();
}

Thanks!!

[ehuss]

The write_function will be called repeatedly as data is received over the wire. In your example, you are feeding a partial JSON blob to serde_json which will fail since it hasn't received the entire output. Depending on what you need to do, you'll either need to buffer the results in memory, or stream them to some other place (like a file, stdout, etc.). The write_function docs go into more detail.

[javiorfo]

Thanks, @ehuss You was right about it. My mistake!

Just in case someone has the same problem a leave my solution for this problem:

fn main() {
    let buf = RefCell::new(Vec::new());
    let mut handle = Easy::new();
    handle.url("https://start.spring.io").unwrap();
    handle.accept_encoding("application/json").unwrap();

    let mut transfer = handle.transfer();
    transfer.write_function(|data| {
        buf.borrow_mut().extend_from_slice(data);
        Ok(data.len())
    }).unwrap();
    transfer.perform().unwrap();

    let borrowed = buf.borrow();
    let value: Value = serde_json::from_slice(borrowed.as_slice()).unwrap();
    println!("{:?}", value);
}

Thanks!