how to implement a LocalInfileHandler for large file?

[t1hq]

I've been using the following handler for a while, and it works like a charm:

conn.set_local_infile_handler(Some(LocalInfileHandler::new(move |file_name, writer| {
    let file_name_str = String::from_utf8(file_name.to_vec()).unwrap();
    println!("file name: {}", file_name_str);
    let file_content = fs::read_to_string(file_str.clone())?;
    writer.write_all(file_content.as_bytes())
})));

let sql = format!(
    r"LOAD DATA LOCAL INFILE '{}' INTO TABLE `{}`
        FIELDS TERMINATED BY X'01' 
        LINES TERMINATED BY '\n'",
    file, table
);
println!("{}", sql);
conn.query_drop(sql).unwrap();

but recently i noticed some failure(server disconnected or winsock error 10053), when loading some file larger than my ram.

my guess is the program consumes all of the ram, so it broke, so i'm changing the code to utilize buffered reader:

conn.set_local_infile_handler(Some(LocalInfileHandler::new(move |file_name, writer| {
    let file_name_str = String::from_utf8(file_name.to_vec()).unwrap();
    println!("file name: {}", file_name_str);
    let reader = BufReader::new(File::open(&file_str)?);
    let lines = reader.split(b'\n');
    for line in lines {
        let line = line?;
        // println!("line: {}", String::from_utf8(line.to_vec()).unwrap());
        writer.write_all(&line)?;
    }
    writer.flush()
})));

now the program only load the first line of the file to the db table, do I mistake something somewhere?

[blackbeam]

Hi. Could it be the case that your server never encounters the line terminator because BufRead::split removes the split character \n from the end of the line?

[t1hq]

You're right! That saves the day! Appreciate the help