Stream txhashset.zip save to disk

[ignopeverell]

Currently the whole txhashset.zip content is read in memory and then written to disk. This is fine now that it's only 65MB but obviously won't when it reaches multiple GB.

[garyyu]

Are you talking about this copy_attachment()? It's split into 8K pieces and not read whole into memory.

    pub fn copy_attachment(&mut self, len: usize, writer: &mut Write) -> Result<(), Error> {
        let mut written = 0;
        while written < len {
            let read_len = cmp::min(8000, len - written);
            let mut buf = vec![0u8; read_len];
            read_exact(&mut self.conn, &mut buf[..], 10000, true)?;
            writer.write_all(&mut buf)?;
            written += read_len;
        }
        Ok(())
    }

Or this BufWriter::new()?

    let mut tmp_zip = BufWriter::new(File::create(file)?);

BufWriter::new is equivalent to a BufWriter::with_capacity with default capacity pub const DEFAULT_BUF_SIZE: usize = 8 * 1024;, so this one should not be a problem.

Or perhaps you're talking about this? in function txhashset_write().

        let mut txhashset =
            txhashset::TxHashSet::open(self.db_root.clone(), self.store.clone(), Some(&header))?;

[ignopeverell]

Looks like I got confused by my own code... I was thinking about the following:

https://github.com/mimblewimble/grin/blob/master/p2p/src/protocol.rs#L241

But that just reads the message, while the txhashset is considered an attachment. So closing this, sorry for the noise.