search

ASU-cubesat / cfdp-rs

A rust implementation of the CCSDS File Delivery Protocol (CFDP)
MIT License
9 stars 2 forks source link

Slow checksum computation #12

Closed xpromache closed 1 year ago

xpromache commented 1 year ago

Experimenting with larger (~200MB) file transfers I discovered that the checksum computation is very slow (~90 seconds for a 200MB file on a Ubuntu 20 with a relatively fast SSD). This will cause the inactivity timer to be triggered.

I think the reason is that the checksum function reads the file 4 bytes at a time: https://github.com/ASU-cubesat/cfdp-rs/blob/main/cfdp-core/src/filestore.rs#L421

I think instead of the 4 bytes vector, it should use a vector of 1 or 2KB.

mkolopanis commented 1 year ago

I think I was more worried about making sure everything fit on the four byte boundary than being smart the first time. I agree it should probably read in more data and just iterate over a larger vector between each read. A few KiB is probably a good start.

mkolopanis commented 1 year ago

I'm doing some benchmarks on a implementation with a BufReader. Seems to give ~1000x speedup according to the rust bencher on a 5MB file on my laptop.

BufReader defaults to 8KB buffer currently.