Using flamegraph, I discovered that most of the read/write time was spent doing CRC calculations. Using the new beta of the crc crate and it's slice16 support, this greatly increases the CRC throughput which in turn greatly speeds up the read and write throughput. The read throughput is now equal to or slightly higher than the libhdfs method. The write is still a little behind, need to look into that more.
Using flamegraph, I discovered that most of the read/write time was spent doing CRC calculations. Using the new beta of the
crccrate and it'sslice16support, this greatly increases the CRC throughput which in turn greatly speeds up the read and write throughput. The read throughput is now equal to or slightly higher than the libhdfs method. The write is still a little behind, need to look into that more.