Kimahriman / hdfs-native

Apache License 2.0
33 stars 12 forks source link

Panics reading erasure coded files #109

Closed Kimahriman closed 5 months ago

Kimahriman commented 5 months ago

https://github.com/Kimahriman/hdfs-native/blob/master/rust/src/hdfs/block_reader.rs#L231 can panic sometimes while reading erasure coded files. I think this is because we aren't always finishing the read from an individual block, and the receiver just gets dropped early before all messages have been consumed. This also means that the datanode connection does not get released back into the cache. Need to make sure the ReplicatedBlockStream gets fully consumed for all reads