ISSUE-64. Hadoop 3.0 NN drops replicas silently.

linkedin / dynamometer

A tool for scale and performance testing of HDFS with a specific focus on the NameNode.

BSD 2-Clause "Simplified" License

131 stars 34 forks source link

ISSUE-64. Hadoop 3.0 NN drops replicas silently. #65

Open jojochuang opened 5 years ago

jojochuang commented 5 years ago

Fixes #64 This PR changes blockMap of dyno DN from hash map to tree map. The fix was verified internally at Cloudera. Additional stress test will be needed to understand the performance impact of this change. But it is functionally correct for Hadoop 3.0/CDH5.7.

xkrogen commented 5 years ago

Great find, thanks @jojochuang ! Sorry for the delay in my response.

Two questions:

Why is the comparator a Comparator<Object> instead of Comparator<Block>?
Will this have any performance implications (on the NameNode) on pre-3.0 versions? I don't want this change to accidentally affect the performance of versions which do not have HDFS-9260. I suspect it will not, but would like your opinion before merging.

jojochuang commented 5 years ago

Hi Eric, Apologies -- my github communication goes into my personal email account and I didn't see this one.

Why is the comparator a Comparator