search

linkedin / dynamometer

A tool for scale and performance testing of HDFS with a specific focus on the NameNode.
BSD 2-Clause "Simplified" License
131 stars 34 forks source link

Closes #16. Use a single shared queue between all threads in each workload mapper. #19

Closed xkrogen closed 6 years ago

xkrogen commented 6 years ago

On a single mapper in the workload replay, currently tasks are distributed to threads in a path-partitioned manner. This results in bad skew issues, where one thread can fall very far behind because it is overloaded. Thus, although all of the issues on a specific path get executed in order (though late), the ordering with other paths is very far out of sync. It is better to let some operations on the same path occur out of order so that (a) overall everything stays more in sync and (b) no thread falls far behind (which can delay the entire mapper from completing).

chliang71 commented 6 years ago

One question, why is this IP based partitioning? To me, it looks like every command gets put into the same queue, and the threads pick up the commands regardless of IP. Am I missing something?

xkrogen commented 6 years ago

Sorry, the commit message was a bit incorrect. I have updated the title & the description to more fully explain the issue we are solving here.

chliang71 commented 6 years ago

LGTM, +1