search

snuspl / cruise

Cruise: A Distributed Machine Learning Framework with Automatic System Configuration
Apache License 2.0
26 stars 2 forks source link

[MINOR] Consider hyper-thread when determining the number of trainer threads #1241

Closed yunseong closed 7 years ago

yunseong commented 7 years ago

In the applications that we are using for evaluation, the number of trainer threads are determined by the available processors that JVM gives via Runtime.getRuntime().availableProcessors(). However, this method counts the number of virtual cores as well, which is known for bad performance especially in the floating point operations.

To address this problem, this PR adds HyperThreadEnabled parameter, with which we can determine the number of training threads toward a better performance.

(Note) A cleaner solution is to disable the HT feature in the OS, but in the virtual environment like AWS EC2, the configuration is not easy to tune. So I ended up with the working solution as in this PR.

yunseong commented 7 years ago

@wynot12 @hjp615 Could one of you have a look at the PR? Thanks!

wynot12 commented 7 years ago

Hmm.. we may try to disable HT, following the article. How do you think? https://aws.amazon.com/ko/blogs/compute/disabling-intel-hyper-threading-technology-on-amazon-linux/

yunseong commented 7 years ago

@wynot12 I tried it, but failed: The ubuntu version is different from his, so the method was not feasible in our instances. Even worse, EC2 VMs are managed by hypervisor and modifying CPU management could bring unexpected results I think.