fulcrumgenomics / dagr

A scala based DSL and framework for writing and executing bioinformatics pipelines as Directed Acyclic GRaphs
MIT License
69 stars 14 forks source link

Why does Dagr use the physical processor count instead of logical for Macs only? #370

Closed clintval closed 3 years ago

clintval commented 4 years ago

On my MacBook Pro with hyper-threading enabled I have 8 physical cores yet 16 addressable cores.

❯ sysctl -n hw.physicalcpu
8
❯ sysctl -n hw.logicalcpu
16

https://github.com/fulcrumgenomics/dagr/blob/e2a634a5054d4a2200aa360ea33843280b2b0b51/core/src/main/scala/dagr/core/execsystem/Resource.scala#L33-L38

I may be under-utilizing my local compute resources if capped to 8 cores instead of the full 16. Here's a screenshot of when I ask Dagr to run a task with the maximum allowed system cores:

Screen Shot 2020-01-05 at 5 21 24 PM

Since hyper-threading is not unique to Macs, I am wondering why this design decision was made.

nh13 commented 4 years ago

Pinging @tfenne if he can remember as I don't. You can override this with --cores 16 in the short-term.