Expose bin/distmap hardcoded number_of_processors? - Githubissues

robmaz / distmap

Sequence alignment on Hadoop

0 stars 1 forks source link

Expose bin/distmap hardcoded number_of_processors? #81

Open magicDGS opened 6 years ago

magicDGS commented 6 years ago

In distmap, there is a variable called $number_of_processors, which is explained as "This number need to define. How many processors Hadoop can use on each nodes". A quick search finds that it is used for some mapper scripts to set how many processors the node would use (e.g., number of threads param in bwa).

I have two suggestions for this to work in an user-friendly way:

Expose as an "advance argument" and default to some constant
Grab this information from somewhere in the hadoop configuration

Other suggestions, @robmaz?

robmaz commented 6 years ago

I guess this number can reasonably depend on both the mapper and the hadoop configuration. I do not think it should be exposed to the user. Currently, it is only set for each mapper type, not taking into account possible hadoop constraints. But probably this does not matter - if the job requests more resources than it can get, it will just get less than requested.