search

hoffmangroup / genomedata

The Genomedata format for storing large-scale functional genomics data.
https://genomedata.hoffmanlab.org/
GNU General Public License v2.0
2 stars 1 forks source link

Consider changing PyTables default number of threads #25

Open EricR86 opened 8 years ago

EricR86 commented 8 years ago

Original report (archived issue) by Coby Viner (Bitbucket: cviner2, GitHub: cviner).

PyTables defaults to using 4 threads. This could pose a problem for cluster usage and could also be made more efficient when more slots are available.

It may be worthwhile for Genomedata to override the default values of tables.parameters.MAX_NUMEXPR_THREADS and tables.parameters.MAX_BLOSC_THREADS.

If not, this behaviour is likely something that Genomedata should explicitly document.

EricR86 commented 7 years ago

Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).

@ericr86 Can you please document this in Genomedata? I don't think there needs to be an option to set this within Genomedata. Client programs can change directly using the tables interface if they need to.

segway-task should also set both of these to one since there is an assumption that one Segway task = one thread. I don't think any other part of Segway or Segtools should be modified, for now. Then you can close this.

EricR86 commented 7 years ago

Original comment by Eric Roberts (Bitbucket: ericr86, GitHub: ericr86).

@michaelmhoffman If there doesn't need to be an option set to this within Genomedata, why document this particular PyTables interface feature? There are plenty of other PyTables parameters that could be documented as well.

EricR86 commented 7 years ago

Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).

Many people will be in an environment where they assume one thread per process unless they do something to change it. And they might violate, say, a cluster use policy to declare the number of threads or cores being used, if they aren't made aware of this.

EricR86 commented 5 years ago

Original comment by Coby Viner (Bitbucket: cviner2, GitHub: cviner).