Open victor987 opened 4 years ago
@victor987 thank you for bringing this to our attention. I just looked at this and the problem may very well be worse. Regardless of what I try, I can only get bloscpack
to run in single threaded mode. My guess is that this is an issue with the underlying python-blosc
but more triage will be needed.
My single-threaded observations may be OSX only. I just tried on a Linux server with 8 cores and can use --nthreads
w/o issues.
Oh, wait it seems like the default number of threads is now 8(?) and I am unable to change that to anything else. Always 8 threads will be used.
It seems it's not detecting/using the available CPU cores.
Testing on Ubuntu server 19.04 on quadruple AMD Opteron 6282 SE, for a total of 64 cores on the system. I'm using incompresible random data for this example but the behaviour is the same with real data.
Activity during compression shows that 4 cores out of 64 are used:
Specifying 64 threads does not change the behaviour:
Activity during compression shows that 4 cores out of 64 are used: