Try Intel Distribution for Python?

[1fish2]

It uses Intel's Math Kernel Library and Threading Building Blocks.

Is it compatible with Sherlock? With Sherlock 2.0?

[1fish2]

It's no longer necessary to create an Intel dev account, download the Intel Distribution for Python, rationalize its packages with ours, nor use conda to install their packages and figure out how to add our packages and how conda coexists with pip.

• Just create a new pyenv virtualenv (or pip uninstall numpy scipy scikit-learn -y), do pip install intel-numpy intel-scipy tbb4py, then pip install our other packages. I drafted intel-requirements.txt instructions to build this virtualenv.
• summarize_environment.py reports that numpy and scipy are linked to 'libraries = ['mkl_rt', 'pthread']`.
• Switching between this new virtualenv and wcEcoli2 works fine. Maybe run make clean compile after switching.
• I haven't tried it on Sherlock yet but it's "just" some pip installs. What could go wrong?
• With it, one can run python -m tbb [-h] [-p P] [-b] [-v] [-m] script ... to coordinate CPU usage across thread and processes to avoid slowdown due to oversubscription contention for cores.
• It doesn't require Python code changes but it offers numpy.random_intel which is supposed to be a drop-in replacement for numpy.random, presumably faster, and offering additional Intel MKL sampling algorithms (whatever those are).
• But so far, it doesn't seem to run faster.