Turn off jit compiler - Githubissues

1fish2 commented 8 months ago

Fix #1400 by turning off the Numba JIT compiler by default in the manual runscripts, fw_queue, and wcm.py. This avoids spending several minutes JIT-compiling runtime-generated math lambdas in process/metabolism.py and process/two_component_system.py. Skipping the JIT doesn't seem to slow down the simulation! (You can still turn it on via the --jit arg or fw_queue's JIT=1 environment variable.)
Replace np.product() with np.prod() since product() is gone in numpy 1.26. I assume a simple replacement suffices. The docs don't make it easy.
Improve the error handling code in tablewriter Column. runSim hit the OS ulimit (OSError: [Errno 24] Too many open files) on my laptop. This tweak just makes Column's close() method not compound the stack traceback by trying to access self._data which didn't get set because __init__() failed.

Adding ulimit -n 4096 to my bash profile fixed "Too many open files". Should we mention that in the development setup instructions? Has anyone else hit this limit?

Later we'll consider whether to remove the code that uses the Numba JIT compiler. That'll simplify library version dependencies.

@thalassemia discovered that the slow JIT compiler happens with Python 3.10.12, not a Numba release. That's a very helpful clue, and it reveals something about the debugging the Numba team will need to do. They already have open issues reported in 2020 on slow jit compilation and exponential jit compile time. I'll try to extract a test case for them.

rjuenemann commented 8 months ago

I think I hit a similar open files limit on my MacBook the first time I tried to run the model on it or maybe running an analysis script the first time and resolved it similarly. It was a while ago so I don't remember the specifics but it does seem familiar. I never saw it on my Windows machine. It could be worth adding to setup instructions.

1fish2 commented 8 months ago

Thanks for testing Numba jit performance on Python 3.9.17! I've been wondering when it was last a net performance win.

1fish2 commented 7 months ago

https://github.com/1fish2/numba-test is a standalone performance test of Numba compiling and running the 4 expressions that wcEcoli uses Numba for. Just create a pyenv and run python src/time_numba.py.

Timings on my Intel MBP show that on Python 3.10+ JIT-compiling the jacobians does makes them run faster but it's hard to make up for how long the compiler takes (3.2 minutes vs. 11 secs).

# Python 3.8.13
Numba timings on Python 3.8.13 on macOS-13.6-x86_64-i386-64bit
2CS RATES: 0.014721775000000025 ms, jitting 896.6857590000004 ms, jitted 0.005806761699999896 ms
2CS RATES JACOBIAN: 0.08696356209999988 ms, jitting 1,683.092415999999 ms, jitted 0.009799358999999975 ms
EQUILIBRIUM RATES: 0.028278581400000037 ms, jitting 981.5095409999994 ms, jitted 0.006703542400000017 ms
EQUILIBRIUM RATES JACOBIAN: 0.26574201829999994 ms, jitting 7,527.260669 ms, jitted 0.014911214700000031 ms

Numba timings on Python 3.8.13 on macOS-13.6-x86_64-i386-64bit
2CS RATES: 0.014561638300000002 ms, jitting 987.0022309999999 ms, jitted 0.005777516299999985 ms
2CS RATES JACOBIAN: 0.08875619550000002 ms, jitting 1,700.862166 ms, jitted 0.009454348099999966 ms
EQUILIBRIUM RATES: 0.028071367599999952 ms, jitting 964.3987979999995 ms, jitted 0.006520770999999925 ms
EQUILIBRIUM RATES JACOBIAN: 0.26334185759999995 ms, jitting 7,517.086749000001 ms, jitted 0.015204164100000028 ms

# Python 3.9.0
Numba timings on Python 3.9.0 on macOS-10.16-x86_64-i386-64bit
2CS RATES: 0.014656033199999996 ms, jitting 981.5643400000002 ms, jitted 0.005700713100000065 ms
2CS RATES JACOBIAN: 0.08845420760000007 ms, jitting 1,740.2744830000004 ms, jitted 0.009545836299999878 ms
EQUILIBRIUM RATES: 0.029305668399999974 ms, jitting 1,006.9870420000004 ms, jitted 0.006674980100000028 ms
EQUILIBRIUM RATES JACOBIAN: 0.2737112841 ms, jitting 7,512.242000999999 ms, jitted 0.014491421600000009 ms

# Python 3.10
Numba timings on Python 3.10.9 on macOS-13.6-x86_64-i386-64bit
2CS RATES: 0.015168395300861448 ms, jitting 866.4746490539983 ms, jitted 0.005704399896785617 ms
2CS RATES JACOBIAN: 0.10583787140203639 ms, jitting 22,942.427759000566 ms, jitted 0.02641847580089234 ms
EQUILIBRIUM RATES: 0.030743514501955362 ms, jitting 1,470.0019049923867 ms, jitted 0.007768389803823084 ms
EQUILIBRIUM RATES JACOBIAN: 0.32779657070059326 ms, jitting 165,673.03289601114 ms, jitted 0.06098547090077773 ms

# Python 3.11.6
Python 3.11.6 on macOS-13.6-x86_64-i386-64bit
2CS RATES: 0.013978633302031085 ms, jitting 808.7223370093852 ms, jitted 0.0059150672983378175 ms
2CS RATES JACOBIAN: 0.09359739659703337 ms, jitting 21,632.29699898511 ms, jitted 0.026892004802357403 ms
EQUILIBRIUM RATES: 0.02840175310266204 ms, jitting 1,358.3953219931573 ms, jitted 0.008712695701979101 ms
EQUILIBRIUM RATES JACOBIAN: 0.3119376969989389 ms, jitting 167,880.91464200988 ms, jitted 0.06526484609930777 ms

Numba timings on Python 3.11.6 on macOS-13.6-x86_64-i386-64bit
2CS RATES: 0.013602064299630002 ms, jitting 698.4790170099586 ms, jitted 0.005492584296735004 ms
2CS RATES JACOBIAN: 0.09405390089959838 ms, jitting 20,596.083651995286 ms, jitted 0.024108504399191585 ms
EQUILIBRIUM RATES: 0.0280356386967469 ms, jitting 1,272.7877250290476 ms, jitted 0.006036852503893897 ms
EQUILIBRIUM RATES JACOBIAN: 0.3077152126992587 ms, jitting 166,229.16107898345 ms, jitted 0.063528852496529 ms

The Numba pip won't install on Python 3.12.

1fish2 commented 7 months ago

To get a measure of JIT compilation time payoffs, the test now calculates how many times each of those functions must be called to pay for its JIT compilation time.

Numba timings on Python 3.9.0 on macOS-10.16-x86_64-i386-64bit
2CS RATES                   0.013764173299999998 ms, jitting 796.98625 ms, jitted 0.005980487200000018 ms, iterations to payoff 102,392
2CS RATES JACOBIAN          0.08951677309999999 ms, jitting 1,622.3466010000002 ms, jitted 0.010484501499999998 ms, iterations to payoff 20,528
EQUILIBRIUM RATES           0.027851035300000014 ms, jitting 953.8706329999993 ms, jitted 0.006555979999999995 ms, iterations to payoff 44,793
EQUILIBRIUM RATES JACOBIAN  0.2610606734 ms, jitting 7,206.177509000001 ms, jitted 0.014342878199999909 ms, iterations to payoff 29,208

Numba timings on Python 3.10.9 on macOS-13.6-x86_64-i386-64bit
2CS RATES                   0.014519099600147457 ms, jitting 1,042.198141978588 ms, jitted 0.005336513597285375 ms, iterations to payoff 113,497
2CS RATES JACOBIAN          0.09892194069689139 ms, jitting 22,125.61608100077 ms, jitted 0.024695230397628623 ms, iterations to payoff 298,082
EQUILIBRIUM RATES           0.029373795201536268 ms, jitting 1,433.5168090183288 ms, jitted 0.007495370204560459 ms, iterations to payoff 65,522
EQUILIBRIUM RATES JACOBIAN  0.3216409869957715 ms, jitting 169,699.05949098757 ms, jitted 0.061549677501898256 ms, iterations to payoff 652,460

Temporary runSim code counted the calls to those 4 functions:

TwoComponentSystem derivatives: 261,056
          derivatives_jacobian:  31,773
Equilibrium rates: 266,818
   rates_jacobian:  58,714

CovertLab / wcEcoli

Turn off jit compiler #1406