MightyBOBcnc
commented
1 year ago Scalene is still incompatible with the use of the multiprocessing library on the Windows platform (it uses the fork method but Windows only has the spawn method) as of this writing, and will terminate early as soon as Nixis reaches the first line that runs with parallel=True. Currently that is as soon as we sample the first octave of simplex noise for the starting elevations.
Also Scalene does not support memory profiling on Windows yet, which is unfortunate.
That being said, it looks like Numba can be told to work with python's threading library as an alternative to multiprocessing and I think Scalene supports that, so that is something that can be explored.
-Edit: It seems that I misunderstood that page. I still get the same multiprocessing error from Scalene when I experiment with Numba's THREADING_LAYER and THREADING_LAYER_PRIORITY config options.
So, the way I could use Scalene is limited, unless I start dual booting linux.
Due to the sheer volume of calculations in Nixis, performance is critical. I've found that the common CPU/memory profilers for python are all lacking. Scalene appears to be superior to basically all other python profilers. See this presentation from Strange Loop 2022.
Investigate the usage of Scalene for profiling in Nixis.
Possible blocker: Scalene on Windows is currently incompatible with numba's
parallel=Trueandprangemultiprocessing that Nixis makes heavy use of per: https://github.com/plasma-umass/scalene/issues/416 This is a problem as my dev environment is a Windows machine. It might still be useful for parts of Nixis that aren't threaded, or if we temporarily disable the parallel computing on a smaller subdivision size purely to get a handle on the resource usage of a function.