M2 Apple Silicon

[rjuenemann]

I installed the model on a new MacBook Pro with an M2 chip and had to upgrade the following libraries (for python 3.8.13). I let NumPy and SciPy install their own copy and let Aesara find NumPy's copy.

numpy 1.20.0 --> 1.23.4 scipy 1.7.3 --> 1.9.3 numba 0.55.0 --> 0.56.3 aesara 2.0.8 --> 2.8.7

These match the library versions @nvivanco is running on her M1.

For the older version of numpy, the lapack_opt_info for numpy was not available (not linked). The upgraded numpy version meant I needed to upgrade scipy and numba for compatibility. Finally, I had to upgrade aesara because the older version was givingAttributeError: module 'numpy.distutils.__config__' has no attribute 'blas_opt_info'

@1fish2 let me know if you have any thoughts

[1fish2]

Good to know, esp. for anyone installing on M2 now. Library dependencies are complicated.

I'm aiming to test on Python 3.11.2 and jump to that once there's a compatible Numba release (https://github.com/numba/numba/issues/8841). That'll require lots of library updates, maybe all of them to the latest releases. It's supposed to be a lot faster and stack tracebacks are more informative.

[rjuenemann]

Thank you @1fish2 ! Looking forward to the Python upgrade. Let me know if you need any help checking compatibility with Windows or an M2 Mac when we get there.

[1fish2]

@rjuenemann Oh yes, compatibility testing would help a lot! This could be the first testing on Windows, so it needn't be a priority, but it would be useful and might reveal some pathname bugs. I'll test on Intel Mac and the PR build will test on Linux (Sherlock).

Months ago I looked at release notes for several of the libraries, esp. Cython, and @ggsun reviewed release notes for Numpy, Scipy, and Matplotlib. We should look again. We didn't check aesara, which is changing rapidly, although wcEcoli uses a small portion of that library.

[rjuenemann]

@1fish2 Sounds good! I'm curious - is there anything in particular that we are looking for in the release notes?

[1fish2]

Good question! Mainly we're looking for library changes that require us to make code changes, or compatibility issues with other libraries & library versions. If a library deprecates or removes API features, we can search for calls to those functions and methods in our code. (BTW, PyCharm's search features are better than grep since they process the code at the AST level.) Changes to API call args and semantics are harder to search on. The hard part with numpy is when they change the semantics of edge cases like returning a scalar vs. a 1-element array or an array copy vs. a view. The hard part with matplotlib is changes to its plot arg strings. I rely on Gwanggyu for these since he knows the code base better than anyone.

Sometimes a few releases of a library will print deprecation warnings, but it's impractical for us to test all the intermediate versions of all the libraries, temporarily installing compatible versions of the other libraries to do that.

Secondarily, it's useful to note new features that could help a lot. E.g. we could speed up code like AnalysisPaths by switching from os.listdir() to os.scandir() to save a lot of isdir() OS calls. (That change calls for additional test cases.) This could speed up all the analysis tasks or it might be a small speedup compared to the analysis work itself.

[1fish2]

The branch python-3.11.3 has most of what's needed for Python 3.11, using the release candidates for numba and llvmlite.

@rjuenemann Can you please update knowledge_base_raw.py in this PR to the current Biopython API? It no longer supports Alphabet. Then we can install the current Biopython library. For now, this PR uses biopython==1.77 but that version doesn't claim to support Python 3.9+.

TODO: Testing.

[rjuenemann]

Hi @1fish2

Thank you for the detailed description of what we usually look for in the release notes!

I pushed a commit to the python-3.11.3 branch removing the usage of Alphabet from knowledge_base_raw.py. I ran a local test with the new genes option on and this removal did not affect the desired behavior, so we should be okay to upgrade to the current Biopython library.

[1fish2]

Perfect! Now I updated the PR to biopython==1.81 again.

Another reason to review release notes occurs to me: Looking for relevant bug fixes. E.g. there's a good chance that the updated matplotlib + PIL (Pillow image processing) will either fix #1360 or provide better error information on it, although it might not be visible in their release notes.