Open DavidHuber-NOAA opened 2 months ago
FYI @malloryprow @climbfuji
Thanks! I was just creating an issue, but you were faster ;-)
I haven't looked into this at all, but I will note that there are many more matplotlib
versions available in spack already. Maybe one of the slightly older ones still works with python@3.11.7
and does make plots?
$ spack info py-matplotlib
PythonPackage: py-matplotlib
Description:
Matplotlib is a comprehensive library for creating static, animated, and
interactive visualizations in Python.
Homepage: https://matplotlib.org/
Preferred version:
3.8.4 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.8.4.tar.gz
Safe versions:
3.8.4 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.8.4.tar.gz
3.8.3 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.8.3.tar.gz
3.8.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.8.2.tar.gz
3.8.1 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.8.1.tar.gz
3.8.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.8.0.tar.gz
3.7.4 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.7.4.tar.gz
3.7.3 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.7.3.tar.gz
3.7.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.7.2.tar.gz
3.7.1 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.7.1.tar.gz
3.7.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.7.0.tar.gz
3.6.3 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.6.3.tar.gz
3.6.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.6.2.tar.gz
3.6.1 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.6.1.tar.gz
3.6.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.6.0.tar.gz
3.5.3 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.5.3.tar.gz
3.5.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.5.2.tar.gz
3.5.1 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.5.1.tar.gz
3.5.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.5.0.tar.gz
3.4.3 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.4.3.tar.gz
3.4.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.4.2.tar.gz
3.4.1 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.4.1.tar.gz
3.4.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.4.0.tar.gz
3.3.4 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.3.4.tar.gz
3.3.3 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.3.3.tar.gz
3.3.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.3.2.tar.gz
3.3.1 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.3.1.tar.gz
3.3.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.3.0.tar.gz
3.2.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.2.2.tar.gz
3.2.1 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.2.1.tar.gz
3.2.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.2.0.tar.gz
3.1.3 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.1.3.tar.gz
3.1.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.1.2.tar.gz
3.1.1 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.1.1.tar.gz
3.1.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.1.0.tar.gz
3.0.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.0.2.tar.gz
3.0.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-3.0.0.tar.gz
Deprecated versions:
2.2.5 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-2.2.5.tar.gz
2.2.4 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-2.2.4.tar.gz
2.2.3 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-2.2.3.tar.gz
2.2.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-2.2.2.tar.gz
2.0.2 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-2.0.2.tar.gz
2.0.0 https://files.pythonhosted.org/packages/source/m/matplotlib/matplotlib-2.0.0.tar.gz
I will try installing 3.8.4 via pip3 with the dependent libraries from the spack-stack build and see if that resolves the issue. If not, I will try bumping other versions (e.g. numpy) to see if that resolves it.
It not, I'll see if I can reopen the matplotlib GitHub issue.
The issue appears to be with numpy, and specifically when compiled with Intel. It's a little tricky to replicate as well:
module use /scratch1/NCEPDEV/nems/Alexander.Richert/spack-stack-py311-aug24/envs/test/install/modulefiles/Core
module load stack-intel stack-python py-matplotlib
python simple_plot.py # returns an error
module unload py-numpy/1.25.2
module load py-pip
python -m venv venv
. venv/bin/activate
pip install py-numpy==1.25.2
python simple_plot.py # success!
I should mention that upgrading to matplotlib==3.8.4 did not resolve the issue.
Thanks for looking into this. Just to confirm, this works fine if py-numpy is compiled with GNU? And this is on Hera, right? I'll try to build a GNU and Intel environment on another system at NRL to reproduce this.
@climbfuji I'm not sure if this works with GNU or not, but probably. All I have confirmed is that it works when pip installs the binaries, which are almost certainly GNU-compiled. numpy
is built in spack-stack with openblas
support, which I'm not sure if the pip installation includes or not, so that is at least theoretically another possibility.
We always compiled numpy with openblas, with both GNU and Intel, for many releases of spack-stack. That hasn't changed on Hera as far as I know.
@DavidHuber-NOAA Unfortunately I cannot reproduce this on Nautilus. The machine uses intel@2021.5.0
with intel-oneapi-mkl
as the linalg provider (blas
, lapack
, fftw-api
). This is different from Hera, which uses openblas
(specifically requested by NOAA). But I am not sure if this is the reason or not.
$ module purge
$ module use /p/work2/heinzell/spack-stack-1.8.0-rc1/envs/ue-intel-2021.5.0/install/modulefiles/Core
$ module load stack-intel/2021.5.0
$ module load stack-openmpi/4.1.6
$ module load stack-python/3.11.7
$ module load py-matplotlib/3.8.0
$ python simple_plot.py
/p/home/heinzell/simple_plot.py:16: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
plt.show()
[heinzell@nautilus08 ~]$ ls -lart
...
-rw-------. 1 ***** ***** 34492 Aug 26 21:50 test.png
...
$ module li
Currently Loaded Modulefiles:
1) intel/tbb/latest 6) penguin/openmpi/4.1.6/intel-classic-2022.0.2 11) libxcrypt/4.4.35 16) stack-python/3.11.7 21) py-numpy/1.25.2 26) py-kiwisolver/1.4.5 31) py-six/1.16.0
2) intel/compiler-rt/latest 7) slurm 12) zlib-ng/2.1.6 17) libpng/1.6.37 22) py-pybind11/2.11.0 27) py-packaging/23.1 32) py-python-dateutil/2.8.2
3) intel/oclfpga/latest 8) stack-openmpi/4.1.6 13) sqlite/3.43.2 18) intel-oneapi-mkl/2022.0.2 23) py-contourpy/1.0.7 28) libjpeg/2.1.0 33) qhull/2020.2
4) intel/compiler/2022.0.2 9) gettext/0.19.8.1 14) util-linux-uuid/2.38.1 19) python-venv/1.0 24) py-cycler/0.11.0 29) py-pillow/9.5.0 34) py-matplotlib/3.8.0
5) stack-intel/2021.5.0 10) glibc/2.28 15) python/3.11.7 20) py-setuptools/63.4.3 25) py-fonttools/4.39.4 30) py-pyparsing/3.1.2
I built a basic gcc environment on Hera with the following spack.yaml:
# spack-stack hash: ac43a4c
# spack hash: d8de455f35
spack:
concretizer:
unify: when_possible
view: false
include:
- site
- common
definitions:
- compilers: ['%gcc']
- packages:
- python@3.11.7
- py-matplotlib@3.8.0
specs:
- matrix:
- [$packages]
- [$compilers]
packages:
all:
prefer: ['%gcc']
Numpy built with openblas and I was able to generate the plot.
I then built this same environment with Intel and was unable to make the plot, receiving the same cannot convert float NaN to integer
error as before. Hera is using the same version of Intel (2021.5.0) as Nautilus, so I'm not sure what exactly is causing this issue. Here is the module list that is loaded with Intel on Hera:
> module use /scratch1/NCEPDEV/global/David.Huber/SPACK/ss_matplotlib/envs/matplotlib_intel/install/modulefiles/Core
> module load stack-intel stack-python py-matplotlib
> module list
Currently Loaded Modules:
1) intel/2022.1.2 6) libxcrypt/4.4.35 11) stack-python/3.11.7 16) py-numpy/1.25.2 21) py-kiwisolver/1.4.5 26) py-six/1.16.0
2) stack-intel/2021.5.0 7) zlib-ng/2.1.6 12) libpng/1.6.37 17) py-pybind11/2.11.0 22) py-packaging/23.1 27) py-python-dateutil/2.8.2
3) glibc/2.28 8) sqlite/3.43.2 13) openblas/0.3.24 18) py-contourpy/1.0.7 23) libjpeg/2.1.0 28) qhull/2020.2
4) tar/1.26 9) util-linux-uuid/2.38.1 14) python-venv/1.0 19) py-cycler/0.11.0 24) py-pillow/9.5.0 29) py-matplotlib/3.8.0
5) gettext/0.21.1 10) python/3.11.7 15) py-setuptools/63.4.3 20) py-fonttools/4.39.4 25) py-pyparsing/3.1.2
Besides the openblas/mkl difference between Nautilus and Hera, the only other difference I see is gettext
. Nautilus is running version 0.19.8.1 and Hera 0.21.1. Also, even though the ifort and icc versions are both 2021.5.0, the entire Intel packages differ by a major version (Hera @ 2022.1.2, Nautilus at 2022.0.2).
@climbfuji I tried rebuilding the environment with gettext@0.19.8.1
to match Nautilus, but gettext
fails to build with the error
==> Installing gettext-0.19.8.1-eqf5yfao2qymautajaznt5mqtivff7dc [28/59]
==> No binary for gettext-0.19.8.1-eqf5yfao2qymautajaznt5mqtivff7dc found: installing from source
==> Fetching file:///scratch1/NCEPDEV/nems/role.epic/spack-stack/source-cache/_source-cache/archive/10/105556dbc5c3fbbc2aa0edb46d22d055748b6f5c7cd7a8d99f8e7eb84e938be4.tar.xz
==> Applied patch /scratch1/NCEPDEV/global/David.Huber/SPACK/ss_matplotlib/spack/var/spack/repos/builtin/packages/gettext/test-verify-parallel-make-check.patch
==> Error: FileNotFoundError: [Errno 2] No such file or directory: 'libtextstyle/configure'
/scratch1/NCEPDEV/global/David.Huber/SPACK/ss_matplotlib/spack/var/spack/repos/builtin/packages/gettext/package.py:85, in patch:
82 "gl_cv_libxml_force_included=yes",
83 "gl_cv_libxml_force_included=no",
84 "libtextstyle/configure",
>> 85 string=True,
86 )
I have two more ideas: 1) try building again with gettext@0.21.1
and Intel-MKL and 2) Try working forward from gettext@0.19.8.1
until I can find a minimum version that builds on Hera and see if the issue is still present.
external gettext?
Building the environment with intel-MKL instead of openblas did not fix the issue.
@climbfuji Good suggestion. I'll try an external gettext package.
I'm also going to open a matplotlib issue.
@climbfuji I'm unable to concretize with the system gettext
. Here is what I added to site/packages.yaml:
gettext:
buildable: false
externals:
- spec: gettext@0.19.8.1
prefix: /usr
And here is the output of spack concretize
:
==> Error: concretization failed for the following reasons:
1. cannot satisfy a requirement for package 'py-setuptools'.
Any suggestions?
@climbfuji I'm unable to concretize with the system
gettext
. Here is what I added to site/packages.yaml:gettext: buildable: false externals: - spec: gettext@0.19.8.1 prefix: /usr
And here is the output of
spack concretize
:==> Error: concretization failed for the following reasons: 1. cannot satisfy a requirement for package 'py-setuptools'.
Any suggestions?
Not really, but what I would try is to unpin py-setuptools or pin it to a different version if you get duplicates
@climbfuji I wasn't able to unpin py-setuptools as I got duplicates and attempting to pin it to something newer caused conflicts with py-numpy@1.25.2, so I updated py-numpy to 1.26.4 instead. This did not solve the issue. I then used the system gettext (0.19.8.1), but that also did not solve the issue. I accidentally installed this last round with openblas instead of Intel-MKL, but I doubt that is the issue given that the problem occurred previously with Intel-MKL/py-numpy@1.25.2.
I'm not sure where to go from here, but would gladly take suggestions for you and/or @AlexanderRichert-NOAA.
I'm going to try rebuilding the environment but pointing at the gcc-compiled version of py-numpy.
@DavidHuber-NOAA I installed the spack-stack-1.8.0 release candidate on S4 for testing, and I was able to create simple plots with matplotlib. S4 uses intel@2021.5.0.
Building with the gcc-compiled py-numpy was successful.
I will wait for an official release candidate installation on Hera then try the test again.
@DavidHuber-NOAA I installed the spack-stack-1.8.0 release candidate on S4 for testing, and I was able to create simple plots with matplotlib. S4 uses intel@2021.5.0.
@DavidHuber-NOAA I need to correct my previous statement. I was able to plot a different, simple plot on S4. Your simple_plot.py fails in the same way on S4 as it does on Hera.
OK, good to know. Thanks for the correction. I'll keep working with my installation on Hera, then. The matplotlib developers may have an idea on how to fix the issue 🤞
That would be great. I am changing py-numpy versions back by one major number at a time, in the hope that it will work with a somewhat more recent version than what we had in spack-stack-1.7.0. I am also checking on the oneapi compilers, and there are no problems with py-numpy@1.25.2 and py-matplotlib@3.8.0.
@DavidHuber-NOAA I got it to work with py-numpy@1.23.5
and py-matplotlib@3.7.4
. Since py-matplotlib@3.8:
depend on py-numpy@1.25:
, we could work around this by pinning py-numpy
to 1.23.5 and py-matplotlib to 3.7.4 for %intel
only (i.e. in configs/common/packages_intel.yaml
.
After all, Intel classic will be going away soon ... and there is no problem with the LLVM-based compilers with 1.25.2 / 3.8.0.
Fantastic! Alright, that sounds like a plan to me.
I'll keep working with the matplotlib folks and let them know of this workaround.
We have a workaround in place for spack-stack-1.8.0: use py-numpy@1.23.5
and py-matplotlib@3.7.4
with newer versions of the Intel classic compilers (intel@2021.x.y
). On Acorn with intel@19
, we use py-numpy@1.24.4
and py-matplotlib@3.7.4
.
We need to keep this issue open until the issue is fixed upstream (https://github.com/matplotlib/matplotlib/issues/28762), or until we switch to the Intel LLVM compilers for C/C++. Whichever solution, hopefully it is in time for spack-stack-1.9.0.
Describe the bug The matplotlib python library is unable to generate a basic plot, i.e. https://matplotlib.org/stable/gallery/lines_bars_and_markers/simple_plot.html. There may be a bug between the version of matplotlib and another python library. Another user reported the same bug, though it was never resolved: https://github.com/matplotlib/matplotlib/issues/23923.
To Reproduce
Expected behavior A basic sine-wave plot would be generated.
System: At least Hera
Additional context Found while testing Python 3.11.7 upgrade #1217.