search

GEOS-ESM / ESMA_cmake

Custom CMake macros for the GEOS Earth System Model
Apache License 2.0
4 stars 9 forks source link

f2py use is broken under Python 3.12 #371

Open mathomp4 opened 6 months ago

mathomp4 commented 6 months ago

GEOSpyD recently moved to Python 3.12. Unfortunately, this breaks our use of f2py as the scripts here assume distutils (or at least work with it). My first test shows a failure much like that in https://github.com/numpy/numpy/issues/24874:

Traceback (most recent call last):
  File "/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/bin/f2py", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 766, in main
    run_compile()
  File "/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 738, in run_compile
    builder.compile()
  File "/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 176, in compile
    self.sources = _prepare_sources(self.modulename, self.sources, self.build_dir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 188, in _prepare_sources
    shutil.copy(source, bdir)
  File "/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/lib/python3.12/shutil.py", line 435, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/lib/python3.12/shutil.py", line 240, in copyfile
    raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: '/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4/f2py3-ShaveMantissa_/ShaveMantissa_.pyf' and '/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4/f2py3-ShaveMantissa_/ShaveMantissa_.pyf' are the same file

I'm not sure how to fix this yet. I know some flags aren't in the meson style of f2py, see https://numpy.org/devdocs/f2py/buildtools/distutils-to-meson.html

My hope is that I can follow around @HaoZeke on the numpy repo and figure out the fix...

mathomp4 commented 6 months ago

First update. It looks like it is the second step that fails. Step 1 (which I think makes the .pyf file) works:

/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/bin/f2py \
   -m ShaveMantissa_ \
   -h ShaveMantissa_.pyf \
   --build-dir /home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4/f2py3-ShaveMantissa_ \
   --include-paths /home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4:/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/lib:/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/include/MAPL_cfio_r4 \
   --overwrite-signature \ 
   /home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/MAPL_cfio/ShaveMantissa_py.F90 \ 
   /home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/MAPL_cfio/ShaveMantissa.c

But then we try the second step (which uses the .pyf file):

/ford1/share/gmao_SIteam/GEOSpyD/24.1.2-0_py3.12/2024-04-11/bin/f2py \
   -m ShaveMantissa_ \
   --build-dir /home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4/f2py3-ShaveMantissa_ \
   -c /home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4/f2py3-ShaveMantissa_/ShaveMantissa_.pyf 
   --f77flags='' \
   --f90flags='' \
   -I/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4 \
   -I/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/lib \
   -I/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/include/MAPL_cfio_r4 
   -L/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4 \
   -L/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/lib \
   -L/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/include/MAPL_cfio_r4 \
   /home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/MAPL_cfio/ShaveMantissa_py.F90 \
   /home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/MAPL_cfio/ShaveMantissa.c \
   -L/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/MAPL_cfio_r4 \
   -L/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/lib \
   -L/home/mathomp4/Models/MAPL2-IFX-2024.1/MAPL/build-Debug/include/MAPL_cfio_r4

and it fails.

Now, I see some stupid library duplication in there, but hmm...

mathomp4 commented 6 months ago

Wait...I'm not sure we ever use the two-step process in GEOS. We probably inherited that...