search

kconnour / pyRT_DISORT

A Python package for helping to compute input arrays to DISORT.
https://kconnour.github.io/pyRT_DISORT/
BSD 3-Clause "New" or "Revised" License
15 stars 5 forks source link

Latest numpy breaks installation #34

Closed kconnour closed 2 months ago

kconnour commented 3 months ago

Bug description

When trying to install pyRT_DISORT (not that it always worked before), numpy throws an error. It describes the error like this:

NumPy 2.0.0 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to downgrade to 'numpy<2' or try to upgrade the affected module. We expect that some modules will need time to support NumPy 2.

Minimal working example

import disort

Expected behavior

There should be no errors

Screenshots

No response

Operating system

Ubuntu 20.04

Python version

3.10

Priority

Highest

Further information

No response

kconnour commented 3 months ago

I took a stab at this today. The numpy docs provide almost no useful info for anyone who doesn't know what meson is (i.e. me).

It seems meson is a tool to help compile C or fortran code into a binarized .so file, and since numpy got rid of that functionality, meson or cmake is required to get pyRT_DISORT to install. All of the scipy ecosystem of code that relies on underlying fortran code is using meson, so it's likely the easier one to work with.

Step 1 seems to be getting f2py to make a C wrapper. In the disort4.0.99 directory, I ran python -m numpy.f2py *.f -m disort. That creates 2 files: disort-f2pywrappers.f and disortmodule.c. I don't exactly know what they do, but the numpy docs indicate they're important.

Step 2 seems to be making a file called meson.build in the directory with the fortran code. Using numpy's example (which doesn't work), I have this file

  version : '0.1',
  license: 'BSD-3',
  meson_version: '>=0.64.0',
  default_options : ['warning_level=2'],
)

add_languages('fortran')

py_mod = import('python')
py = py_mod.find_installation()
py_dep = py.dependency()

incdir_numpy = run_command(py,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()

incdir_f2py = run_command(py,
    ['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
    check : true
).stdout().strip()

inc_np = include_directories(incdir_numpy, incdir_f2py)

py.extension_module('disort',
  [
    'BDREF.f',
    'DISOBRDF.f',
    'DISORT.f',
    'ERRPACK.f',
    'LAPACK.f',
    'LINPAK.f',
    'RDI1MACH.f',
    'disortmodule.c',  # note: this assumes f2py was manually run before!
  ],
  incdir_f2py / 'fortranobject.c',
  include_directories: inc_np,
  dependencies : py_dep,
  install : true
)

Step 3 seems to be to run meson. I can install meson on Ubuntu with apt install meson, but they do not update apt with the latest meson frequently (this one seems 2 years out of date or so). If I went with this solution, I couldn't install it on Mac though. Somewhat fortunately, if you install meson with pip, it includes the latest meson binary. In the disort4.0.99 directory, I tried running ../.venv/bin/python ../.venv/bin/meson setup builddir to run the meson code, and it fails. It's mad at line 12 of the meson.build file. I see this as the log file. The ultimate error is that it cannot find python.h, but it seems to me that this is caused by something called pkg-config not being able to find which python I'm using, DESPITE IT ALREADY FINDING MY PYTHON on line 177 of the log file.

Main binary: /home/kyle/repos/pyrt_testing/disort4.0.99/../.venv/bin/python
Build Options: 
Python system: Linux
The Meson build system
Version: 1.4.1
Source dir: /home/kyle/repos/pyrt_testing/disort4.0.99
Build dir: /home/kyle/repos/pyrt_testing/disort4.0.99/builddir
Build type: native build
Project name: pyrt_testing
Project version: 0.1
-----------
Detecting compiler via: `cc --version` -> 0
stdout:
cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-----------
Running command: cc -E -dM -
-----
-----------
Detecting linker via: `cc -Wl,--version` -> 0
stdout:
GNU ld (GNU Binutils for Ubuntu) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
-----------
stderr:
collect2 version 11.4.0
/usr/bin/ld -plugin /usr/lib/gcc/x86_64-linux-gnu/11/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper -plugin-opt=-fresolution=/tmp/cceqBLlh.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -z relro /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/11/../../.. --version -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/11/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crtn.o
-----------
Sanity testing C compiler: cc
Is cross compiler: False.
Sanity check compiler command line: cc sanitycheckc.c -o sanitycheckc.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command:  /home/kyle/repos/pyrt_testing/disort4.0.99/builddir/meson-private/sanitycheckc.exe
C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
-----------
Detecting archiver via: `gcc-ar --version` -> 0
stdout:
GNU ar (GNU Binutils for Ubuntu) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.
-----------
-----------
Detecting compiler via: `cc --version` -> 0
stdout:
cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-----------
Running command: cc -E -dM -
-----
-----------
Detecting linker via: `cc -Wl,--version` -> 0
stdout:
GNU ld (GNU Binutils for Ubuntu) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
-----------
stderr:
collect2 version 11.4.0
/usr/bin/ld -plugin /usr/lib/gcc/x86_64-linux-gnu/11/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper -plugin-opt=-fresolution=/tmp/ccq3sNIm.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -z relro /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/11/../../.. --version -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/11/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crtn.o
-----------
Sanity testing C compiler: cc
Is cross compiler: False.
Sanity check compiler command line: cc sanitycheckc.c -o sanitycheckc.exe -D_FILE_OFFSET_BITS=64
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command:  /home/kyle/repos/pyrt_testing/disort4.0.99/builddir/meson-private/sanitycheckc.exe
C compiler for the build machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the build machine: cc ld.bfd 2.38
-----------
Detecting archiver via: `gcc-ar --version` -> 0
stdout:
GNU ar (GNU Binutils for Ubuntu) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.
-----------
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: x86_64
Target machine cpu: x86_64
meson.build:8: WARNING: add_languages is missing native:, assuming languages are wanted for both host and build.
-----------
Detecting compiler via: `gfortran --version` -> 0
stdout:
GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-----------
Running command: gfortran -E -dM -
-----
-----------
Detecting linker via: `gfortran -Wl,--version` -> 0
stdout:
GNU ld (GNU Binutils for Ubuntu) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
-----------
stderr:
collect2 version 11.4.0
/usr/bin/ld -plugin /usr/lib/gcc/x86_64-linux-gnu/11/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper -plugin-opt=-fresolution=/tmp/ccpjvKGi.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -z relro /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/11/../../.. --version -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/11/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crtn.o
-----------
Sanity testing Fortran compiler: gfortran
Is cross compiler: False.
Sanity check compiler command line: gfortran sanitycheckf.f90 -o sanitycheckf.exe
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command:  /home/kyle/repos/pyrt_testing/disort4.0.99/builddir/meson-private/sanitycheckf.exe
Fortran compiler for the build machine: gfortran (gcc 11.4.0 "GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
Fortran linker for the build machine: gfortran ld.bfd 2.38
-----------
Detecting compiler via: `gfortran --version` -> 0
stdout:
GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-----------
Running command: gfortran -E -dM -
-----
-----------
Detecting linker via: `gfortran -Wl,--version` -> 0
stdout:
GNU ld (GNU Binutils for Ubuntu) 2.38
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
-----------
stderr:
collect2 version 11.4.0
/usr/bin/ld -plugin /usr/lib/gcc/x86_64-linux-gnu/11/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper -plugin-opt=-fresolution=/tmp/ccFQ9RJr.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -z relro /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/11/../../.. --version -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/11/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crtn.o
-----------
Sanity testing Fortran compiler: gfortran
Is cross compiler: False.
Sanity check compiler command line: gfortran sanitycheckf.f90 -o sanitycheckf.exe
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command:  /home/kyle/repos/pyrt_testing/disort4.0.99/builddir/meson-private/sanitycheckf.exe
Fortran compiler for the host machine: gfortran (gcc 11.4.0 "GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
Fortran linker for the host machine: gfortran ld.bfd 2.38
Program python3 found: YES (/home/kyle/repos/pyrt_testing/disort4.0.99/../.venv/bin/python)
Searching for 'python-3.12' via pkgconfig lookup in LIBPC
Pkg-config binary missing from cross or native file, or env var undefined.
Trying a default Pkg-config fallback at pkg-config
Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
Determining dependency 'python-3.12' with pkg-config executable '/usr/bin/pkg-config'
env[PKG_CONFIG_LIBDIR]: /usr/lib/x86_64-linux-gnu/pkgconfig
env[PKG_CONFIG_PATH]: 
env[PKG_CONFIG]: /usr/bin/pkg-config
-----------
Called: `/usr/bin/pkg-config --modversion python-3.12` -> 1
stderr:
Package python-3.12 was not found in the pkg-config search path.
Perhaps you should add the directory containing `python-3.12.pc'
to the PKG_CONFIG_PATH environment variable
No package 'python-3.12' found
-----------
"python-3.12" could not be found in LIBPC, this is likely due to a relocated python installation
Searching for 'python-3.12' via fallback pkgconfig lookup in default paths
Running compile:
Working directory:  /home/kyle/repos/pyrt_testing/disort4.0.99/builddir/meson-private/tmpz9hz801o
Code:

        #ifdef __has_include
         #if !__has_include("Python.h")
          #error "Header 'Python.h' could not be found"
         #endif
        #else
         #include <Python.h>
        #endif
-----------
Command line: `cc -I/usr/include/python3.12 /home/kyle/repos/pyrt_testing/disort4.0.99/builddir/meson-private/tmpz9hz801o/testfile.c -E -P -D_FILE_OFFSET_BITS=64 -P -O0` -> 1
stderr:
/home/kyle/repos/pyrt_testing/disort4.0.99/builddir/meson-private/tmpz9hz801o/testfile.c:4:12: error: #error "Header 'Python.h' could not be found"
    4 |           #error "Header 'Python.h' could not be found"
      |            ^~~~~
-----------
Run-time dependency python found: NO (tried pkgconfig, pkgconfig and sysconfig)

meson.build:12:12: ERROR: Python dependency not found

The next step might be to add my venv lib to the path, but meson seems to use its own markup language that I don't know how to do that in the build file. If I try to specify my python path in line 10 or 11, it usually gets mad at something.

kconnour commented 3 months ago

After no luck on this issue, I tried going through the simplest example on the meson website: https://meson-python.readthedocs.io/en/latest/tutorials/introduction.html#

It also fails with the same error message, so I'm left to believe that this is a meson issue. I'll follow up with the devs...

kconnour commented 3 months ago

I inadvertently learned that installing the dev tools via sudo apt install python3.12-dev fixes that issue. However, the newest issue is that it cannot detect Ninja, despite having installed it via pip. I'm afraid it's looking for a system-wide installation and not a local one.

kconnour commented 3 months ago

Well it seems that if you just bypass the debugging steps and try a pip install, there are no issues (for my simple test project). I think this is close to fixed, but meson is somewhat mad if you try to pip install the project and the meson.build files is not at the same level where you run that command. Scipy has nested fortran code so it must be possible.

shanebyrneonmars commented 2 months ago

Hi Kyle, I've been using pyRT_DISORT for a few months (thanks! it's great) and just ran into this bug after I needed to refresh my conda environment. I'm trying to understand your last comment above. When you say 'try a pip install', which package do you mean?
Thanks, Shane.

kconnour commented 2 months ago

@shanebyrneonmars I'm happy to hear you've been getting some use from this project! Right now, no one can easily install this project. I'm working on the installation script after work when I have the time, but so far I haven't had too much luck figuring out what's wrong here.

To answer your question, I mean you can go to your terminal app and type pip install . or better yet <path to python in a virtual environment> -m pip install .. That should install pyRT_DISORT in whatever environment of your choice. Right now, if you use numpy v1.26.0 you should be able to install pyRT_DISORT. Any higher versions are likely to not be compatible and I'd really like to be able to get the latest and greatest numpy working with this code.

shanebyrneonmars commented 2 months ago

yep, reverting to numpy 1.26.4 works no problem. Thanks for your work on this - the package is a great tool.

kconnour commented 2 months ago

At long last I believe I have a fix. I cannot figure out how to get meson to walk through a directory structure, and even if I could, I recall reading that it doesn't like to mix build projects (i.e. it seems difficult/impossible to get it to install the disort code as a module and then the pyrt code as its own module). The best solution I have is to isolate the builds and put the fortran code and compilation in its own repo. More to come on this soon.

kconnour commented 2 months ago

This is now fixed with the latest pull request.