Closed 2sn closed 9 months ago
I also note that meson
and ninja
are not in the dependencies for NumPy
.
After installing these, I replace the line 140 of
/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py
shutil.copy(source, bdir)
by
try:
shutil.copy(source, bdir)
except shutil.SameFileError:
pass
but then compilation fails
FAILED: _interface.cpython-312-x86_64-linux-gnu.so.p/interface.f90.o
gfortran -I_interface.cpython-312-x86_64-linux-gnu.so.p -I. -I.. -I/home/alex/Python/lib/python3.12/site-packages/numpy/core/include -I/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/src -I/home/alex/Python/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -O3 -fPIC -J_interface.cpython-312-x86_64-linux-gnu.so.p -o _interface.cpython-312-x86_64-linux-gnu.so.p/interface.f90.o -c ../interface.f90
../interface.f90:8:7:
8 | use typedef, only: &
| 1
Fatal Error: Cannot open module file ‘typedef.mod’ for reading at (1): No such file or directory
```shell
The Meson build system
Version: 1.2.2
Source dir: /home/alex/python/source/multistar/_build
Build dir: /home/alex/python/source/multistar/_build/bbdir
Build type: native build
Project name: _interface
Project version: 0.1
Fortran compiler for the host machine: gfortran (gcc 13.2.1 "GNU Fortran (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)")
Fortran linker for the host machine: gfortran ld.bfd 2.39-9
C compiler for the host machine: cc (gcc 13.2.1 "cc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)")
C linker for the host machine: cc ld.bfd 2.39-9
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/home/alex/Python/bin/python3.12)
Found pkg-config: /usr/bin/pkg-config (1.8.0)
Run-time dependency python found: YES 3.12
Build targets in project: 1
Found ninja-1.11.1.git.kitware.jobserver-1 at /home/alex/Python/bin/ninja
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/alex/Python/bin/ninja -C /home/alex/python/source/multistar/_build/bbdir
ninja: Entering directory `/home/alex/python/source/multistar/_build/bbdir'
[3/7] Compiling Fortran object _interface.cpython-312-x86_64-linux-gnu.so.p/interface.f90.o
FAILED: _interface.cpython-312-x86_64-linux-gnu.so.p/interface.f90.o
gfortran -I_interface.cpython-312-x86_64-linux-gnu.so.p -I. -I.. -I/home/alex/Python/lib/python3.12/site-packages/numpy/core/include -I/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/src -I/home/alex/Python/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -O3 -fPIC -J_interface.cpython-312-x86_64-linux-gnu.so.p -o _interface.cpython-312-x86_64-linux-gnu.so.p/interface.f90.o -c ../interface.f90
../interface.f90:8:7:
8 | use typedef, only: &
| 1
Fatal Error: Cannot open module file ‘typedef.mod’ for reading at (1): No such file or directory
compilation terminated.
[4/7] Compiling Fortran object _interface.cpython-312-x86_64-linux-gnu.so.p/_interface-f2pywrappers2.f90.o
FAILED: _interface.cpython-312-x86_64-linux-gnu.so.p/_interface-f2pywrappers2.f90.o
gfortran -I_interface.cpython-312-x86_64-linux-gnu.so.p -I. -I.. -I/home/alex/Python/lib/python3.12/site-packages/numpy/core/include -I/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/src -I/home/alex/Python/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -O3 -fPIC -J_interface.cpython-312-x86_64-linux-gnu.so.p -o _interface.cpython-312-x86_64-linux-gnu.so.p/_interface-f2pywrappers2.f90.o -c ../_interface-f2pywrappers2.f90
../_interface-f2pywrappers2.f90:7:21:
7 | use typedef, only: real64
| 1
Fatal Error: Cannot open module file ‘typedef.mod’ for reading at (1): No such file or directory
compilation terminated.
[6/7] Compiling C object _interface.cpython-312-x86_64-linux-gnu.so.p/_interfacemodule.c.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/alex/Python/bin/f2py3", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 732, in main
run_compile()
File "/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 705, in run_compile
builder.compile()
File "/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 131, in compile
self.run_meson(self.build_dir)
File "/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 124, in run_meson
raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['meson', 'compile', '-C', 'bbdir']' returned non-zero exit status 1.
Apparently, the _meson
module of f2py
does not pass on the --f90flags
(see above), "-fPIC -fno-second-underscore -O3 -funroll-loops -fconvert=big-endian -I /home/alex/python/source/multistar/_library"
that would tell the compiler where to find the mode files (-I /home/alex/python/source/multistar/_library
) and other compile specs.
Happy to help with debugging.
I try to add the include file manually, adding -I /home/alex/python/source/multistar/_library
to the second call, but to no avail. In any case, I requite other parameters to be passed to the compiler anyway, such as -fconvert=big-endian
where the include solution would fail.
another error seems to be
[6/6] Linking target _interface.cpython-312-x86_64-linux-gnu.so
Traceback (most recent call last):
File "/home/alex/Python/bin/f2py3", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 732, in main
run_compile()
File "/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 705, in run_compile
builder.compile()
File "/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 132, in compile
self._move_exec_to_root(self.build_dir)
File "/home/alex/Python/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 88, in _move_exec_to_root
shutil.move(path_object, Path.cwd())
File "/home/alex/Python/lib/python3.12/shutil.py", line 860, in move
raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path '/home/alex/python/source/multistar/_interface.cpython-312-x86_64-linux-gnu.so' already exists
here, in /home/alex/Python/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py
def _move_exec_to_root(self, build_dir: Path):
walk_dir = Path(build_dir) / self.meson_build_dir
path_objects = walk_dir.glob(f"{self.modulename}*.so")
for path_object in path_objects:
shutil.move(path_object, Path.cwd())
needs to be replaced by
def _move_exec_to_root(self, build_dir: Path):
walk_dir = Path(build_dir) / self.meson_build_dir
path_objects = walk_dir.glob(f"{self.modulename}*.so")
for path_object in path_objects:
if (file := (Path.cwd() / path_object.name)).exists:
file.unlink()
shutil.move(path_object, Path.cwd())
... after some more wiggling I get it to produce a binary module, however, however, I can't import stuff from it.
Since compiler flags are not passed on, optimisations and special settings are lost. This requires a fix.
The object library /home/alex/python/source/multistar/_library/library.a
is not linked, so even if I update the names in all python scripts (seems a really unnecessary incompatibility), it can' load the library because of missing dependencies:
File ~/python/source/multistar/driver.py:51
---> 51 from ._interface import add_stardata__ as fadd_stardata
ImportError: /home/alex/python/source/multistar/_interface.cpython-312-x86_64-linux-gnu.so: undefined symbol: __orientation_MOD_regularize_orientation
It seems f2py
's _meson
module is unusable in its present form.
Some modifications that seem to work, except it does not allow to build in its own directory, the the specified include directory for fortran mod
files is still ignored - generally, the the compiler flags are ignored.
template meson.build.template
project('${modulename}',
['c', 'fortran'],
version : '0.1',
meson_version: '>= 1.1.0',
default_options : [
'warning_level=1',
'buildtype=${buildtype}'
])
py = import('python').find_installation(pure: false)
py_dep = py.dependency()
incdir_numpy = run_command(py,
['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
check : true
).stdout().strip()
incdir_f2py = run_command(py,
['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
check : true
).stdout().strip()
inc_np = include_directories(incdir_numpy)
np_dep = declare_dependency(include_directories: inc_np)
incdir_f2py = incdir_numpy / '..' / '..' / 'f2py' / 'src'
inc_f2py = include_directories(incdir_f2py)
fortranobject_c = incdir_f2py / 'fortranobject.c'
inc_np = include_directories(incdir_numpy, incdir_f2py)
py.extension_module('${modulename}',
[
${source_list},
fortranobject_c
],
include_directories: [inc_np],
dependencies : [
py_dep,
${dep_list}
],
link_args: [
${extra_objects}
],
install : true)
module _meson.py
from __future__ import annotations
import errno
import shutil
import subprocess
from pathlib import Path
from ._backend import Backend
from string import Template
import warnings
class MesonTemplate:
"""Template meson build file generation class."""
def __init__(
self,
modulename: str,
sources: list[Path],
deps: list[str],
object_files: list[Path],
linker_args: list[str],
c_args: list[str],
build_type: str,
extra_objects: list[str],
):
self.modulename = modulename
self.build_template_path = (
Path(__file__).parent.absolute() / "meson.build.template"
)
self.sources = sources
self.deps = deps
self.substitutions = {}
self.objects = object_files
self.extra_objects = extra_objects
self.pipeline = [
self.initialize_template,
self.sources_substitution,
self.deps_substitution,
self.extra_substitution,
]
self.build_type = build_type
def meson_build_template(self) -> str:
if not self.build_template_path.is_file():
raise FileNotFoundError(
errno.ENOENT,
"Meson build template"
f" {self.build_template_path.absolute()}"
" does not exist.",
)
return self.build_template_path.read_text()
def initialize_template(self) -> None:
self.substitutions["modulename"] = self.modulename
self.substitutions["buildtype"] = self.build_type
def sources_substitution(self) -> None:
indent = " " * 21
self.substitutions["source_list"] = f",\n{indent}".join(
[f"'{source}'" for source in self.sources]
)
def deps_substitution(self) -> None:
indent = " " * 21
self.substitutions["dep_list"] = f",\n{indent}".join(
[f"dependency('{dep}')" for dep in self.deps]
)
def extra_substitution(self) -> None:
indent = " " * 21
self.substitutions["extra_objects"] = f",\n{indent}".join(
[f"'{extra}'" for extra in self.extra_objects]
)
def generate_meson_build(self):
for node in self.pipeline:
node()
template = Template(self.meson_build_template())
return template.substitute(self.substitutions)
class MesonBackend(Backend):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.dependencies = self.extra_dat.get("dependencies", [])
self.meson_build_dir = "bbdir"
self.build_type = (
"debug" if any("debug" in flag for flag in self.fc_flags) else "release"
)
def _move_exec_to_root(self, build_dir: Path):
walk_dir = Path(build_dir) / self.meson_build_dir
path_objects = walk_dir.glob(f"{self.modulename}*.so")
for path_object in path_objects:
if (file := (Path.cwd() / path_object.name)).exists:
file.unlink()
shutil.move(path_object, Path.cwd())
def _get_build_command(self):
return [
"meson",
"setup",
self.meson_build_dir,
]
def write_meson_build(self, build_dir: Path) -> None:
"""Writes the meson build file at specified location"""
meson_template = MesonTemplate(
self.modulename,
self.sources,
self.dependencies,
self.extra_objects,
self.flib_flags,
self.fc_flags,
self.build_type,
self.extra_objects,
)
src = meson_template.generate_meson_build()
Path(build_dir).mkdir(parents=True, exist_ok=True)
meson_build_file = Path(build_dir) / "meson.build"
meson_build_file.write_text(src)
return meson_build_file
def run_meson(self, build_dir: Path):
completed_process = subprocess.run(self._get_build_command(), cwd=build_dir)
if completed_process.returncode != 0:
raise subprocess.CalledProcessError(
completed_process.returncode, completed_process.args
)
completed_process = subprocess.run(
["meson", "compile", "-C", self.meson_build_dir], cwd=build_dir
)
if completed_process.returncode != 0:
raise subprocess.CalledProcessError(
completed_process.returncode, completed_process.args
)
def compile(self) -> None:
self.sources = _prepare_sources(self.modulename, self.sources, self.build_dir)
self.write_meson_build(self.build_dir)
self.run_meson(self.build_dir)
self._move_exec_to_root(self.build_dir)
def _prepare_sources(mname, sources, bdir):
extended_sources = sources.copy()
Path(bdir).mkdir(parents=True, exist_ok=True)
# Copy sources
for source in sources:
try:
shutil.copy(source, bdir)
except shutil.SameFileError:
pass
generated_sources = [
Path(f"{mname}module.c"),
Path(f"{mname}-f2pywrappers2.f90"),
Path(f"{mname}-f2pywrappers.f"),
]
bdir = Path(bdir)
for generated_source in generated_sources:
if generated_source.exists():
shutil.copy(generated_source, bdir / generated_source.name)
extended_sources.append(generated_source.name)
generated_source.unlink()
extended_sources = [
Path(source).name
for source in extended_sources
if not Path(source).suffix == ".pyf"
]
return extended_sources
Now a version that seems to work for my use case, also allowing to pass parameters to the fortran compiler. Obviously, this can be system-dependent and care is on the user for now. But t least it can be done.
meson.build.template
project('${modulename}',
['c', 'fortran'],
version : '0.1',
meson_version: '>= 1.1.0',
default_options : [
'warning_level=1',
'buildtype=${buildtype}'
])
py = import('python').find_installation(pure: false)
py_dep = py.dependency()
incdir_numpy = run_command(py,
['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
check : true
).stdout().strip()
incdir_f2py = run_command(py,
['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
check : true
).stdout().strip()
inc_np = include_directories(incdir_numpy)
np_dep = declare_dependency(include_directories: inc_np)
incdir_f2py = incdir_numpy / '..' / '..' / 'f2py' / 'src'
inc_f2py = include_directories(incdir_f2py)
fortranobject_c = incdir_f2py / 'fortranobject.c'
inc_np = include_directories(incdir_numpy, incdir_f2py)
py.extension_module('${modulename}',
[
${source_list},
fortranobject_c
],
include_directories: [inc_np],
dependencies : [
py_dep,
${dep_list}
],
link_args: [
${extra_objects}
],
fortran_args: [
${fortran_args}
],
install : true)
_meson.py
from __future__ import annotations
import errno
import shutil
import subprocess
from pathlib import Path
from ._backend import Backend
from string import Template
import warnings
class MesonTemplate:
"""Template meson build file generation class."""
def __init__(
self,
modulename: str,
sources: list[Path],
deps: list[str],
object_files: list[Path],
linker_args: list[str],
c_args: list[str],
build_type: str,
extra_objects: list[str],
fc_flags: list[str],
):
self.modulename = modulename
self.build_template_path = (
Path(__file__).parent.absolute() / "meson.build.template"
)
self.sources = sources
self.deps = deps
self.substitutions = {}
self.objects = object_files
self.extra_objects = extra_objects
self.fc_flags = fc_flags
self.sources = sources
self.pipeline = [
self.initialize_template,
self.sources_substitution,
self.deps_substitution,
self.extra_substitution,
self.determine_f90,
self.fc_flags_substitution,
]
self.build_type = build_type
def meson_build_template(self) -> str:
if not self.build_template_path.is_file():
raise FileNotFoundError(
errno.ENOENT,
"Meson build template"
f" {self.build_template_path.absolute()}"
" does not exist.",
)
return self.build_template_path.read_text()
def initialize_template(self) -> None:
self.substitutions["modulename"] = self.modulename
self.substitutions["buildtype"] = self.build_type
def sources_substitution(self) -> None:
indent = " " * 21
self.substitutions["source_list"] = f",\n{indent}".join(
[f"'{source}'" for source in self.sources]
)
def deps_substitution(self) -> None:
indent = " " * 21
self.substitutions["dep_list"] = f",\n{indent}".join(
[f"dependency('{dep}')" for dep in self.deps]
)
def extra_substitution(self) -> None:
indent = " " * 21
self.substitutions["extra_objects"] = f",\n{indent}".join(
[f"'{extra}'" for extra in self.extra_objects]
)
def determine_f90(self) -> None:
for source in self.sources:
if (source.endswith('f2pywrappers.f90') or
source.endswith('f2pywrappers2.f90')):
self.f90 = True
break
else:
self.f90 = False
def fc_flags_substitution(self) -> None:
# TODO - options should be filteretd and split up
indent = " " * 21
if self.f90:
select = '--f90flags='
else:
select = '--f77flags='
for flags in self.fc_flags:
if flags.startswith(select):
fortran_flags = flags[11:].split()
break
else:
fortrtran_flags = list()
self.substitutions["fortran_args"] = f",\n{indent}".join(
[f"'{flag}'" for flag in fortran_flags]
)
def generate_meson_build(self):
for node in self.pipeline:
node()
template = Template(self.meson_build_template())
return template.substitute(self.substitutions)
class MesonBackend(Backend):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.dependencies = self.extra_dat.get("dependencies", [])
self.meson_build_dir = "bbdir"
self.build_type = (
"debug" if any("debug" in flag for flag in self.fc_flags) else "release"
)
def _move_exec_to_root(self, build_dir: Path):
walk_dir = Path(build_dir) / self.meson_build_dir
path_objects = walk_dir.glob(f"{self.modulename}*.so")
for path_object in path_objects:
if (file := (Path.cwd() / path_object.name)).exists:
file.unlink()
shutil.move(path_object, Path.cwd())
def _get_build_command(self):
return [
"meson",
"setup",
self.meson_build_dir,
]
def write_meson_build(self, build_dir: Path) -> None:
"""Writes the meson build file at specified location"""
meson_template = MesonTemplate(
self.modulename,
self.sources,
self.dependencies,
self.extra_objects,
self.flib_flags,
self.fc_flags,
self.build_type,
self.extra_objects,
self.fc_flags,
)
src = meson_template.generate_meson_build()
Path(build_dir).mkdir(parents=True, exist_ok=True)
meson_build_file = Path(build_dir) / "meson.build"
meson_build_file.write_text(src)
return meson_build_file
def run_meson(self, build_dir: Path):
completed_process = subprocess.run(self._get_build_command(), cwd=build_dir)
if completed_process.returncode != 0:
raise subprocess.CalledProcessError(
completed_process.returncode, completed_process.args
)
completed_process = subprocess.run(
["meson", "compile", "-v", "-C", self.meson_build_dir], cwd=build_dir
)
if completed_process.returncode != 0:
raise subprocess.CalledProcessError(
completed_process.returncode, completed_process.args
)
def compile(self) -> None:
self.sources = _prepare_sources(self.modulename, self.sources, self.build_dir)
self.write_meson_build(self.build_dir)
self.run_meson(self.build_dir)
self._move_exec_to_root(self.build_dir)
def _prepare_sources(mname, sources, bdir):
extended_sources = sources.copy()
Path(bdir).mkdir(parents=True, exist_ok=True)
# Copy sources
for source in sources:
try:
shutil.copy(source, bdir)
except shutil.SameFileError:
pass
generated_sources = [
Path(f"{mname}module.c"),
Path(f"{mname}-f2pywrappers2.f90"),
Path(f"{mname}-f2pywrappers.f"),
]
bdir = Path(bdir)
for generated_source in generated_sources:
if generated_source.exists():
shutil.copy(generated_source, bdir / generated_source.name)
extended_sources.append(generated_source.name)
generated_source.unlink()
extended_sources = [
Path(source).name
for source in extended_sources
if not Path(source).suffix == ".pyf"
]
return extended_sources
If there is interest, I can make a pull request of it.
Thanks for going over the new module and apologies for not noticing this earlier (feel free to ping me on f2py issues generally). In general, none of the old flags to pass parameters will work, and the idea is to use meson
native methods. While fc_flags_substitution
and similar methods might be useful for backwards compatibility, (and there was some work porting the front-end to be more generic) this is a bit of a losing game, in that we don't gain much from the switch if we still need to parse arguments to compilers and sort them within f2py
. Rather than try to support all the older methods of configuring the build we now decouple it to the bare basics and let users customize the meson
build system instead.
It should have worked by just populating FFLAGS
as well instead of modifying the meson
setup in numpy
.
To me, personally, I would rather document this better and let users rework their build systems to meson
than try to re-implement / proxy methods of configuring from f2py
-> distutils
-> meson
at our end.
I also note that
meson
andninja
are not in the dependencies forNumPy
.
Tracked in #24670
@HaoZeke Tanks for the clarification. Yes, I had to know what I wanted in the meson
file in the first place to adjust the code. I did this for one of my projects, but want to use it for others, ideally, transparently. I think I did not see meson
instruction on the f2py
doc pages at the time. I am not sure the FFLAGS
would have worked for my use case, needed link to a library.
Would it be ok to add these flange for some modest out-of-the-box backwards compatibility? No one has to use these for new projects or at all. I would make a pull request.
My background is that I use these in a python script that links f2py with external library also built by the script, but using an old-fashioned (extended) makefile. Ongoing scientific code development, not using distutils
, compiles code automatically when module is loaded.
@HaoZeke Tanks for the clarification. Yes, I had to know what I wanted in the
meson
file in the first place to adjust the code. I did this for one of my projects, but want to use it for others, ideally, transparently. I think I did not seemeson
instruction on thef2py
doc pages at the time. I am not sure theFFLAGS
would have worked for my use case, needed link to a library.
That's a good point, sorry I should have read the issue more carefully.
I think LDFLAGS=-Wl,--linker-flag
will work (docs)
Also (not well documented) there was a -dep
argument added for defining dependencies (which should setup everything needed in meson
if the dependency('library')
is supported. This is noted in the original feature PR.
e.g. For LAPACK: f2py -c --backend meson GaussJacobiQuadCCPy.pyf ../src/gjp_lapack.f90 - -dep lapack
This also supports external libraries with a .wrap
file (needs to be documented better).
-f90-flags
was a great user-facing CLI option before, but it does too much, not all compilers and linkers will accept a string of arguments where some are meant for the linker and some for the compiler. It is unfortunate to require this additional knowledge from users, but the alternative is to continue to deal with build issues here instead of on meson
where they should be.
Would it be ok to add these flange for some modest out-of-the-box backwards compatibility? No one has to use these for new projects or at all. I would make a pull request.
I would be quite strongly against this. The reason being that the plan is to have f2py
not build anything moving forward. If there are CLI flags or environment variables which meson
does not support then that would be a meson
bug, not an f2py
bug. It would be best to have a clean-up PR for the frontend to prevent these options from being passed at all, since there is atleast one other issue https://github.com/numpy/numpy/issues/24838 caused by this behavior of not clearly alerting the user to options which are no longer supported.
My background is that I use these in a python script that links f2py with external library also built by the script, but using an old-fashioned (extended) makefile.
I understand, and in general backwards compatibility is a major concern but, NumPy 2.0 requires users to come and look at the documentation anyway since it is a major version change with a lot of breaking changes. In the same vein, f2py
users should also update their dependent code from 2.0 onwards. The problem is that the f2py
documentation is currently lagging (though @melissawm might be able to add to it in the near-future).
It would be really great if you could add an example showing the translation from pre-2.0 f2py
to the current version instead..
^- I understand this is a more annoying request than accepting the PR for the changes which work for your code (and will generally, for gfortran
work more or less for everyone), since
this would require fiddling with the environment variables / meson documentation but it would really would be much better in the long run.
Ongoing scientific code development, not using
distutils
, compiles code automatically when module is loaded.
So the python process runs F2PY as a shell command? (via subprocess or something?) That should be a good fit for setting environment variables in the dependent shell as well, via modifications of os.environ
@HaoZeke I understand that you want to offload responsibility for the link and compile stage. My proposal aims at adding more basic functionality to allow people a quick start for simple project.
I trigger f2py using a subprocess call.
os.environ does not work for this purpose, retains environment python was called with. At least, this is what it was s few years back.
I understand that you want to offload responsibility for the link and compile stage. My proposal aims at adding more basic functionality to allow people a quick start for simple project.
If there are settings which cannot be passed as environment variables and are not amenable to being upstreamed I could see this but I don't think I understand which options would require this (yet).
os.environ does not work for this purpose, retains environment python was called with. At least, this is what it was s few years back. It does, but it returns a dictionary so the common use case is (honoring environment variables) cenv = os.environ cenv['FFLAGS'] = cenv.get('FFLAGS', '') + ' -O2' result = subprocess.run(['whatever'], text=True, capture_output=True, env=cenv) Or to use a "blank" environment one could call subprocess.run with any dictionary, but I prefer this approach. On Oct 20 2023, at 3:30 pm, Alexander Heger @.***> wrote:
@HaoZeke @.***/0?redirect=https%3A%2F%2Fgithub.com%2FHaoZeke&recipient=cmVwbHkrYWJiY3V0Mjd4MnMzYzRubTJwM29lYWdkaDdvMnBldmJuaGhoZ2cyYnVlQHJlcGx5LmdpdGh1Yi5jb20%3D) I understand that you want to offload responsibility for the link and compile stage. My proposal aims at adding more basic functionality to allow people a quick start for simple project.
I trigger f2py using a subprocess call. os.environ does not work for this purpose, retains environment python was called with. At least, this is what it was s few years back. — Reply to this email directly, view it on GitHub @./1?redirect=https%3A%2F%2Fgithub.com%2Fnumpy%2Fnumpy%2Fissues%2F24874%23issuecomment-1772958923&recipient=cmVwbHkrYWJiY3V0Mjd4MnMzYzRubTJwM29lYWdkaDdvMnBldmJuaGhoZ2cyYnVlQHJlcGx5LmdpdGh1Yi5jb20%3D), or unsubscribe @./2?redirect=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABBCUT767IS6J5TAENOWV43YAKKJPAVCNFSM6AAAAAA5WTXUCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZSHE2TQOJSGM&recipient=cmVwbHkrYWJiY3V0Mjd4MnMzYzRubTJwM29lYWdkaDdvMnBldmJuaGhoZ2cyYnVlQHJlcGx5LmdpdGh1Yi5jb20%3D). You are receiving this because you were mentioned.
@HaoZeke Ah, yes, now I recall, I wanted to use environment variables in a different way, access them in a library compiled/linked with f2py and these set before loading the module - or used on the module in general. I had not found a way to change that in this case.
I will give your suggestion of workaround a try in the next few days or week. (Still waiting for the port of vtk before transitioning to Python 3.12)
@HaoZeke Ah, yes, now I recall, I wanted to use environment variables in a different way, access them in a library compiled/linked with f2py and these set before loading the module - or used on the module in general. I had not found a way to change that in this case.
Makes sense, I think for that there would need to be changes to the C code generated by F2PY (to capture the environment and pass it through). I will open an enhancement issue for it.
I will give your suggestion of workaround a try in the next few days or week. (Still waiting for the port of vtk before transitioning to Python 3.12)
Wonderful, thanks a ton.
In general, none of the old flags to pass parameters will work, and the idea is to use meson native methods.
Could this be documented somewhere? I just lost several hours to this, wondering why none of my arguments are being picked up.
In general, none of the old flags to pass parameters will work, and the idea is to use meson native methods.
Could this be documented somewhere? I just lost several hours to this, wondering why none of my arguments are being picked up.
Sorry about that, it was in the release notes, but not prominently listed. For now, examples are still in https://github.com/numpy/numpy/pull/24532, but the documentation has been updated in https://github.com/numpy/numpy/pull/25124/files if you'd like to take a look.
I'm leaving this open because I want to at some point have a "cheatsheet" to translate between common distutils
flows and their equivalent meson
forms.
@HaoZeke
Could you still fix/include the edit for numpy/f2py/_backends/_meson.py
? This seems to be a simple and genuine bug, i.e., replace
def _move_exec_to_root(self, build_dir: Path):
walk_dir = Path(build_dir) / self.meson_build_dir
path_objects = walk_dir.glob(f"{self.modulename}*.so")
for path_object in path_objects:
shutil.move(path_object, Path.cwd())
by
def _move_exec_to_root(self, build_dir: Path):
walk_dir = Path(build_dir) / self.meson_build_dir
path_objects = walk_dir.glob(f"{self.modulename}*.so")
for path_object in path_objects:
if (file := (Path.cwd() / path_object.name)).exists:
file.unlink()
shutil.move(path_object, Path.cwd())
And also
def _prepare_sources(mname, sources, bdir):
extended_sources = sources.copy()
Path(bdir).mkdir(parents=True, exist_ok=True)
# Copy sources
for source in sources:
shutil.copy(source, bdir)
by
def _prepare_sources(mname, sources, bdir):
extended_sources = sources.copy()
Path(bdir).mkdir(parents=True, exist_ok=True)
# Copy sources
for source in sources:
if Path(source).parent == Path(bdir):
continue
shutil.copy(source, bdir)
@HaoZeke Could you still fix/include the edit for
numpy/f2py/_backends/_meson.py
? This seems to be a simple and genuine bug, i.e., replacedef _move_exec_to_root(self, build_dir: Path): walk_dir = Path(build_dir) / self.meson_build_dir path_objects = walk_dir.glob(f"{self.modulename}*.so") for path_object in path_objects: shutil.move(path_object, Path.cwd())
by
def _move_exec_to_root(self, build_dir: Path): walk_dir = Path(build_dir) / self.meson_build_dir path_objects = walk_dir.glob(f"{self.modulename}*.so") for path_object in path_objects: if (file := (Path.cwd() / path_object.name)).exists: file.unlink() shutil.move(path_object, Path.cwd())
This is related to in tree builds right? I am not sure it's a good idea to unconditionally remove pre existing files, though it is annoying in development environments..
@HaoZeke
Why would you want to keep the old version?
Would then a command parameter or f2py
flag be an option?
Yes, I do development, I rebuild frequently.
My focus is not distribution.
Most packages need to be developed before distributed ...
@HaoZeke How about the first change? Not trying to move to the same file and then fail?
@HaoZeke Why would you want to keep the old version? Would then a command parameter or
f2py
flag be an option? Yes, I do development, I rebuild frequently. My focus is not distribution.
Most packages need to be developed before distributed ...
I also spend more time on development and find it annoying... However it is the default behavior, even signature files are not overwritten by default. A flag would be a reasonable solution but also seems a little convoluted (would have to store state and make sure it works for distutils as well) for what is basically a single line externally, i.e:
rm -rf *.so && f2py -c ...
Perhaps a note in the documentation? I'm not actually against a flag as a PR, it might even be useful in some CI situations, but it would need to account for the different naming schemes on windows and stuff like that...
@HaoZeke
OK, yes, the rm
can be added.
This leaves the issue with the move to same file I mentioned first.
@HaoZeke How about the first change? Not trying to move to the same file and then fail?
This makes more sense but it still adds an assumption of state. So you might be left with files which aren't updated because of the continue clause. Which would again be an edge case someone can run into. The current behavior is clear and unambiguous IMO..
Since what's happening is that we copy the files to a new folder and run meson. Trying to handle the case of copying files back is kind of problematic to deal with.
It back to the problem of in place builds, which I guess is best not supported. This is also the stance taken by meson.
The easiest solution is to just use the build directory which can be easily removed when needed without worrying about changes made to the source files.
@HaoZeke My point is that I believe that the current code does not make sense in any circumstance, trying to move a file onto itself, and then failing to do so. Maybe, usually, this should not occur as usually the directories differ, but what is the harm in the code not failing if it does occur?
The harm here is that it is then silenced, and a user would not know why their files aren't being updated. The current code is verbose, and lets the user know exactly what went wrong. In general, as mentioned earlier, this will only mostly happen either in development environments or when trying to use the same directory for build-dir (which might as well be disallowed perhaps).
On 11/30/23 4:49 AM, Alexander Heger @.***> wrote:
@HaoZeke https://github.com/HaoZeke My point is that I believe that the current code does not make sense in any circumstance, trying to move a file onto itself, and then failing to do so. Maybe, usually, this should not occur as usually the directories differ, but what is the harm in the code not failing if it does occur?
— Reply to this email directly, view it on GitHub https://github.com/numpy/numpy/issues/24874#issuecomment-1833103988, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBCUT2DQOILSWWGEZRGWDLYHAF6JAVCNFSM6AAAAAA5WTXUCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGEYDGOJYHA. You are receiving this because you were mentioned.Message ID: @.***>
@HaoZeke
My apologies, I am not following you on this reasoning.
That fix is only for copying a file onto itself.
Nothing is being updated.
No other case will be affected.
I would guess that the case where this is a problems may be more common than where this would be the only point of failure when directories are messed up.
I found the numpy
community always extremely forthcoming and helpful making things work, but I have the impression the focus has now shifted toward a distribution-based support approach alone. Whereas I appreciate the push toward new technologies (as I often like to do myself to the frustration of my collaborators and students), I think many users would appreciate having easy ways to continue being able to just do local development, and, ideally, continue using their existing setups as much as possible. This is also to give users the room to make switches to new technology on their own timeframe, as much as possible; having to adjust and rewrite lots of code at times or on timescales controlled by others - in part w/o need - is rather frustrating and discouraging.
@HaoZeke My apologies, I am not following you on this reasoning. That fix is only for copying a file onto itself. Nothing is being updated. No other case will be affected. I would guess that the case where this is a problems may be more common than where this would be the only point of failure when directories are messed up.
So the problem is when you have a file being copied onto itself either:
.so
fileos.unlink
on the files which have the same namecontinue
if the filename is the sameThis seems like a trivial thing to document, and since the distutils
backend actual does unconditionally replace the file, I guess it could be added. Personally I like the more explicit error, pointing out that there is a name clash, and forcing the user to take action. For me, the "solution" here is probably documentation (which is not often read, however), so the user knows to write small wrapper scripts (like the rm && f2py
one) instead of relying on f2py
's argument handling and defaults.
That is, given that this:
cat << EOF > fib.f
SUBROUTINE FIB(A,N)
C
C CALCULATE FIRST N FIBONACCI NUMBERS
C
INTEGER N
REAL*8 A(N)
DO I=1,N
IF (I.EQ.1) THEN
A(I) = 0.0D0
ELSEIF (I.EQ.2) THEN
A(I) = 1.0D0
ELSE
A(I) = A(I-1) + A(I-2)
ENDIF
ENDDO
END
EOF
# No errors no matter how many times it is run, overwrites automatically
python -m numpy.f2py -c fib.f -m fib --backend distutils
# Will error out if the .so is present
python -m numpy.f2py -c fib.f -m fib --backend meson
So it is OK to remove the file to be consistent with distutils
. The fix I thought of was just simply:
# Same behavior as distutils, just more explicit
rm -rf *.so && python -m numpy.f2py -c fib.f -m fib --backend meson
Which seems so trivially simple I wouldn't have normally bothered adding automatic file overwriting (which the user will then need to remember). For example, I forgot distutils
overwrites the .so
file. I would have to check the documentation to remember this. With the meson
backend, the behavior is explicit and I don't need to check anything other than the command output.
I found the
numpy
community always extremely forthcoming and helpful making things work, but I have the impression the focus has now shifted toward a distribution-based support approach alone. Whereas I appreciate the push toward new technologies (as I often like to do myself to the frustration of my collaborators and students), I think many users would appreciate having easy ways to continue being able to just do local development, and, ideally, continue using their existing setups as much as possible. This is also to give users the room to make switches to new technology on their own timeframe, as much as possible; having to adjust and rewrite lots of code at times or on timescales controlled by others - in part w/o need - is rather frustrating and discouraging.
I'm sorry it seems this way, but the bottleneck is really in the frontend. All of these would be trivial if adding flags was much easier, but it isn't because of #25179. The migration from distutils
just highlighted this bottleneck a lot, but hopefully over time this will get better. If flag additions were easy, I'd just as well support all these options. However at the moment, almost every change to f2py2e
stirs up a hornet's nest of bugs.
Personally, I'm always glad there's engagement from users like yourself, and the distribution based slant in development is just because for the most part (still) the biggest consumer is scipy
and its main problems revolve around distribution, not user development ease.
At the end of the day its about how much additional effort it takes to support each workflow at the user level. For example, if we change to unconditionally overwrite the .so
file, the user who doesn't want that behavior has to write their own logic to handle checking it exists and then raising a warning / error. This seemed like more effort than the current "wrap rm
" approach the alternative requires.
Also pinging @melissawm for another F2PY maintainer perspective :)
@HaoZeke Thank you for your detailed reply.
I probably should have made a PR from the original post, but then was unavailable for a few weeks due to an accident, hence the long silence.
My use case is that I have a detailed Python
module that compiles a library in Fortran
, but does automatic re-build if anything changes - any source file, NumPy
version, or Python
version. This is very convenient, not even having to call a build system, just load the module. That module generated the many input flags for both calls to f2py
(generating signature and final compilation), depending on project settings, etc. This now has to be adjusted to generate the input for meson
and call it. And I have to learn meson
, which seems a little bit magic how it finds all the module dependencies automatically, a bit too much magic for me. My module is used for a range of different projects - just in general, to write Fortran
code that can be called from Python
--- even individual functions and module data ---, with the output easily accessed and plotted directly in Python
, all set up in a few minutes (< 1 hour) for a new projects --- instead of having Fortran write out files that then have to be read in in NumPy
after starting Python
, and every time the Fortran code is changed, one first has to build, and run ... . (Maybe, some time, when all the other private modules it depends upon can be moved to github, so can this module, which will make review of issues a lot easier.)
... I am sure I will find a way around in the end.
Thanks for all your support.
@HaoZeke Thank you for your detailed reply.
I probably should have made a PR from the original post, but then was unavailable for a few weeks due to an accident, hence the long silence.
I'm very sorry to hear about that, I hope you are in better health now.
My use case is that I have a detailed
Python
module that compiles a library inFortran
, but does automatic re-build if anything changes - any source file,NumPy
version, orPython
version. This is very convenient, not even having to call a build system, just load the module. That module generated the many input flags for both calls tof2py
(generating signature and final compilation), depending on project settings, etc. This now has to be adjusted to generate the input formeson
and call it. And I have to learnmeson
, which seems a little bit magic how it finds all the module dependencies automatically, a bit too much magic for me. My module is used for a range of different projects - just in general, to writeFortran
code that can be called fromPython
--- even individual functions and module data ---, with the output easily accessed and plotted directly inPython
, all set up in a few minutes (< 1 hour) for a new projects --- instead of having Fortran write out files that then have to be read in inNumPy
after startingPython
, and every time the Fortran code is changed, one first has to build, and run ... . (Maybe, some time, when all the other private modules it depends upon can be moved to github, so can this module, which will make review of issues a lot easier.) ... I am sure I will find a way around in the end.Thanks for all your support.
Would this be calling numpy.f2py.compile()
? That's been removed in the upcoming 2.x
releases.. Though we are looking for more robust alternatives. If it does become public it would be much easier to debug, but I think these discussions are also very useful.
@2sn, we've restored the previous behavior, i.e. .so
files will be overwritten by default.
@HaoZeke I think this error had resulted from the .pyf
file when I first encountered it. I will check. When I write my own meson
file, however, I assume all of these issue may go away, and I only need the call to make the signature and wrapper files.
As for calling f2py
, I just use the direct call to f2py
from the shell subprocess.run
.
As for making my packages available on github/PyPI, there are too many small and interconnected packages I wrote over the years to make this an easy task, and splitting them up in the independent way they should be, makes handling dozens of git repositories quite a pain compared to just one private one, considering that development changes usually involve several packages. There must be a better way.
@2sn, we've restored the previous behavior, i.e.
.so
files will be overwritten by default.
FWIW, I think that that is the expected behavior. It matches what any other build steps/systems do for generated files when the source file changes.
FWIW, I think that that is the expected behavior. It matches what any other build steps/systems do for generated files when the source file changes.
yes, but not how Fortran
code executables are traditionally run, that they build automatically. Sure, one could just put the execution into the build system as well and define an alias, maybe an extra script wrapper for command line parameters. Default build systems many not re-build when the compiler version (or Python
or NumPy
) changes, but probably can be added as well with a bit of trickery.
Describe the issue:
In python 3.12 there seems to be an issue with directory handling that is not present in 3.11. In the very least it does behave differently. It seems to be ignoring some path specifications as well and created files in directories where there were none before, ignoring path specifications. Same numpy version, only difference is Python 3.12 instead of 3.11.6.
Likely related to
meson
backend, and that I use a runtime parameter combination that may not have been used/tested before/otherwise.Maybe this can be fixed easily by someone familiar with the meson backend. (or advise what I am doing wrong - that said, it does work with Python 3.11 and compatibility may be preferable)
Reproduce the code example:
Runtime information:
1.26.0 3.12.0 (main, Oct 7 2023, 10:56:13) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)]
Python 3.12
built from source on Fedora 38,numpy
installed frompip
.Context for the issue:
My f2py module no longer compiles. Can't use Python 3.12.