Meson failure installing numpy with graalpy-maven-plugin

zenbones commented 1 week ago

On windows 11, using the graalpy-maven-plugin...

`

org.graalvm.python graalpy-maven-plugin numpy .* ${project.basedir}/python-resources process-graalpy-resources

`

...yields this error from Meson...

`Collecting numpy Downloading numpy-2.0.2.tar.gz (18.9 MB) ---------------------------------------- 18.9/18.9 MB 3.1 MB/s eta 0:00:00 auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\include\numpy\ndarrayobject.h auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\include\numpy\npy_3kcompat.h auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\src\multiarray\compiled_base.c auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\src\multiarray\dlpack.c auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\src\multiarray\dtype_transfer.c auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\src\multiarray\methods.c auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\src\multiarray\stringdtype\dtype.c auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\src\umath\override.c auto-patching C API usages in C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\numpy_core\src\umath_rational_tests.c Looking for GraalPy patches for numpy Patching package numpy using C:\Users\david\AppData\Local\org.graalvm.polyglot\python\python-home\7f99e649dc51b21db2b83ce56bc9742eefe66415\lib-graalpython\patches\numpy\numpy-2.0.0.patch WARNING: GraalPy needs the 'patch' utility to apply compatibility patches. Please install it using your system's package manager. Installing build dependencies: started Installing build dependencies: still running... Installing build dependencies: still running... Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Installing backend dependencies: started Installing backend dependencies: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): still running... Preparing metadata (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error

? Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 2 ?─> [56 lines of output]

C:\Users\david\Documents\Nutshell\empyrean\aeon\pantheon\com\forio\epicenter\grid\worker\python-resources\venv\Scripts\graalpy.exe C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\meson.py setup C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b.mesonpy-46jrmj2j -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b.mesonpy-46jrmj2j\meson-python-native-file.ini The Meson build system Version: 1.4.99 Source dir: C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b Build dir: C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b.mesonpy-46jrmj2j Build type: native build Project name: NumPy Project version: 2.0.2 Traceback (most recent call last): File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\mesonmain.py", line 188, in run return options.run_func(options) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\msetup.py", line 363, in run app.generate() File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\msetup.py", line 187, in generate return self._generate(env, capture, vslite_ctx) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\msetup.py", line 209, in _generate intr = interpreter.Interpreter(b, user_defined_options=user_defined_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 320, in init self.parse_project() File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 129, in parse_project self.evaluate_codeblock(self.ast, end=1) File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 195, in evaluate_codeblock raise e File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 187, in evaluate_codeblock self.evaluate_statement(cur) File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 201, in evaluate_statement return self.function_call(cur) ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 528, in function_call res = func(node, func_args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreterbase\decorators.py", line 250, in wrapper return f(*nargs, *wrapped_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreterbase\decorators.py", line 569, in wrapper return f(wrapped_args, **wrapped_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 1308, in func_project mesonlib.setup_vsenv(force_vsenv) File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\utils\vsenv.py", line 121, in setup_vsenv return _setup_vsenv(force) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\utils\vsenv.py", line 78, in _setup_vsenv if windows_detect_native_arch() == 'arm64': ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\david\AppData\Local\Temp\pip-install-_qpjhkgz\numpy_cca38dee130b41b0884b1cf835bfb30b\vendored-meson\meson\mesonbuild\utils\universal.py", line 736, in windows_detect_native_arch process = ctypes.c_void_p(kernel32.GetCurrentProcess()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ OverflowError: can't convert negative value to unsigned int

..\meson.build:1:0: ERROR: Unhandled python exception
```
  This is a Meson bug and should be reported!
```
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

? Encountered error while generating package metadata. ?─> See above for output.`

msimacek commented 1 week ago

Hi, @zenbones, you need to install the patch utility so that pip can apply patches to add compatibility fixes. It says that in the output:

WARNING: GraalPy needs the 'patch' utility to apply compatibility patches. Please install it using your system's package manager.

Sorry that it's not more visible, in the next release it should be printed in color. Maybe we should make it hard error instead.

zenbones commented 3 days ago

Thanks. A few comments.

1) It's a warning, which generally means the process can work around his on its own, so yes, an error might be better. 2) The warning (or error) assumes I know what the 'patch' utility is, or how to install it properly. Maybe adding the shell (or other) installation instructions would help others in the same situation. This is not a utility I've ever had to deal with before in either Python or Java.

So... what's the patch utility (which I'm about to ask my web search client)?

zenbones commented 3 days ago

Having looked up the patch utility, the AI generated 'answer' is 'apt install patch' or 'brew install patch'... oh, hey, I'm on a windows system. Trying to find the equivalent, but, you know, any help appreciated.

msimacek commented 3 days ago

True, on Linux it's something that you typically already have and on Mac it's easy to install. This patching logic was added way before we had any windows support and we didn't properly re-evaluate it for windows. The patch tool should be available as a part of [Git for Windows], but we need to look at how to make it available so that the maven plugin would pick it up. We're also working on building the numpy wheel for windows, so you wouldn't have to do any of this.

timfel commented 3 days ago

FYI, I used the one from gnuwin32 on my machine

timfel commented 3 days ago

Note that this is the setup I have:

Install (at least) Visual Studio 2022 Build Tools (e.g. winget install Microsoft.VisualStudio.2022.BuildTools on a recent Win11)
Get patch.exe and put it somewhere you'll find it again, in a path without spaces
Start a "Developer PowerShell for VS 2022"
Put patch.exe on the PATH (e.g. $Env:PATH+=";c:\my\full\path\to\the\folder\that\containes\patchexe\")
Now use this prompt to make your graalpy venv and install packages

You can also fork the graalpy repository and run the github action to build windows wheels, download the resulting zip file, and pip install the numpy.whl file from that zip file. (see https://github.com/oracle/graalpython/blob/master/.github/workflows/build-windows-amd64-wheels.yml#L14)

oracle / graalpython

Meson failure installing numpy with graalpy-maven-plugin #434