Failure to build memray

EricVogl-carta commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

Memray is failing to build from source via the instructions in the README.

I'm doing this in a aarch64 debian-based Docker image, python:3.10.12-bookworm. The libraries liblz4-dev and libunwind-dev are installed. A Dockerfile is included under steps to reproduce.

When building Memray, I'm seeing the following error:

#0 6.409 Requirement already satisfied: pip in /memray-env/lib/python3.10/site-packages (23.0.1)
#0 6.529 Collecting pip
#0 6.655   Downloading pip-23.2-py3-none-any.whl (2.1 MB)
#0 6.991      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 6.2 MB/s eta 0:00:00
#0 7.015 Installing collected packages: pip
#0 7.015   Attempting uninstall: pip
#0 7.015     Found existing installation: pip 23.0.1
#0 7.099     Uninstalling pip-23.0.1:
#0 7.103       Successfully uninstalled pip-23.0.1
#0 7.814 Successfully installed pip-23.2
#0 8.162 Obtaining file:///memray
#0 8.164   Installing build dependencies: started
#0 10.18   Installing build dependencies: finished with status 'done'
#0 10.18   Checking if build backend supports build_editable: started
#0 10.28   Checking if build backend supports build_editable: finished with status 'done'
#0 10.28   Getting requirements to build editable: started
#0 12.01   Getting requirements to build editable: finished with status 'error'
#0 12.02   error: subprocess-exited-with-error
#0 12.02
#0 12.02   × Getting requirements to build editable did not run successfully.
#0 12.02   │ exit code: 1
#0 12.02   ╰─> [43 lines of output]
#0 12.02       Compiling src/memray/_memray.pyx because it changed.
#0 12.02       Compiling src/memray/_memray_test_utils.pyx because it changed.
#0 12.02       [1/2] Cythonizing src/memray/_memray.pyx
#0 12.02       warning: src/memray/_memray.pyx:863:8: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
#0 12.02       warning: src/memray/_memray.pyx:509:14: Rvalue-reference as function argument not supported
#0 12.02       [2/2] Cythonizing src/memray/_memray_test_utils.pyx
#0 12.02       warning: src/memray/_memray_test_utils.pyx:50:0: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
#0 12.02       warning: src/memray/_memray_test_utils.pyx:57:4: The 'IF' statement is deprecated and will be removed in a future Cython version. Consider using runtime conditions or C macros instead. See https://github.com/cython/cython/issues/4310
#0 12.02
#0 12.02       Error compiling Cython file:
#0 12.02       ------------------------------------------------------------
#0 12.02       ...
#0 12.02               return self.ptr != NULL
#0 12.02
#0 12.02           @cython.profile(True)
#0 12.02           def run_in_pthread(self, callback):
#0 12.02               cdef pthread_t thread
#0 12.02               cdef int ret = pthread_create(&thread, NULL, &_pthread_worker, <void*>callback)
#0 12.02                                                            ^
#0 12.02       ------------------------------------------------------------
#0 12.02
#0 12.02       src/memray/_memray_test_utils.pyx:120:53: Cannot assign type 'void *(*)(void *) except * nogil' to 'void *(*)(void *) noexcept nogil'
#0 12.02       Traceback (most recent call last):
#0 12.02         File "/memray-env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#0 12.02           main()
#0 12.02         File "/memray-env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#0 12.02           json_out['return_val'] = hook(**hook_input['kwargs'])
#0 12.02         File "/memray-env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
#0 12.02           return hook(config_settings)
#0 12.02         File "/tmp/pip-build-env-2ferdb8m/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 450, in get_requires_for_build_editable
#0 12.02           return self.get_requires_for_build_wheel(config_settings)
#0 12.02         File "/tmp/pip-build-env-2ferdb8m/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
#0 12.02           return self._get_build_requires(config_settings, requirements=['wheel'])
#0 12.02         File "/tmp/pip-build-env-2ferdb8m/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
#0 12.02           self.run_setup()
#0 12.02         File "/tmp/pip-build-env-2ferdb8m/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
#0 12.02           exec(code, locals())
#0 12.02         File "<string>", line 309, in <module>
#0 12.02         File "/tmp/pip-build-env-2ferdb8m/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
#0 12.02           cythonize_one(*args)
#0 12.02         File "/tmp/pip-build-env-2ferdb8m/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
#0 12.02           raise CompileError(None, pyx_file)
#0 12.02       Cython.Compiler.Errors.CompileError: src/memray/_memray_test_utils.pyx
#0 12.02       [end of output]
#0 12.02
#0 12.02   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 12.02 error: subprocess-exited-with-error
#0 12.02
#0 12.02 × Getting requirements to build editable did not run successfully.
#0 12.02 │ exit code: 1
#0 12.02 ╰─> See above for output.
#0 12.02
#0 12.02 note: This error originates from a subprocess, and is likely not a problem with pip.

This also occurs when the wheel attempts to build from source when attempting to install memray via pip.

pip install memray==1.8.1

Expected Behavior

Memray builds per the instructions.

Steps To Reproduce

Create a Dockerfile with the following contents:
```
FROM python:3.10.12-bookworm
```

ARG TARGETPLATFORM ARG TARGETARCH RUN echo "Building for platform: $TARGETPLATFORM" RUN echo "Building for architecture: $TARGETARCH" ENV DEBIAN_FRONTEND=noninteractive ENV PYTHONUNBUFFERED=1

RUN apt-get update \ && apt-get install -y \ liblz4-dev \ libunwind-dev

SHELL ["/bin/bash", "-c"] RUN git clone --depth 1 --branch v1.8.1 https://github.com/bloomberg/memray.git memray \ && cd memray \ && python3 -m venv ../memray-env/ \ && source ../memray-env/bin/activate \ && python3 -m pip install --upgrade pip \ && python3 -m pip install -e . -r requirements-test.txt -r requirements-extra.txt


2. Build the Dockerfile

docker build -t .



### Memray Version

1.8.1

### Python Version

3.10

### Operating System

Linux

### Anything else?

The version of Cython used is not pinned. Cython 3.0.0 was released on 7/17/2023 - is this the issue?

godlygeek commented 1 year ago

The version of Cython used is not pinned. Cython 3.0.0 was released on 7/17/2023 - is this the issue?

Yep, that's exactly the issue, and the error message you're seeing is the same as #424. That's fixed on main, but we haven't cut a release with the fixed sdist yet.

You should be able to get past this if you do

git clone --depth 1 https://github.com/bloomberg/memray.git memray

without specifying a --branch.

If you need to build from a 1.8.1 sdist, I think it would work to use (installing a compatible version of Cython in your environment, and then using --no-build-isolation when building Memray so that it will pick up versions of dependencies from your environment).

python3 -m pip install setuptools wheel 'Cython<3'
python3 -m pip install --no-build-isolation memray

Hopefully one of those two options will unblock you until the fix makes it into a release.

dennis-glx commented 1 year ago

Noticed this same error for version 1.8.0 using Python3.8 as well.

godlygeek commented 1 year ago

Noticed this same error for version 1.8.0 using Python3.8 as well.

Yes, building any released version from source will behave this way until we can publish a new sdist. In the meantime, building from the main branch of a git clone is probably the easiest option. We'll get a new release out soon.

EricVogl-carta commented 1 year ago

Thanks, folks. Confirmed building from main works.

godlygeek commented 1 year ago

Memray 1.9.0 has just been published, and its sdist should be able to build successfully with Cython 3.0.

Thanks for the patience!

bloomberg / memray