Security issue running on osx-arm64 (Mac Mini M1): Cannot allocate write+execute memory for ffi.callback()

[sveinugu]

• [x ] I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

rpy2 running on Apple Silicon unsuccessfully tries to override MacOS X Big Sur security measure.

# python/conda version in `base` is `osx-64` (Intel), using Rosetta 2.
(base) $ CONDA_SUBDIR=osx-arm64 conda create -n rpy2_arm64 python=3 r-base rpy2
[...]
(base) $ conda activate rpy2_arm64
(rpy2_arm64) $ python -c 'from rpy2.robjects import r'                                                                                                                                                     

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/robjects/__init__.py", line 16, in <module>
    import rpy2.rinterface as rinterface
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface.py", line 13, in <module>
    import rpy2.rinterface_lib._rinterface_capi as _rinterface
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/_rinterface_capi.py", line 12, in <module>
    from rpy2.rinterface_lib import embedded
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/embedded.py", line 7, in <module>
    from rpy2.rinterface_lib import callbacks
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/callbacks.py", line 39, in <module>
    def _consoleflush() -> None:
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/ffi_proxy.py", line 44, in decorator
    res = _rinterface_cffi.ffi.callback(definition.callback_def)(func)
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

This issue is related to https://github.com/pyca/pyopenssl/issues/873, but PyOpenSSL is not installed. Rather, this is a problem with rpy2 itself.

https://cffi.readthedocs.io/en/latest/using.html#callbacks points to a workaround for Mac OS X (which I have been unable to understand how to apply), but states: "If you are using the out-of-line API mode, it is recommended to use the extern “Python” mechanism instead of callbacks".

Since version 3.2.0 (https://github.com/rpy2/rpy2/blob/4549736a37a3dd9da1044a9013398f54d04206c7/NEWS#L441), Rpy2 supports compilation to both ABI (default) and API modes, or BOTH:

• rpy2 can built and used with :mod:cffi's ABI or API modes (releases 3.0.x and 3.1.x were using the ABI mode exclusively). At the time of writing the default is still the ABI mode but the choice can be controlled through the environment variable RPY2_CFFI_MODE. If set, possible values are ABI (default if the environment variable is not set), API, or BOTH. When the latter, both API and ABI modes are built, and the choice of which one to use can be made at run time.

And it seems the extern "Python" mechanism is used in the API case, as per recommendations above: https://github.com/rpy2/rpy2/blob/85ffe7cd2a88e2ad0e735cc0078d6696e2e65658/rpy2/_rinterface_cffi_build.py#L207

Currently, however rpy2 is compiled (as default) using the ABI mode, not API or BOTH:

(rpy2_arm64) $ RPY2_CFFI_MODE=API python -c "from rpy2.robjects import r"                                                                                                                                    Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/robjects/__init__.py", line 16, in <module>
    import rpy2.rinterface as rinterface
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface.py", line 12, in <module>
    from rpy2.rinterface_lib import openrlib
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/openrlib.py", line 9, in <module>
    import _rinterface_cffi_api as _rinterface_cffi
ModuleNotFoundError: No module named '_rinterface_cffi_api'

I would assume compiling with RPY2_CFFI_MODE=BOTH would be the best option, in order to not break current behaviour.

Uninstalling rpy2 and installing from pip works, using the conda-forge compilers installed together with r-base:

(rpy2_arm64) $ conda uninstall rpy2
[...]

(rpy2_arm64) $ pip cache remove rpy2
[...]

(rpy2_arm64) $ RPY2_CFFI_MODE=BOTH pip install rpy2
[...]
Building wheels for collected packages: rpy2
  Building wheel for rpy2 (setup.py) ... done
  Created wheel for rpy2: filename=rpy2-3.4.3-cp39-cp39-macosx_11_0_arm64.whl size=222568 sha256=44389736c27197d9d6fb4632c3962e14e9238ba27c5d721942571ba5338a336a
  Stored in directory: /Users/sveinugu/Library/Caches/pip/wheels/4b/ce/bc/cfb5cc663ecd43edcdccd6efb71a58caacdf9fc22f59482a3a
Successfully built rpy2
Installing collected packages: rpy2
Successfully installed rpy2-3.4.3

(rpy2_arm64) $ RPY2_CFFI_MODE=ABI python -c "from rpy2 import robjects; robjects.r('print(\"Hello, world\")')"                                                                                             ~
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/robjects/__init__.py", line 16, in <module>
    import rpy2.rinterface as rinterface
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface.py", line 13, in <module>
    import rpy2.rinterface_lib._rinterface_capi as _rinterface
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/_rinterface_capi.py", line 12, in <module>
    from rpy2.rinterface_lib import embedded
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/embedded.py", line 7, in <module>
    from rpy2.rinterface_lib import callbacks
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/callbacks.py", line 39, in <module>
    def _consoleflush() -> None:
  File "/opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64/lib/python3.9/site-packages/rpy2/rinterface_lib/ffi_proxy.py", line 44, in decorator
    res = _rinterface_cffi.ffi.callback(definition.callback_def)(func)
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

(rpy2_arm64) $ RPY2_CFFI_MODE=API python -c "from rpy2 import robjects; robjects.r('print(\"Hello, world\")')"                                                                                           
[1] "Hello, world"

• [ ] Hence, setting RPY2_CFFI_MODE=BOTH at compilation time for rpy >= 3.2.0 should fix the issue. I do not have conda-forge workflow set up to test this.
• [ ] Also, should test_on_native_only be set to false in order to test this? Or are there other issues/concerns (I do not know the details of the conda-forge osx-arm64 compilation setup)?
  Environment (conda list):

$ conda list --canonical                                                                                                                                                                      ~
conda-forge/noarch::_r-mutex-1.0.1-anacondar_1
conda-forge/osx-arm64::bwidget-1.9.14-hce30654_0
conda-forge/osx-arm64::bzip2-1.0.8-h27ca646_4
conda-forge/osx-arm64::c-ares-1.17.1-h27ca646_1
conda-forge/osx-arm64::ca-certificates-2020.12.5-h4653dfc_0
conda-forge/osx-arm64::cairo-1.16.0-he69dfd1_1008
conda-forge/osx-arm64::cctools_osx-arm64-949.0.1-h1c8944f_20
conda-forge/osx-arm64::certifi-2020.12.5-py39h2804cbe_1
conda-forge/osx-arm64::cffi-1.14.5-py39h702c04f_0
conda-forge/osx-arm64::clang-11.1.0-hce30654_0
conda-forge/osx-arm64::clang-11-11.1.0-default_h87665d4_0
conda-forge/osx-arm64::clang_osx-arm64-11.1.0-h54d7cd3_0
conda-forge/osx-arm64::clangxx-11.1.0-default_hbe4449c_0
conda-forge/osx-arm64::clangxx_osx-arm64-11.1.0-hb84c830_0
conda-forge/osx-arm64::compiler-rt-11.1.0-h7d99272_0
conda-forge/noarch::compiler-rt_osx-arm64-11.1.0-h103ad0d_0
conda-forge/osx-arm64::curl-7.75.0-hb25ae9e_0
conda-forge/osx-arm64::fontconfig-2.13.1-h751047c_1004
conda-forge/osx-arm64::freetype-2.10.4-h17b34a0_1
conda-forge/osx-arm64::fribidi-1.0.10-h27ca646_0
conda-forge/osx-arm64::gettext-0.19.8.1-hea66d9f_1005
conda-forge/osx-arm64::gfortran_impl_osx-arm64-11.0.1.dev0-h4f36874_20
conda-forge/osx-arm64::gfortran_osx-arm64-11.0.1.dev0-h57527a5_14
conda-forge/osx-arm64::gmp-6.2.1-h9f76cd9_0
conda-forge/osx-arm64::graphite2-1.3.13-h9f76cd9_1001
conda-forge/osx-arm64::gsl-2.6-hc8a51ae_2
conda-forge/osx-arm64::harfbuzz-2.8.0-h2a75196_1
conda-forge/osx-arm64::icu-68.1-h17758a7_0
conda-forge/osx-arm64::isl-0.22.1-hb904e53_2
conda-forge/noarch::jinja2-2.11.3-pyh44b312d_0
conda-forge/osx-arm64::jpeg-9d-h27ca646_0
conda-forge/osx-arm64::krb5-1.17.2-h17618d6_0
conda-forge/osx-arm64::ld64_osx-arm64-530-h8a2aa15_20
conda-forge/osx-arm64::ldid-2.1.2-h34db0f2_2
conda-forge/osx-arm64::libblas-3.9.0-8_openblas
conda-forge/osx-arm64::libcblas-3.9.0-8_openblas
conda-forge/osx-arm64::libclang-cpp11.1-11.1.0-default_h87665d4_0
conda-forge/osx-arm64::libcurl-7.75.0-h222edf9_0
conda-forge/osx-arm64::libcxx-11.1.0-h168391b_0
conda-forge/osx-arm64::libedit-3.1.20191231-hc8eb9b7_2
conda-forge/osx-arm64::libev-4.33-h642e427_1
conda-forge/osx-arm64::libffi-3.3-h9f76cd9_2
conda-forge/osx-arm64::libgfortran-5.0.0.dev0-11_0_1_hf114ba7_20
conda-forge/noarch::libgfortran-devel_osx-arm64-11.0.1.dev0-hf114ba7_20
conda-forge/osx-arm64::libgfortran5-11.0.1.dev0-hf114ba7_20
conda-forge/osx-arm64::libglib-2.68.0-haaa185a_2
conda-forge/osx-arm64::libiconv-1.16-h642e427_0
conda-forge/osx-arm64::liblapack-3.9.0-8_openblas
conda-forge/osx-arm64::libllvm11-11.1.0-h4468dd5_0
conda-forge/osx-arm64::libnghttp2-1.43.0-hf3018f0_0
conda-forge/osx-arm64::libopenblas-0.3.12-openmp_h2ecc587_1
conda-forge/osx-arm64::libpng-1.6.37-hf7e6567_2
conda-forge/osx-arm64::libssh2-1.9.0-hb80f160_6
conda-forge/osx-arm64::libtiff-4.2.0-h70663a0_0
conda-forge/osx-arm64::libwebp-base-1.2.0-h27ca646_2
conda-forge/osx-arm64::libxml2-2.9.10-h8f9ca65_3
conda-forge/osx-arm64::llvm-openmp-11.1.0-hb3022d6_0
conda-forge/osx-arm64::llvm-tools-11.1.0-h4468dd5_0
conda-forge/osx-arm64::lz4-c-1.9.3-h9f76cd9_0
conda-forge/osx-arm64::make-4.3-he57ea6c_1
conda-forge/osx-arm64::markupsafe-1.1.1-py39h46acfd9_3
conda-forge/osx-arm64::mpc-1.1.0-hb760245_1009
conda-forge/osx-arm64::mpfr-4.0.2-hbc63f68_1
conda-forge/osx-arm64::ncurses-6.2-h9aa5885_4
conda-forge/osx-arm64::openssl-1.1.1j-h27ca646_0
conda-forge/osx-arm64::pango-1.42.4-h9aa5ae2_5
conda-forge/osx-arm64::pcre-8.44-hb904e53_0
conda-forge/osx-arm64::pcre2-10.36-hdd8d5aa_1
conda-forge/noarch::pip-21.0.1-pyhd8ed1ab_0
conda-forge/osx-arm64::pixman-0.40.0-h27ca646_0
conda-forge/noarch::pycparser-2.20-pyh9f0ad1d_2
conda-forge/osx-arm64::python-3.9.2-hcbd9b3a_0_cpython
conda-forge/osx-arm64::python_abi-3.9-1_cp39
conda-forge/noarch::pytz-2021.1-pyhd8ed1ab_0
conda-forge/osx-arm64::r-base-4.0.3-hebbb28a_7
conda-forge/osx-arm64::readline-8.1-hfbdcbf2_0
conda-forge/osx-arm64::rpy2-3.4.3-py39r40h038c1d3_0
conda-forge/osx-arm64::setuptools-49.6.0-py39h2804cbe_3
conda-forge/noarch::simplegeneric-0.8.1-py_1
conda-forge/osx-arm64::sqlite-3.35.2-h6d56c25_0
conda-forge/osx-arm64::tapi-1100.0.11-he4954df_0
conda-forge/osx-arm64::tk-8.6.10-hf7e6567_1
conda-forge/osx-arm64::tktable-2.10-h4161312_3
conda-forge/noarch::tzdata-2021a-he74cb21_0
conda-forge/noarch::tzlocal-2.1-pyh9f0ad1d_0
conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
conda-forge/osx-arm64::xz-5.2.5-h642e427_1
conda-forge/osx-arm64::zlib-1.2.11-h31e879b_1009
conda-forge/osx-arm64::zstd-1.4.9-h5b28eab_0

Details about conda and system ( conda info ):

``` $ conda info ~ active environment : rpy2_arm64 active env location : /opt/homebrew/Caskroom/miniforge/base/envs/rpy2_arm64 shell level : 5 user config file : /Users/sveinugu/.condarc populated config files : /opt/homebrew/Caskroom/miniforge/base/.condarc /Users/sveinugu/.condarc /Users/sveinugu/.condarc_py3 conda version : 4.9.2 conda-build version : not installed python version : 3.8.8.final.0 virtual packages : __osx=10.16=0 __unix=0=0 __archspec=1=x86_64 base environment : /opt/homebrew/Caskroom/miniforge/base (writable) channel URLs : https://conda.anaconda.org/conda-forge/osx-64 https://conda.anaconda.org/conda-forge/noarch https://conda.anaconda.org/bioconda/osx-64 https://conda.anaconda.org/bioconda/noarch https://repo.anaconda.com/pkgs/main/osx-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/osx-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /opt/homebrew/Caskroom/miniforge/base/pkgs /Users/sveinugu/.conda/pkgs envs directories : /opt/homebrew/Caskroom/miniforge/base/envs /Users/sveinugu/.conda/envs platform : osx-64 user-agent : conda/4.9.2 requests/2.25.1 CPython/3.8.8 Darwin/20.3.0 OSX/10.16 UID:GID : 502:20 netrc file : None offline mode : False ```

[sveinugu]

rpy2 still fails to run on Apple silicon on a clean install. Anyone able to test out a rebuild with the solution I suggested above? @ocefpaf? @xhochy?

[ocefpaf]

Looks like support was added in https://github.com/conda-forge/rpy2-feedstock/pull/71. I'm not a macOS users let alone an expert. Usually @xhochy and @isuruf know how to fix it but they are very busy and may take a while to get here.

[sveinugu]

@ocefpaf It is a bit strange, and it seems to me that compilation of rpy2 has not been tested on actual Apple Silicon. Which I believe it should be.

rpy2 is at least failing out of the box on two M1 machines I have tried, but adding the abovementioned RPY2_CFFI_MODE=BOTH environment variable prior to compilation (through pip) fixes it for me. I am a bit surprised that there are no other complaints about this, though...

[isuruf]

We can add RPY2_CFFI_MODE=BOTH when building. Please send a PR to add this

[isuruf]

It is a bit strange, and it seems to me that compilation of rpy2 has not been tested on actual Apple Silicon. Which I believe it should be.

We don't have the resources to do that. If you can provide Apple Silicon hardware to test them, we'd be glad to have them.

[ocefpaf]

@ocefpaf It is a bit strange, and it seems to me that compilation of rpy2 has not been tested on actual Apple Silicon. Which I believe it should be.

So do we but Apple hardware is too expensive for a bunch of volunteers doing free work to buy. If you have some and can test these for us please do! A PR to fix that is also welcomed!

[lgautier]

RPY2_CFFI_MODE=API is likely a better fix if building binary (compiled) packages. The ABI mode only exists to allow the installation of the source of package on machines without the tool chain or dependencies to build Python C extensions and R C extensions.