Official s390x support

[seafolk-cn]

Hi,

I installed a Qiskit 0.30 env on an IBM LInuxONE machine and want to utilize kaleidoscope to get a good presentation. The installation of kaleidoscope is successful, but when i attempting to run the sample test code, i immediately get the "Segmentation fault (core dumped)" error, see below.

The maintainer of kaleidoscope confirmed that the issue located somewhere in llvmlite.
issue link: https://github.com/QuSTaR/kaleidoscope/issues/63

Can someone help on this?

Thanks!

(qiskit0.30) [root@rhel84 tutorials]# python Python 3.8.11 (default, Aug 6 2021, 10:33:43) [GCC 7.5.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.

        import numpy as np
        from qiskit import QuantumCircuit
        from qiskit.quantum_info import DensityMatrix
        import kaleidoscope.qiskit
        Segmentation fault (core dumped)
        (qiskit0.30) [root@rhel84 tutorials]#

Try again with fault handler

(qiskit0.30) [root@rhel84 tutorials]# python -q -X faulthandler

        import numpy as np
        from qiskit import QuantumCircuit
        from qiskit.quantum_info import DensityMatrix
        import kaleidoscope.qiskit
        Fatal Python error: Segmentation fault

Current thread 0x000003ff9fbf8240 (most recent call first): File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/ctypes/init.py", line 373 in init File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/llvmlite/binding/ffi.py", line 185 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1042 in _handle_fromlist File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/llvmlite/binding/dylib.py", line 3 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/llvmlite/binding/init.py", line 4 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/numba/core/config.py", line 16 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 1042 in _handle_fromlist File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/numba/init.py", line 19 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/kaleidoscope/interactive/bloch/utils.py", line 19 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/kaleidoscope/interactive/bloch/bloch2d.py", line 26 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/kaleidoscope/interactive/bloch/init.py", line 15 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 961 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/kaleidoscope/interactive/init.py", line 51 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "/opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages/kaleidoscope/init.py", line 26 in File "", line 219 in _call_with_frames_removed File "", line 843 in exec_module File "", line 671 in _load_unlocked File "", line 975 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 219 in _call_with_frames_removed File "", line 961 in _find_and_load_unlocked File "", line 991 in _find_and_load File "", line 1 in Segmentation fault (core dumped) (qiskit0.30) [root@rhel84 tutorials]#

[seafolk-cn]

package version in my env ....

(qiskit0.30) [root@rhel84 ~]# (qiskit0.30) [root@rhel84 ~]# conda list | grep llvmlite llvmlite 0.37.0 pypi_0 pypi (qiskit0.30) [root@rhel84 ~]# (qiskit0.30) [root@rhel84 ~]# conda list | grep numba numba 0.54.1 pypi_0 pypi (qiskit0.30) [root@rhel84 ~]#

[gmarkall]

What's the output of numba -s in your qiskit environment?

[seibert]

Just to check, does IBM Linux One have IBM z (called s390x on Debian) CPU?

[seafolk-cn]

Hi gmarkall,

What's the output of numba -s in your qiskit environment?

(qiskit0.30) [root@rhel84 ~]# uname -ar Linux rhel84 4.18.0-305.el8.s390x #1 SMP Thu Apr 29 09:06:01 EDT 2021 s390x s390x s390x GNU/Linux (qiskit0.30) [root@rhel84 ~]# (qiskit0.30) [root@rhel84 ~]# numba -s Segmentation fault (core dumped) (qiskit0.30) [root@rhel84 ~]# (qiskit0.30) [root@rhel84 ~]#

[seafolk-cn]

Hi seibert,

IBM LinuxONE has the same physical structure with IBM Z, just a new Brand for running Linux on the machine. Yes, it's arch is s390x.

We called the "CPU" on the machine as CP or IFL depending on the license ordered for the machine, if ordered CP then you can run z/OS and Linux(s390x) on the machine, if ordered IFL, then you can only run Linux(s390x) on the machine.

[seafolk-cn]

I can share a env, if someone would like to log onto the server to have a look! thanks!

[seafolk-cn]

So, Numba on s390x is not supported yet?

https://github.com/numba/numba

For more information about Numba, see the Numba homepage: https://numba.pydata.org Supported Platforms

Operating systems and CPUs:
    Linux: x86 (32-bit), x86_64, ppc64le (POWER8 and 9), ARMv7 (32-bit), ARMv8 (64-bit).
    Windows: x86, x86_64.
    macOS: x86_64, (M1/Arm64, unofficial support only).
    *BSD: (unofficial support only).
(Optional) Accelerators and GPUs:
    NVIDIA GPUs (Kepler architecture or later) via CUDA driver on Linux, Windows, macOS (< 10.14).

Dependencies

Python versions: 3.7-3.9
llvmlite 0.38.*
NumPy >=1.17 (can build with 1.11 for ABI compatibility).

Optionally:

SciPy >=1.0.0 (for numpy.linalg support).

[seibert]

Correct, Numba isn't tested on s390x and we don't know what bugs may exist in LLVM on that platform as well.

[stuartarchibald]

This this happens:

(qiskit0.30) [root@rhel84 ~]# numba -s
Segmentation fault (core dumped)

suggests that the DSOs are probably segfaulting on dlopen. IIRC the numba -s command doesn't attempt to compile anything but will load the llvmlite bindings to LLVM.

@seafolk-cn do either of the following work? :

• python -c "from llvmlite import binding"

• python -c "from llvmlite import binding; print(binding.get_host_cpu_name())"

Also, in the environment you have, what, if any, LLVM packages are there present?

Finally: https://github.com/QuSTaR/kaleidoscope/issues/63#issuecomment-945622345 suggests llvmlite came from pypi? Was this a pip install? Did it build from source?

[seafolk-cn]

This this happens:

(qiskit0.30) [root@rhel84 ~]# numba -s
Segmentation fault (core dumped)

suggests that the DSOs are probably segfaulting on dlopen. IIRC the numba -s command doesn't attempt to compile anything but will load the llvmlite bindings to LLVM.

@seafolk-cn do either of the following work? :

• python -c "from llvmlite import binding"

• python -c "from llvmlite import binding; print(binding.get_host_cpu_name())"

Also, in the environment you have, what, if any, LLVM packages are there present?

Finally: QuSTaR/kaleidoscope#63 (comment) suggests llvmlite came from pypi? Was this a pip install? Did it build from source?

@stuartarchibald

Thanks!

Below is my check.

(qiskit0.30) [root@qiskit qiskit-machine-learning]# (qiskit0.30) [root@qiskit qiskit-machine-learning]# conda list | grep llvm llvmlite 0.37.0 pypi_0 pypi (qiskit0.30) [root@qiskit qiskit-machine-learning]# (qiskit0.30) [root@qiskit qiskit-machine-learning]# rpm -qa | grep llvm llvm-test-11.0.0-2.module+el8.4.0+8598+a071fcd5.s390x llvm-devel-11.0.0-2.module+el8.4.0+8598+a071fcd5.s390x llvm-11.0.0-2.module+el8.4.0+8598+a071fcd5.s390x llvm-libs-11.0.0-2.module+el8.4.0+8598+a071fcd5.s390x llvm-static-11.0.0-2.module+el8.4.0+8598+a071fcd5.s390x (qiskit0.30) [root@qiskit qiskit-machine-learning]# conda deactivate (base) [root@qiskit qiskit-machine-learning]# (base) [root@qiskit qiskit-machine-learning]# python -c "from llvmlite import binding" Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'llvmlite' (base) [root@qiskit qiskit-machine-learning]# conda activate qiskit0.30 (qiskit0.30) [root@qiskit qiskit-machine-learning]# (qiskit0.30) [root@qiskit qiskit-machine-learning]# python -c "from llvmlite import binding" Segmentation fault (core dumped) (qiskit0.30) [root@qiskit qiskit-machine-learning]# (qiskit0.30) [root@qiskit qiskit-machine-learning]# python -c "from llvmlite import binding; print(binding.get_host_cpu_name())" Segmentation fault (core dumped) (qiskit0.30) [root@qiskit qiskit-machine-learning]#

here is part of the installation log of "pip install kaleidoscope" command, you can see that cached version wheels of numba and llvmlite for s390x arch were used, but we don't have s390x arch wheels on pypi website. so i guess it should be built from source as dependent install of some of the Qiskit component.

Requirement already satisfied: numpy>=1.15 in /opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages (from kaleidoscope) (1.21.2) Requirement already satisfied: seaborn>=0.9.0 in /opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages (from kaleidoscope) (0.11.2) Collecting numba>=0.46 Using cached numba-0.54.1-cp38-cp38-linux_s390x.whl Requirement already satisfied: matplotlib>=3.1 in /opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages (from kaleidoscope) (3.4.3) Requirement already satisfied: cycler>=0.10 in /opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages (from matplotlib>=3.1->kaleidoscope) (0.10.0

....................................

Requirement already satisfied: six in /opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages (from cycler>=0.10->matplotlib>=3.1->kaleidoscope) (1.16.0) Collecting llvmlite<0.38,>=0.37.0rc1 Using cached llvmlite-0.37.0-cp38-cp38-linux_s390x.whl Requirement already satisfied: setuptools in /opt/anaconda/envs/qiskit0.30/lib/python3.8/site-packages (from numba>=0.46->kaleidoscope) (58.0.4) Collecting numpy>=1.15 Using cached numpy-1.20.3-cp38-cp38-linux_s390x.whl ....................................................................................................................

[seafolk-cn]

Correct, Numba isn't tested on s390x and we don't know what bugs may exist in LLVM on that platform as well.

@seibert

Thanks for the confirmation!

We also don't know how many dependency already or will happen on these packages.

Do we have some test routine can run against every arch or need a CI env for the test?

Do this means big efforts for community to support s390x arch?

[seafolk-cn]

These two packages were installed from source during the installation of qiskit-machine-learning component.

Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/anaconda/envs/qiskit0.26_test/lib/python3.8/site-packages (from scikit-learn>=0.20.0->qiskit-machine-learning==0.1) (3.0.0) Requirement already satisfied: mpmath>=0.19 in /opt/anaconda/envs/qiskit0.26_test/lib/python3.8/site-packages (from sympy>=1.3->qiskit-terra>=0.17.0->qiskit-machine-learning==0.1) (1.2.1) Collecting numba>=0.49 Downloading numba-0.54.1.tar.gz (2.2 MB) |████████████████████████████████| 2.2 MB 578 kB/s Collecting llvmlite<0.38,>=0.37.0rc1 Downloading llvmlite-0.37.0.tar.gz (125 kB) |████████████████████████████████| 125 kB 546 kB/s Building wheels for collected packages: numba, llvmlite Building wheel for numba (setup.py) ... done Created wheel for numba: filename=numba-0.54.1-cp38-cp38-linux_s390x.whl size=3148779 sha256=92076b1740933ce8cee97c4eb4292555bf71c734679a60468a48ce075c70a181 Stored in directory: /root/.cache/pip/wheels/fa/30/84/81b40877f8ada05b9af846f350677d3b123614821b6d9ddc18 Building wheel for llvmlite (setup.py) ... done Created wheel for llvmlite: filename=llvmlite-0.37.0-cp38-cp38-linux_s390x.whl size=2453481 sha256=9d87198debd419e4098e10c4de1952d983ff28a7c41416c76638f20eb1e58a7a Stored in directory: /root/.cache/pip/wheels/18/b6/88/132a1f634a0953532715e35ce2567608ee96b82929eb6ebf8d Successfully built numba llvmlite Installing collected packages: llvmlite, numba, sparse, qiskit-machine-learning Successfully installed llvmlite-0.37.0 numba-0.54.1 qiskit-machine-learning-0.1.0 sparse-0.13.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[seafolk-cn]

sorry, wrong click!

[esc]

should we rename this to "feature request for s390x wheels?".

[gmarkall]

It sounds like it's more of a feature request for s390x support? It seems that we'd also need to start testing and potentially bug fixing on s390x before we build wheels?

[dongcc]

It sounds like it's more of a feature request for s390x support? It seems that we'd also need to start testing and potentially bug fixing on s390x before we build wheels?

I'm also need this feature, if there is a document describing how to do that, I'm willing to help to tested on my LinuxONE env.

[esc]

It sounds like it's more of a feature request for s390x support? It seems that we'd also need to start testing and potentially bug fixing on s390x before we build wheels?

Yes.

[musicinmybrain]

We don’t have Numba packaged in Fedora Linux, but we do have llvmlite 0.39.1, and s390x is a primary architecture for us. We find that it builds from source and the tests pass.

[esc]

We don’t have Numba packaged in Fedora Linux, but we do have llvmlite 0.39.1, and s390x is a primary architecture for us. We find that it builds from source and the tests pass.

The real test will be to see if the Numba test-suite passes. Would suggest to install the llvmlite package from the distribution and then to build and test numba from source against that. Most of the heavy lifting is in the llvmlite package so that should be a reasonably straightforward task.

[musicinmybrain]

The real test will be to see if the Numba test-suite passes. Would suggest to install the llvmlite package from the distribution and then to build and test numba from source against that. Most of the heavy lifting is in the llvmlite package so that should be a reasonably straightforward task.

That’s a good idea, but it’s easier said than done right now. Ignoring Fedora 36, which is about to go end-of-life, the supported Fedora releases (37 and 38) and Rawhide are all on Python 3.11, which I believe is only supported by Numba 0.40.0. However, we also still have TBB 2020.3, which is too old for that version of Numba. By the time TBB is updated in the Rawhide/F39 branch, that branch will also be on a pre-release of Python 3.12.

[esc]

The real test will be to see if the Numba test-suite passes. Would suggest to install the llvmlite package from the distribution and then to build and test numba from source against that. Most of the heavy lifting is in the llvmlite package so that should be a reasonably straightforward task.

That’s a good idea, but it’s easier said than done right now. Ignoring Fedora 36, which is about to go end-of-life, the supported Fedora releases (37 and 38) and Rawhide are all on Python 3.11, which I believe is only supported by Numba 0.40.0. However, we also still have TBB 2020.3, which is too old for that version of Numba. By the time TBB is updated in the Rawhide/F39 branch, that branch will also be on a pre-release of Python 3.12.

llvmlite 0.40.* is the first llvmlite to support Python 3.11 -- that is correct as per:

https://numba.readthedocs.io/en/stable/user/installing.html#version-support-information