How to create platform portable wheels (manylinux) ?

When I was trying to upload the wheels to test.pypi.org then I was seeing:

$ python3.6 -m twine upload --repository-url https://test.pypi.org/legacy/ wheelhouse/NEURON-7.8-cp36-cp36m-linux_x86_64.whl
Enter your username: pramodk
Enter your password:
Uploading distributions to https://test.pypi.org/legacy/
Uploading NEURON-7.8-cp36-cp36m-linux_x86_64.whl
100%|██████████████████████████████████████████████████████████████████████| 3.00M/3.00M [00:01<00:00, 2.02MB/s]
NOTE: Try --verbose to see response content.
HTTPError: 400 Client Error: Binary wheel 'NEURON-7.8-cp36-cp36m-linux_x86_64.whl' 
has an unsupported platform tag 'linux_x86_64'. for url: https://test.pypi.org/legacy/

So I thought we have to create manylinux wheel. I created following Dockerfile based on manylinux platform:

FROM quay.io/pypa/manylinux2010_x86_64
MAINTAINER Pramod Kumbhar <pramod.s.kumbhar@gmail.com>

# Need to install sudo first:
RUN yum install -y sudo
# Can only run sudo from a terminal on CentOS 5.

# default arguments
ARG username=kumbhar
ARG password=kumbhar123
ARG git_name="Pramod Kumbhar"
ARG git_email="pramod.s.kumbhar@gmail.com"

# username password
ENV USERNAME $username
ENV PASSWORD $password
ENV GIT_NAME $git_name
ENV GIT_EMAIL $git_email

# Create a user called "$USERNAME", give it sudo access and remove the requirement for a password:
RUN useradd --create-home --shell /bin/bash --groups wheel ${USERNAME}
RUN bash -c 'echo "${USERNAME} ALL=(ALL:ALL) NOPASSWD: ALL" | (EDITOR="tee -a" visudo)'

# While we are still root, install the necessary dependencies for Malmo:
RUN yum -y install \
    git \
    python34-devel \
    swig \
    doxygen \
    libxslt \
    gcc-c++ \
    bzip2-devel \
    python34-tkinter \
    wget \
    software-properties-common \
    xpra \
    libgl1-mesa-dri \
    make \
    python34-pip \
    vim \
    curl \
    gfortran \
    unzip \
    bison \
    flex \
    pkg-config \
    autoconf \
    automake \
    make \
    python2.7-dev \
    libncurses-dev \
    openssh-server \
    libopenmpi-dev \
    libhdf5-serial-dev \
    python-minimal \
    libtool \
    cmake \
    zlib-devel

# Switch to the malmo user:
USER $USERNAME
WORKDIR /home/${USERNAME}

RUN git clone https://github.com/neuronsimulator/nrn.git -b cmake neuron-yale
RUN git clone https://github.com/bluebrain/nrn.git -b cmake neuron-bbp

RUN sudo yum -y install cmake3 ncurses-devel.x86_64
RUN sudo yum -y install rh-python36-python-devel.x86_64 rh-python35-python-devel.x86_64 python27-python-devel.x86_64
ENV PATH /home/${USERNAME}/.local/bin:$PATH

ENV PATH /opt/rh/python27/root/usr/bin:/opt/rh/rh-python35/root/usr/bin/:/opt/rh/rh-python36/root/usr/bin/:$PATH

ENV LD_LIBRARY_PATH /opt/rh/python27/root/usr/lib64:/opt/rh/rh-python35/root/usr/lib64:/opt/rh/rh-python36/root/usr/lib64:$LD_LIBRARY_PATH

# default software required
RUN  python3.6 -m pip install --user --upgrade scipy matplotlib bokeh ipython cython pytest wheel twine auditwheel

RUN mkdir -p neuron-yale/build-py37 \
    && cd neuron-yale/build-py37 \
    && cmake3 .. \
       -DCMAKE_INSTALL_PREFIX=`pwd`/install-py37 \
       -DCMAKE_BUILD_TYPE=Debug  \
       -DPYTHON_EXECUTABLE=`which python3.6` \
       -DNRN_ENABLE_CORENEURON=OFF \
       -DNRN_ENABLE_INTERVIEWS=OFF \
       -DNRN_ENABLE_PYTHON_DYNAMIC=OFF \
       -DNRN_ENABLE_MPI=OFF \
    && make -j 2 \
    && make install

WORKDIR /home/${USERNAME}/neuron-yale

RUN  git checkout setuppy \
     && python3.6 setup.py bdist_wheel

# repair wheel? see : https://malramsay.com/post/perils_of_packaging/
RUN python3.6 -m pip install --user --upgrade auditwheel
RUN LD_LIBRARY_PATH=`pwd`/_install/lib auditwheel repair dist/NEURON-7.8-cp36-cp36m-linux_x86_64.whl

And build this as:

docker build --build-arg username=kumbhar --build-arg password=kumbhar123 --build-arg git_name="Pramod Kumbhar" --build-arg git_email="pramod.s.kumbhar@gmail.com" -t neuron_manylinux .

With this we will have wheel:

→ docker run -it neuron_manylinux
$ ls wheelhouse/NEURON-7.8-cp36-cp36m-manylinux2010_x86_64.whl
wheelhouse/NEURON-7.8-cp36-cp36m-manylinux2010_x86_64.whl

And then I was able to successfully able to upload to test pypi server:

$ python3.6 -m twine upload --repository-url https://test.pypi.org/legacy/ wheelhouse/NEURON-7.8-cp36-cp36m-manylinux2010_x86_64.whl
Enter your username: pramodk
Enter your password:
Uploading distributions to https://test.pypi.org/legacy/
Uploading NEURON-7.8-cp36-cp36m-manylinux2010_x86_64.whl
100%|██████████████████████████████████████████████| 6.75M/6.75M [00:02<00:00, 3.03MB/s]

View at:
https://test.pypi.org/project/NEURON/7.8/

But when I tried to install this test wheel on BB5 (Red Had 7.4) I see:

$ pip install -i https://test.pypi.org/simple/ NEURON==7.8
Collecting NEURON==7.8
  Could not find a version that satisfies the requirement NEURON==7.8 (from versions: )
No matching distribution found for NEURON==7.8
You are using pip version 9.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ pip install https://test-files.pythonhosted.org/packages/11/22/7949b7b6a50bf758590e894457818fed35e4ae27de29e175bee829eda224/NEURON-7.8-cp36-cp36m-manylinux2010_x86_64.whl
NEURON-7.8-cp36-cp36m-manylinux2010_x86_64.whl is not a supported wheel on this platform.
You are using pip version 9.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

So definitely this is not what I thought manylinux is about.

@ferdonline : I didn't get time to look into this further. You know better and would be great if you could look into this when time permits.

cc: @ohm314

@tristan0x @mgeplf : you might have more experience on this (I am doing this first time). Do you have any suggestion?

~~I looks like you need to run auditwheel~~ Nevermind, I see you did. I'll have to look more carefully.

Are you sure the python installation is manylinux compatible? Can you try:

def is_manylinux1_compatible():
    # Only Linux, and only x86-64 / i686
    from distutils.util import get_platform
    if get_platform() not in ["linux-x86_64", "linux-i686"]:
        return False

    # Check for presence of _manylinux module
    try:
        import _manylinux
        return bool(_manylinux.manylinux1_compatible)
    except (ImportError, AttributeError):
        # Fall through to heuristic check below
        pass

    # Check glibc version. CentOS 5 uses glibc 2.5.
    return have_compatible_glibc(2, 5)

from https://www.python.org/dev/peps/pep-0513/

Do you happen to have the logs from auditwheel? I'm also surprised that you need to install it RUN python3.6 -m pip install --user --upgrade [...] auditwheel I'm also not sure injecting LD_LIBRARY_PATH=`pwd`/_install/lib plays well w/ auditwheel, but I've never tried.

Are you sure the python installation is manylinux compatible?

I would recommend using the /opt/python/* paths instead though, in case how the docker image is built changes, not the /opt/rh ones.

@pramodk: I just had a quick look, if I reduce the complexity, by: a) only using docker run -it -vpwd:/io quay.io/pypa/manylinux2010_x86_64 b) only installing:

yum -y install \
    bison \
    flex \
    libtool \
    readline-devel \
    cmake3 \

c) Using the manywheels python version: export PATH=/opt/python/cp37-cp37m/bin:$PATH d) And running:

pip install -U pip
pip install numpy cython
python3.7 setup.py bdist_wheel

I get this output:

...
Building lib to: /io/neuron-yale/_install
-- The C compiler identification is GNU 8.3.1
-- The CXX compiler identification is GNU 8.3.1
-- Check for working C compiler: /opt/rh/devtoolset-8/root/usr/bin/cc
-- Check for working C compiler: /opt/rh/devtoolset-8/root/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/rh/devtoolset-8/root/usr/bin/c++
-- Check for working CXX compiler: /opt/rh/devtoolset-8/root/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /opt/python/cp37-cp37m/bin/python3.7 (found version "3.7.5")
-- Found Cython: /opt/python/cp37-cp37m/bin/cython (found version "0.29.14")
-- Found BISON: /usr/bin/bison (found version "2.4.1")
-- Found FLEX: /usr/bin/flex (found version "2.5.35")
-- Found Readline: /usr/include
-- Found PythonInterp: /opt/python/cp37-cp37m/bin/python3.7 (found suitable version "3.7.5", minimum required is "3")
-- Found PythonLibs: python3.7m
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
...
-- --------------+--------------------------------------------------------------
--  Build option | Status
-- --------------+--------------------------------------------------------------
-- MPI           | OFF
-- Python        | ON
--   EXE         | /opt/python/cp37-cp37m/bin/python3.7
--   INC         | /opt/python/cp37-cp37m/include/python3.7m
--   LIB         | python3.7m
--   DYNAMIC     | OFF
-- RX3D          | OFF
-- Shared        | ON
-- Interviews    | OFF
-- CoreNEURON    | OFF
...
[ 99%] Linking CXX shared library libnrniv.so
/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: cannot find -lpython3.7m

This appears to be because the path to the libraries isn't being passed properly (ie: -L/opt/_internal/cpython-3.7.5/lib/python3.7/config-3.7m-x86_64-linux-gnu -L/opt/_internal/cpython-3.7.5/lib). However; this seems to conflict with the intent of DYNAMIC=OFF, no? Maybe I don't understand what that option means.

DYNAMIC=OFF (the default) is intended to mean that the nrnpython interface is linked into libnrniv.so and that the nrniv executable is linked to a specific python. I.e. generally usable only on the machine that built it unless the target machine has that exact python version and location (unless one employs a LD_LIBRARY_PATH that points to the correct python library) In the autotools, in addition to a LIB, there was a LIBDIR that gave the location of LIB

Ok, good to know. The method that the above is handled may need to change, as per the documentation of manylinux (https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1):

Note that libpythonX.Y.so.1 is not on the list of libraries that a manylinux1
extension is allowed to link to. Explicitly linking to libpythonX.Y.so.1 is
unnecessary in almost all cases: the way ELF linking works, extension modules
that are loaded into the interpreter automatically get access to all of the
interpreter's symbols, regardless of whether or not the extension itself is
explicitly linked against libpython. Furthermore, explicit linking to libpython
creates problems in the common configuration where Python is not built with
--enable-shared. In particular, on Debian and Ubuntu systems, apt install
pythonX.Y does not even install libpythonX.Y.so.1, meaning that any wheel that
did depend on libpythonX.Y.so.1 could fail to import.

One more aspect of DYNAMIC=OFF is perhaps relevant. I.e. when nrniv is launched, then the python library that was linked against is automatically loaded into memory by the launcher.

Thank you very much @mgeplf and @tristan0x for comments. I am bit late here but here are answers/details:

Are you sure the python installation is manylinux compatible? Can you try:

def is_manylinux1_compatible(): ...

I used method provided here and all versions seems manylinux compatible:

[kumbhar@2c47de99f5f6 neuron-yale]$ /opt/rh/python27/root/usr/bin/python2 test.py
True
[kumbhar@2c47de99f5f6 neuron-yale]$ /opt/rh/rh-python36/root/usr/bin/python test.py
True
[kumbhar@2c47de99f5f6 neuron-yale]$ /opt/python/cp36-cp36m/bin/python test.py
True
[kumbhar@2c47de99f5f6 neuron-yale]$ /opt/python/cp27-cp27m/bin/python test.py
True
[kumbhar@2c47de99f5f6 neuron-yale]$ /opt/python/cp38-cp38/bin/python test.py
True

I would recommend using the /opt/python/* paths instead though, in case how the docker image is built changes, not the /opt/rh ones.

The reason that I shifted from /opt/python/ is because of missinglibpython:

[kumbhar@2c47de99f5f6 neuron-yale]$ find /opt/rh/ -name "libpyt*"
/opt/rh/python27/root/usr/lib64/libpython2.7.so.1.0
/opt/rh/python27/root/usr/lib64/python2.7/config/libpython2.7.so
/opt/rh/python27/root/usr/lib64/libpython2.7.so

[kumbhar@2c47de99f5f6 neuron-yale]$ find /opt/python/ -name "libpyt*"

But I see the point of why libraries are not there in first place.

This appears to be because the path to the libraries isn't being passed properly (ie: -L/opt/_internal/cpython-3.7.5/lib/python3.7/config-3.7m-x86_64-linux-gnu -L/opt/_internal/cpython-3.7.5/lib).

I see that the python library paths are incomplete:

-- Python        | ON
--   EXE         | /opt/python/cp37-cp37m/bin/python3.7
--   INC         | /opt/python/cp37-cp37m/include/python3.7m
--   LIB         | python3.7m

We are using module from pybond11 to find python library paths and it is / was quite stable except in this manylinux platform where things are incomplete (?). I also tried autoconf build to see if it detect python libraries correctly if python is used from /opt/python/:

$ ../configure --prefix=`pwd`/install --without-iv --without-mpi --with-nrnpython=`which python3.6`
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... yes
....
checking for cos in -lm... yes
checking for python3... python3
Python binary found (/opt/python/cp36-cp36m/bin/python3.6)
checking nrnpython configuration... get_python_version()  '3.6'
sys.version_info[0]  '3'
get_python_inc(0)  '/opt/python/cp36-cp36m/include/python3.6m'
get_config_var('LIBS')  '-lpthread -ldl  -lutil -lrt'
get_config_var('LIBRARY')  'libpython3.6m.a'
get_config_var('LIBDIR')  '/opt/_internal/cpython-3.6.9/lib'
get_config_var('LIBS')  '-lpthread -ldl  -lutil -lrt'
get_config_var('MODLIBS')  '' returning ''
checking if python include files and libraries work... configure: error: could not run a test that used the python library.
Examine config.log to see error details. Something wrong with
    PYLIB=-L/opt/_internal/cpython-3.6.9/lib -lpython3.6m -lpthread -ldl  -lutil -lrt  -R/opt/_internal/cpython-3.6.9/lib
or
    PYLIBDIR=/opt/_internal/cpython-3.6.9/lib
or
    PYLIBLINK=-L/opt/_internal/cpython-3.6.9/lib -lpython3.6m -lpthread -ldl  -lutil -lrt
or
    PYINCDIR=/opt/python/cp36-cp36m/include/python3.6m

After reading Vtk and OpenCV related posts, I see that the common practice (suggestion) is -undefined dynamic_lookup (with clang) or -Wl,-undefined,dynamic_lookup (with gcc).

With this, I am able to build neuron with python from /opt/python/ and without linking to libpython*.so.

I still have to test these builds and see if wheels are usable on different platforms (will do this tomorrow).

Using -Wl,-undefined,dynamic_lookup I re-built wheel. No libraries is now explicitly linking to Python:

$ ldd _install/lib/libnrniv.so
ldd: warning: you do not have execution permission for `_install/lib/libnrniv.so'
    linux-vdso.so.1 =>  (0x00007ffc3ef8f000)
    libncurses.so.5 => /lib64/libncurses.so.5 (0x00007f869d7e8000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f869d4e2000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f869d25e000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f869d048000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f869ccb4000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f869cab0000)
    libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f869c88f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f869ded0000)

$ ldd _install/lib/libnrnpython3.so
ldd: warning: you do not have execution permission for `_install/lib/libnrnpython3.so'
    linux-vdso.so.1 =>  (0x00007ffe74764000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f82db3cb000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f82db147000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f82daf31000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f82dab9d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f82db911000)

$ ldd _install/lib/python/neuron/hoc.cpython-36m-x86_64-linux-gnu.so
ldd: warning: you do not have execution permission for `_install/lib/python/neuron/hoc.cpython-36m-x86_64-linux-gnu.so'
    linux-vdso.so.1 =>  (0x00007ffec8bc0000)
    libnrnpython3.so => /home/kumbhar/neuron-yale/_install/lib/libnrnpython3.so (0x00007f9c11497000)
    libnrniv.so => /home/kumbhar/neuron-yale/_install/lib/libnrniv.so (0x00007f9c10fd1000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f9c10ccb000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f9c10a47000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9c10831000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9c10614000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f9c10280000)
    libncurses.so.5 => /lib64/libncurses.so.5 (0x00007f9c1005e000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9c118da000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f9c0fe5a000)
    libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f9c0fc39000)

I uploaded new wheel to test.pypi but see same error while installing those on centos:

$ pip3.6 install https://test-files.pythonhosted.org/packages/14/29/eb14a73d9ab186e4d7ecab404f204984afe8d3e66bb0ed21b4f85259402a/NEURON-7.9-cp36-cp36m-manylinux2010_x86_64.whl
NEURON-7.9-cp36-cp36m-manylinux2010_x86_64.whl is not a supported wheel on this platform.

$ pip3.6 install https://test-files.pythonhosted.org/packages/4b/32/79691ca47d76efd6613372db7de3bcb820d19e1b2e4dd7f2e74eebc70431/NEURON-7.8.1-cp36-cp36m-manylinux2010_x86_64.whl
NEURON-7.8.1-cp36-cp36m-manylinux2010_x86_64.whl is not a supported wheel on this platform.

So there is still something else.

@mgeplf : In NSE stack, are you guys building wheels using manylinux platform?

Here is another update:

 pip3.6 install https://test-files.pythonhosted.org/packages/4b/32/79691ca47d76efd6613372db7de3bcb820d19e1b2e4dd7f2e74eebc70431/NEURON-7.8.1-cp36-cp36m-manylinux2010_x86_64.whl
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
NEURON-7.8.1-cp36-cp36m-manylinux2010_x86_64.whl is not a supported wheel on this platform.

Tried upgrading pip:

# pip3.6 install --upgrade pip
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting pip
  Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
    100% |################################| 1.4MB 1.0MB/s
Installing collected packages: pip
Successfully installed pip-19.3.1
[root@8085f3d20f78 /]# pip3.6 install https://test-files.pythonhosted.org/packages/4b/32/79691ca47d76efd6613372db7de3bcb820d19e1b2e4dd7f2e74eebc70431/NEURON-7.8.1-cp36-cp36m-manylinux2010_x86_64.whl
Traceback (most recent call last):
  File "/usr/bin/pip3.6", line 16, in <module>
    sys.exit(main())
TypeError: 'module' object is not callable

And then tried following which seems to work:

# python3.6 -m pip install https://test-files.pythonhosted.org/packages/4b/32/79691ca47d76efd6613372db7de3bcb820d19e1b2e4dd7f2e74eebc70431/NEURON-7.8.1-cp36-cp36m-manylinux2010_x86_64.whl
Collecting NEURON==7.8.1
  Downloading https://test-files.pythonhosted.org/packages/4b/32/79691ca47d76efd6613372db7de3bcb820d19e1b2e4dd7f2e74eebc70431/NEURON-7.8.1-cp36-cp36m-manylinux2010_x86_64.whl (4.5MB)
     |################################| 4.5MB 1.6MB/s
Collecting numpy>=1.13.1
  Downloading https://files.pythonhosted.org/packages/d2/ab/43e678759326f728de861edbef34b8e2ad1b1490505f20e0d1f0716c3bf4/numpy-1.17.4-cp36-cp36m-manylinux1_x86_64.whl (20.0MB)
     |################################| 20.0MB 2.3MB/s
Installing collected packages: numpy, NEURON
Successfully installed NEURON-7.8.1 numpy-1.17.4

but I still need to dig into:

# python3.6
Python 3.6.8 (default, Aug  7 2019, 17:28:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from neuron import h
Traceback (most recent call last):
  File "/usr/local/lib64/python3.6/site-packages/neuron/__init__.py", line 113, in <module>
    import hoc
ModuleNotFoundError: No module named 'hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib64/python3.6/site-packages/neuron/__init__.py", line 117, in <module>
    import neuron.hoc
ImportError: /usr/local/lib64/python3.6/site-packages/neuron/hoc.cpython-36m-x86_64-linux-gnu.so: undefined symbol: nrnpy_site_problem

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/python3.6/site-packages/neuron/__init__.py", line 119, in <module>
    exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1]))
  File "<string>", line 1, in <module>

Will do couple of more tests and will update here.

@pramodk: In NSE stack, are you guys building wheels using manylinux platform?

I have been using the manylinux1. We also transitioned from building them 'custom' (like above), to using the multibuild For example: https://github.com/BlueBrain/libsonata-wheels

From my experience, I would work out the bugs to make sure you can build the wheels by hand before moving to multibuild - it cleans things up, but also makes it harder to debug.

Can you post the output of auditwheel? I'm surprised it's converting a *cp36-cp36m-linux_x86_64.whl to *-cp36-cp36m-manylinux2010_x86_64.whl that doesn't work.

Also, can you make sure that pip3.6 and python3.6 are part of the same install?

Note; the traceback you posted here:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/python3.6/site-packages/neuron/__init__.py", line 119, in <module>
    exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1]))
  File "<string>", line 1, in <module>

Is trying to do a import neuron.hoc36, when module name probably (?) doesn't end in 36; Looking at the code, this seems to be because import neuron.hoc failed, which it seems to have w/: *** ImportError: /io/venv/lib/python3.6/site-packages/neuron/hoc.cpython-36m-x86_64-linux-gnu.so: undefined symbol: nrnpy_site_problem

I have resolved most of the issues but what is making bit problematic is link to other shared libraries like readline, ncurses etc from neuron.

If I avoid linking to these shared libraries then I get:

# python3.6 -c "import neuron"
Traceback (most recent call last):
  File "/myenv/lib64/python3.6/site-packages/neuron/__init__.py", line 113, in <module>
    import hoc
ModuleNotFoundError: No module named 'hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/myenv/lib64/python3.6/site-packages/neuron/__init__.py", line 117, in <module>
    import neuron.hoc
ImportError: /myenv/lib64/python3.6/site-packages/neuron/.libs/libnrniv-a30c83d0.so: undefined symbol: rl_event_hook

During handling of the above exception, another exception occurred:

obviously because we haven't linked to readline. If I pre-load (just for testing) then it works:

LD_PRELOAD=/usr/lib64/libreadline.so python3.6 -c "import neuron"

So in principal there is no issue with the wheel itself.

Linking to readline/ncurses libraries lead to other errors like this one. If I look into generate wheel, indeed there those .libs directories:

$ unzip ../wheelhouse/NEURON-7.8-cp36-cp36m-manylinux2010_x86_64.whl
...
  inflating: neuron/neuroml/neuroml.py
  inflating: neuron/.libs/libnrniv-55460046.so
  inflating: neuron/.libs/librxdmath-67eecf3d.so
  inflating: neuron/.libs/libreadline-5adeae88.so.6.0
  inflating: neuron/.libs/libtinfo-6ad58be8.so.5.7
....

I will pass by your desk tomorrow or Monday.

Is readline really a requirement? I guess it depends on the scope of what you're doing: if it's only a python library (and none of the auxiliary neuron tooling) I would try and drop that dep as it's possible that different pythons are using different version, depending on how they are compiled, and you may get conflicts.

Again, it would be helpful to see what auditwheel -v is saying; if you're getting a KeyError: '.libs', it might be easier to track down with that log.

Come by whenever, I'm busy until lunch, and Monday's are usually more busy than Fridays, but whatever works for you.

Huge thanks to @mgeplf and @ferdonline for debugging this issue of .libs directory while installing repaired wheel.

In summary: when nrniv is included into wheel, auditwheel detects that binary file is linked to libnrniv and other libraries and create additional .libs directory with multiple libraries:

  inflating: neuron/.libs/libnrniv-55460046.so
  inflating: neuron/.libs/librxdmath-67eecf3d.so
  inflating: neuron/.libs/libreadline-5adeae88.so.6.0
  inflating: neuron/.libs/libtinfo-6ad58be8.so.5.7

In the latest commit I disabled building nrniv binary for now. One of the suggestion was to write python shim for nrniv to avoid such issues for portable issues without workarounds.

Renaming linux wheel as manylinux works. But after trying wheel built on manylinux onto centos, using auditwheel repair is preferred to make sure libraries are correctly placed.

Here is docker image:

FROM quay.io/pypa/manylinux2010_x86_64
MAINTAINER Pramod Kumbhar <pramod.s.kumbhar@gmail.com>

# Need to install sudo first:
RUN yum install -y sudo
# Can only run sudo from a terminal on CentOS 5.

# While we are still root, install the necessary dependencies for Malmo:
RUN yum -y install \
    git \
    swig \
    doxygen \
    libxslt \
    gcc-c++ \
    bzip2-devel \
    python34-tkinter \
    wget \
    software-properties-common \
    xpra \
    libgl1-mesa-dri \
    make \
    vim \
    curl \
    gfortran \
    unzip \
    bison \
    flex \
    pkg-config \
    autoconf \
    automake \
    make \
    openssh-server \
    libopenmpi-dev \
    libhdf5-serial-dev \
    libtool \
    zlib-devel \
    cmake3 \
    ncurses-devel.x86_64 \
    gcc \
    patch \
    make

RUN sudo yum -y install cmake3 ncurses-devel.x86_64 gcc patch make
ENV PATH ${HOME}/.local/bin:$PATH

ENV PATH /opt/python/cp27-cp27m/bin:/opt/python/cp27-cp27mu/bin:/opt/python/cp35-cp35m/bin:/opt/python/cp36-cp36m/bin:/opt/python/cp37-cp37m/bin:$PATH

# default software required
RUN  python2.7 -m pip install --upgrade --user scipy matplotlib bokeh ipython cython pytest
RUN  python3.5 -m pip install --upgrade --user scipy matplotlib bokeh ipython cython pytest
RUN  python3.6 -m pip install --upgrade --user scipy matplotlib bokeh ipython cython pytest
RUN  python3.7 -m pip install --upgrade --user scipy matplotlib bokeh ipython cython pytest

# for wheel 
RUN  python2.7 -m pip install --upgrade --user twine auditwheel pip
RUN  python3.5 -m pip install --upgrade --user twine auditwheel pip
RUN  python3.6 -m pip install --upgrade --user twine auditwheel pip
RUN  python3.7 -m pip install --upgrade --user twine auditwheel pip

RUN git clone https://github.com/neuronsimulator/nrn.git -b setuppy neuron-yale

WORKDIR ${HOME}/neuron-yale
RUN git checkout setuppy
RUN git pull
RUN python2.7 setup.py bdist_wheel
RUN python3.5 setup.py bdist_wheel
RUN python3.6 setup.py bdist_wheel
RUN python3.7 setup.py bdist_wheel

# repair wheel? see : https://malramsay.com/post/perils_of_packaging/
RUN LD_LIBRARY_PATH=`pwd`/_install2717/lib python3.6 -m auditwheel repair dist/NEURON-7.8.11-cp27-cp27m-linux_x86_64.whl
RUN LD_LIBRARY_PATH=`pwd`/_install359/lib  python3.6 -m auditwheel repair dist/NEURON-7.8.11-cp35-cp35m-linux_x86_64.whl
RUN LD_LIBRARY_PATH=`pwd`/_install369/lib  python3.6 -m auditwheel repair dist/NEURON-7.8.11-cp36-cp36m-linux_x86_64.whl
RUN LD_LIBRARY_PATH=`pwd`/_install375/lib  python3.6 -m auditwheel repair dist/NEURON-7.8.11-cp37-cp37m-linux_x86_64.whl

and build image as:

docker build -t neuron_manylinux .

And this creates wheels successfully:

$ ls -lrt wheelhouse/
total 18172
-rw-r--r-- 1 kumbhar kumbhar 4505433 Nov 29 23:27 NEURON-7.8.11-cp27-cp27m-manylinux2010_x86_64.whl
-rw-r--r-- 1 kumbhar kumbhar 4657387 Nov 29 23:27 NEURON-7.8.11-cp35-cp35m-manylinux2010_x86_64.whl
-rw-r--r-- 1 kumbhar kumbhar 4718151 Nov 29 23:27 NEURON-7.8.11-cp36-cp36m-manylinux2010_x86_64.whl
-rw-r--r-- 1 kumbhar kumbhar 4721144 Nov 29 23:28 NEURON-7.8.11-cp37-cp37m-manylinux2010_x86_64.whl

@nrnhines : here are instructions to re-produce various issue with wheel that we discussed today. All are related to path detection :

clone setuppy branch of nrn

git clone git@github.com:neuronsimulator/nrn.git -b setuppy
cd nrn

Let's remove old/extra wheels

→ rm -rf dist/*

Use NRN_ENABLE_PYTHON_DYNAMIC=ON and NRN_ENABLE_BINARY=ON to build nrniv without linking python directly: update setup.py as:

→ git diff
diff --git a/setup.py b/setup.py
index 57911bd0..d28982fc 100644
--- a/setup.py
+++ b/setup.py
@@ -167,10 +167,10 @@ def setup_package():
             '-DNRN_ENABLE_CORENEURON=OFF',
             '-DNRN_ENABLE_INTERVIEWS=OFF',
             '-DNRN_ENABLE_RX3D=OFF',
-            '-DNRN_ENABLE_PYTHON_DYNAMIC=OFF',
+            '-DNRN_ENABLE_PYTHON_DYNAMIC=ON',
             '-DNRN_ENABLE_MPI=OFF',
             '-DLINK_AGAINST_PYTHON=OFF',
-            '-DNRN_ENABLE_BINARY=OFF',
+            '-DNRN_ENABLE_BINARY=ON',
         ],
         include_dirs = [
             neuron_root + "/include",

Build wheel with python3 (3.7 in my case)

→ python3.7 setup.py bdist_wheel
....
adding 'NEURON-7.8.11.dist-info/RECORD'
removing build/bdist.macosx-10.14-x86_64/wheel

This will create wheel in dist:

→ ls -lrt dist/
total 5312
-rw-r--r--  1 kumbhar  10067  2719361 Nov 30 23:01 NEURON-7.8.11-cp37-cp37m-macosx_10_14_x86_64.whl

Importantly, remove install and build directory used during wheel creation:

→ rm -rf build _install*

Create virtual environment and install wheel:

→ python3.7 -m venv myenv
→ . myenv/bin/activate
→ pip install dist/NEURON-7.8.11-cp37-cp37m-macosx_10_14_x86_64.whl
Processing ./dist/NEURON-7.8.11-cp37-cp37m-macosx_10_14_x86_64.whl
Collecting numpy>=1.13.1 (from NEURON==7.8.11)
  Using cached https://files.pythonhosted.org/packages/60/9a/a6b3168f2194fb468dcc4cf54c8344d1f514935006c3347ede198e968cb0/numpy-1.17.4-cp37-cp37m-macosx_10_9_x86_64.whl
Installing collected packages: numpy, NEURON
Successfully installed NEURON-7.8.11 numpy-1.17.4
You are using pip version 19.0.3, however version 19.3.1 is available.

Try wheel
- nrniv with oc : works but we see warning/errors

[ ] needs to be fixed


→ ./myenv/bin/nrniv
NEURON -- VERSION 7.8.0-160-g3ac62b5d+ setuppy (3ac62b5d+) 2019-11-30
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

bash: /Users/kumbhar/Downloads/tmp/nrn/_install374/share/nrn/../../bin/nrnpyenv.sh: No such file or directory failed: bash /Users/kumbhar/Downloads/tmp/nrn/_install374/share/nrn/../../bin/nrnpyenv.sh oc>sh: /Users/kumbhar/Downloads/tmp/nrn/_install374/share/nrn/lib/cleanup: No such file or directory


 * nrniv with python
- [ ] needs to be fixed


try with PYTHONHOME

→ PYTHONHOME=/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.7/ NRNHOME=pwd/myenv NEURONHOME=pwd/myenv/share/nrn ./myenv/bin/nrniv -python NEURON -- VERSION 7.8.0-160-g3ac62b5d+ setuppy (3ac62b5d+) 2019-11-30 Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019 See http://neuron.yale.edu/neuron/credits

ls: /Users/kumbhar/Downloads/tmp/nrn/myenv/bin/../lib/libpython*.dylib: No such file or directory

Launch with python

[ ] needs to be fixed


→ python3.7 -c "import neuron"
Traceback (most recent call last):
File "/Users/kumbhar/Downloads/tmp/nrn/myenv/lib/python3.7/site-packages/neuron/__init__.py", line 113, in <module>
import hoc
ModuleNotFoundError: No module named 'hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/kumbhar/Downloads/tmp/nrn/myenv/lib/python3.7/site-packages/neuron/init.py", line 117, in import neuron.hoc ImportError: dlopen(/Users/kumbhar/Downloads/tmp/nrn/myenv/lib/python3.7/site-packages/neuron/hoc.cpython-37m-darwin.so, 2): Symbol not found: _nrnpy_site_problem Referenced from: /Users/kumbhar/Downloads/tmp/nrn/myenv/lib/python3.7/site-packages/neuron/hoc.cpython-37m-darwin.so Expected in: flat namespace in /Users/kumbhar/Downloads/tmp/nrn/myenv/lib/python3.7/site-packages/neuron/hoc.cpython-37m-darwin.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/Users/kumbhar/Downloads/tmp/nrn/myenv/lib/python3.7/site-packages/neuron/init.py", line 119, in exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1])) File "", line 1, in ModuleNotFoundError: No module named 'neuron.hoc37'


* try with nrnivmodl
- [ ] needs to be fixed 
```bash
→ ./myenv/bin/nrnivmodl .
/Users/kumbhar/Downloads/tmp/nrn
ls: ./*.mod: No such file or directory
ls: ./*.inc: No such file or directory

make: /Users/kumbhar/Downloads/tmp/nrn/_install374/bin/nrnmech_makefile: No such file or directory
make: *** No rule to make target `/Users/kumbhar/Downloads/tmp/nrn/_install374/bin/nrnmech_makefile'.  Stop.

with env variables only it works:

→ NRNHOME=`pwd`/myenv NEURONHOME=`pwd`/myenv/share/nrn ./myenv/bin/nrnivmodl .
/Users/kumbhar/Downloads/tmp/nrn
ls: ./*.mod: No such file or directory
ls: ./*.inc: No such file or directory

 -> Compiling mod_func.c
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -O3 -DNDEBUG    -I/Users/kumbhar/Downloads/tmp/nrn/myenv/include   -fPIC -c mod_func.c -o mod_func.o
 => LINKING library x86_64/libnrnmech.0.0.dylib Mod files:
(cd .. ; /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -O3 -DNDEBUG -DVERSION_INFO='7.8.11' -std=c++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -fPIC -L/usr/local/opt/ruby/lib -I /Users/kumbhar/Downloads/tmp/nrn/myenv/include -o x86_64/libnrnmech.0.0.dylib -Wl,-install_name,@rpath/libnrnmech.0.0.dylib \
      x86_64/mod_func.o  -L/Users/kumbhar/Downloads/tmp/nrn/myenv/lib -lnrniv -Wl,-rpath,/Users/kumbhar/Downloads/tmp/nrn/myenv/lib   -lreadline)
(cd .. ; rm -f x86_64/.libs/libnrnmech.so ; mkdir -p x86_64/.libs ; ln -s ../../x86_64/libnrnmech.0.0.dylib x86_64/.libs/libnrnmech.so)
Successfully created x86_64/special

but note that CC and CXX compiler paths are hard-coded which needs to be fixed:

[ ] needs to be fixed

→ cat myenv/bin/nrnmech_makefile | grep "CC\|CXX"
CC = /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
CXX = /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++

We can simply avoid setting CC and CXX and assume that Makefile will use default C/C++ compiler?

The puzzle that I'm having is that the

hines@hines-T7500:~/neuron/setuppy$ python3.7 setup.py bdist_wheel

generates a hoc module that dynamically loads

hines@hines-T7500:~/neuron/setuppy$ ldd _install375/lib/python/neuron/hoc.cpython-37m-x86_64-linux-gnu.so 
    linux-vdso.so.1 (0x00007fffda07d000)
    libnrnpython3.so => /home/hines/neuron/setuppy/_install375/lib/libnrnpython3.so (0x00007f996c961000)
    libnrniv.so => /home/hines/neuron/setuppy/_install375/lib/libnrniv.so (0x00007f996c47b000)
...

but the

pip install setuppy/dist/NEURON-7.8.11-cp37-cp37m-linux_x86_64.whl

ends up with a hoc module that does NOT refer to libnrnpython3.so (it does refer to libnrniv.so) Note that libnrnpython3.so is installed in the same location as myenv/lib/libnrniv.so

I had success with

hines@hines-T7500:~/neuron/setuppy$ git diff setup.py
diff --git a/setup.py b/setup.py
index 57911bd0..9c37f147 100644
--- a/setup.py
+++ b/setup.py
@@ -154,7 +154,7 @@ def setup_package():
         ] if sys.platform[:6] == "darwin" else [
             "-Wl,-rpath,{}".format("$ORIGIN/../../../")
         ],
-        libraries = ["nrniv"]  # "nrnpython{}".format(sys.version_info[0]), "nrniv"],
+        libraries = ["nrniv","nrnpython3"]  # "nrnpython{}".format(sys.version_info[0]), "nrniv"],
     )

     extensions = [CMakeAugmentedExtension(
@@ -167,10 +167,10 @@ def setup_package():
             '-DNRN_ENABLE_CORENEURON=OFF',
             '-DNRN_ENABLE_INTERVIEWS=OFF',
             '-DNRN_ENABLE_RX3D=OFF',
-            '-DNRN_ENABLE_PYTHON_DYNAMIC=OFF',
+            '-DNRN_ENABLE_PYTHON_DYNAMIC=ON',
             '-DNRN_ENABLE_MPI=OFF',
             '-DLINK_AGAINST_PYTHON=OFF',
-            '-DNRN_ENABLE_BINARY=OFF',
+            '-DNRN_ENABLE_BINARY=ON',
         ],
         include_dirs = [
             neuron_root + "/include",
hines@hines-T7500:~/neuron/setuppy$

@nrnhines : my guess is that you haven't removed build & installed directories created during creation of wheel? i.e.

python3.7 setup.py bdist_wheel
rm -rf build _install*

once above directories are removed, you should be able to produce the warnings/errors:

→ python3.7 -m venv myenv
→ . myenv/bin/activate
→ pip install dist/NEURON-7.8.11-cp37-cp37m-macosx_10_14_x86_64.whl

→ nrniv
NEURON -- VERSION 7.8.0-160-g3ac62b5d+ setuppy (3ac62b5d+) 2019-11-30
....
bash: /Users/kumbhar/workarena/repos/bbp/nn/_install374/share/nrn/../../bin/nrnpyenv.sh: No such file or directory
failed: bash /Users/kumbhar/workarena/repos/bbp/nn/_install374/share/nrn/../../bin/nrnpyenv.sh
oc>

I'm not having a problem whether or not

rm -rf build _install*

is executed after

python3.8 setup.py bdist_wheel

I'm installing with pip using, in a fresh terminal,

rm -rf myenv
unset PYTHONPATH
python3.8 -m venv myenv
. myenv/bin/activate
pip install setuppy/dist/NEURON-7.8.11-cp38-cp38-linux_x86_64.whl 
export NEURONHOME=`pwd`/myenv/share/nrn
export PYTHONHOME="/home/hines/soft/Python-3.8.0/install"
export NRN_PYLIB="/home/hines/soft/Python-3.8.0/install/lib/libpython3.8.so.1.0"

This allows successful 'import neuron' with either a launch of python or nrniv. But my issues now are to get NEURONHOME setup correctly without an explicit environment variable and to get nrnpyenv.sh to return the correct values of PYTHONHOME and NRN_PYLIB. I believe that NEURONHOME can be robustly setup (if it is not already an environment variable) with the dlopen strategy. I hope that nrnpyenv.sh can be fixed up to do the right thing when python environments are being used.

When this simplest case is working then I think it will be straightforward to have InterViews dynamically loaded and it may be the case that dynamic loading of mpi/python will work as is with the robust NEURONHOME. Note that this is all for linux/mac. Windows is position independent with respect to installing with setup.exe but I believe it makes use of registry variables and I expect that for pip install, it might be able to use the same kind of methods as linux/mac. We'll see...

By the way, I think a lot of improvments are being blocked waiting for merging the .pramodk/cmake_master_merge into master. If we make a major change to that pull request the successful modeldb test will become stale and have to be done again with autotools and cmake. Currently that takes about 2*90 minutes followed by a half hour or more of manual labor to verify that the differences are not critical

By the way, I think a lot of improvments are being blocked waiting for merging the .pramodk/cmake_master_merge into master. If we make a major change to that pull request the successful modeldb test will become stale and have to be done again with autotools and cmake.

Agree. Taking a look at that right away!

Note from Fernando : On OSX we set following env variable to make wheel compatible with previous OS X releases:

export MACOSX_DEPLOYMENT_TARGET=10.9

@ferdonline @iomaganaris : This is on BB5 with Red Hat 7.6 and wheel is being built on manylinux1 by Fernando:

module load archive/2019-11
module load python
python -m venv env-36
. env-36/bin/activate

pip install dist/NEURON-7.8.1-cp36-cp36m-manylinux1_x86_64.whl

pip install dist/NEURON-7.8.1-cp36-cp36m-manylinux1_x86_64.whl
Processing ./dist/NEURON-7.8.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy>=1.13.1 (from NEURON==7.8.1)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/07/08/a549ba8b061005bb629b76adc000f3caaaf881028b963c2e18f811c6edc1/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl (20.2MB)
    100% |████████████████████████████████| 20.2MB 64kB/s
Installing collected packages: numpy, NEURON
Successfully installed NEURON-7.8.1 numpy-1.18.2

All installs fine. And now using it:

(env-36) kumbhar@bbpv1:~/tmp$ nrniv
NEURON -- VERSION 7.8.0-147-g21af5339+ python_wheel (21af5339+) 2020-03-27
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 mod//halfgap.mod
oc>quit()

(env-36) kumbhar@bbpv1:~/tmp$ nrniv -python
NEURON -- VERSION 7.8.0-147-g21af5339+ python_wheel (21af5339+) 2020-03-27
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 mod//halfgap.mod
>>> import neuron
>>>

(env-36) kumbhar@bbpv1:~/tmp$ python
Python 3.6.5 (default, Dec 19 2018, 05:00:45)
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import neuron
>>>

(env-36) kumbhar@bbpv1:~/tmp$ rm -rf x86_64/
(env-36) kumbhar@bbpv1:~/tmp$ nrnivmodl mod/
Creating x86_64 directory for .o files.

/gpfs/bbp.cscs.ch/home/kumbhar/tmp
ls: cannot access mod//*.inc: No such file or directory
mod//halfgap.mod
halfgap.mod
mkdir -p x86_64
 -> Compiling mod_func.c
/gpfs/bbp.cscs.ch/apps/hpc/jenkins/deploy/compilers/2018-12-18/linux-rhel7-x86_64/gcc-4.8.5/gcc-6.4.0-i6lyqfscua/bin/gcc -pthread -O2   -I. -I..   -I/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include   -fPIC -c mod_func.c -o mod_func.o
 -> NMODL halfgap.mod
MODLUNIT=/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
  /gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/bin/nocmodl halfgap.mod
Translating halfgap.mod into halfgap.c
Thread Safe
 -> Compiling x86_64/halfgap.c
(cd .. ; /gpfs/bbp.cscs.ch/apps/hpc/jenkins/deploy/compilers/2018-12-18/linux-rhel7-x86_64/gcc-4.8.5/gcc-6.4.0-i6lyqfscua/bin/gcc -pthread -O2   -I. -I..   -I/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include   -fPIC -c x86_64/halfgap.c -o x86_64/halfgap.o)
 => LINKING library x86_64/libnrnmech.0.0.so Mod files: halfgap.mod
(cd .. ; /gpfs/bbp.cscs.ch/apps/hpc/jenkins/deploy/compilers/2018-12-18/linux-rhel7-x86_64/gcc-4.8.5/gcc-6.4.0-i6lyqfscua/bin/g++ -pthread -O2 -DVERSION_INFO='7.8.1' -std=c++11 -shared -fPIC  -I /gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include -o x86_64/libnrnmech.0.0.so -Wl,-soname,libnrnmech.0.0.so \
  x86_64/mod_func.o x86_64/halfgap.o -L/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/lib -lnrniv -Wl,-rpath,/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/lib    -lreadline)
(cd .. ; rm -f x86_64/.libs/libnrnmech.so ; mkdir -p x86_64/.libs ; ln -s ../../x86_64/libnrnmech.0.0.so x86_64/.libs/libnrnmech.so)
Successfully created x86_64/special

## CHANGING GCC COMPILERS

(env-36) kumbhar@bbpv1:~/tmp$ CC=`which gcc` CXX=`which g++` nrnivmodl mod/
/gpfs/bbp.cscs.ch/home/kumbhar/tmp
ls: cannot access mod//*.inc: No such file or directory
mod//halfgap.mod
halfgap.mod
 -> Compiling mod_func.c
/usr/lib64/ccache/gcc -O2   -I. -I..   -I/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include   -fPIC -c mod_func.c -o mod_func.o
 -> NMODL halfgap.mod
MODLUNIT=/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
  /gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/bin/nocmodl halfgap.mod
Translating halfgap.mod into halfgap.c
Thread Safe
 -> Compiling x86_64/halfgap.c
(cd .. ; /usr/lib64/ccache/gcc -O2   -I. -I..   -I/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include   -fPIC -c x86_64/halfgap.c -o x86_64/halfgap.o)
 => LINKING library x86_64/libnrnmech.0.0.so Mod files: halfgap.mod
(cd .. ; /usr/lib64/ccache/g++ -O2 -DVERSION_INFO='7.8.1' -std=c++11 -shared -fPIC  -I /gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/include -o x86_64/libnrnmech.0.0.so -Wl,-soname,libnrnmech.0.0.so \
  x86_64/mod_func.o x86_64/halfgap.o -L/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/lib -lnrniv -Wl,-rpath,/gpfs/bbp.cscs.ch/home/kumbhar/tmp/env-36/lib/python3.6/site-packages/neuron/.data/lib    -lreadline)
(cd .. ; rm -f x86_64/.libs/libnrnmech.so ; mkdir -p x86_64/.libs ; ln -s ../../x86_64/libnrnmech.0.0.so x86_64/.libs/libnrnmech.so)
Successfully created x86_64/special

There is one improvement we should do is:

(env-36) kumbhar@bbpv1:~/tmp$ ./x86_64/special
NEURON -- VERSION 7.8.0-147-g21af5339+ python_wheel (21af5339+) 2020-03-27
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from /gpfs/bbp.cscs.ch/home/kumbhar/tmp/x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 mod//halfgap.mod
bash: /root/dev/nrn/nrn/build/cmake_install/share/nrn/../../bin/nrnpyenv.sh: Permission denied
failed: bash /root/dev/nrn/nrn/build/cmake_install/share/nrn/../../bin/nrnpyenv.sh
oc>

i.e. shim for bin/nrnpyenv.sh should be created similar to nrnivmodl. And then this will be ready for more wider testing!

Also, I pushed the Dockefile I have been using in : https://github.com/neuronsimulator/nrn/blob/python_wheel/dist/Dockerfile

The Python3.7-manylinux2010 wheel I built is working fine on BB5. But python 3.5 and 3.6 wheels shows errors like:

$ pip install dist/NEURON-7.8.11-cp36-cp36m-manylinux2010_x86_64.whl
NEURON-7.8.11-cp36-cp36m-manylinux2010_x86_64.whl is not a supported wheel on this platform.
You are using pip version 9.0.3, however version 20.0.2 is available.

Corresponding non-repaired wheels are installing fine:

$ pip install dist/NEURON-7.8.11-cp36-cp36m-linux_x86_64.whl
Processing ./dist/NEURON-7.8.11-cp36-cp36m-linux_x86_64.whl
Requirement already satisfied: numpy>=1.13.1 in ./env-36/lib/python3.6/site-packages (from NEURON==7.8.11)
Installing collected packages: NEURON
Successfully installed NEURON-7.8.11

This means something is missing in Dockerfile? Have to discuss / check this.

Even thought we could move to manulinux2010, manylinux1 is still the very standard. For instance to handle manylinux2010 we need at least pip 19.0. That might be at the root of some strange incompat messages and therefore I ended up setting up a manylinux1 environment. Either way I find myself now always doing a pip install -U setuptools pip for every new venv.

I will create and test wheel for various versions of Python. 3.5-3.8 ? At the same time I would like to try to generate a wheel with version tag abi3 so that we have a single wheel for all recent python3.
try to address existing warnings
Create a page (confluence?) on how to build these wheels, which require a patched auditwheel (for the Rpath append)

Even thought we could move to manulinux2010, manylinux1 is still the very standard. For instance to handle manylinux2010 we need at least pip 20.0.

I was using manulinux2010 just because I had docker recipe setup. We can definitely move to newer manylinux1.

I will create and test wheel for various versions of Python. 3.5-3.8 ?

that would be ideal!

At the same time I would like to try to generate a wheel with version tag abi3 so that we have a single wheel for all recent python3.

I don't know much details about this. certainly !

Create a page (confluence?) on how to build these wheels, which require a patched auditwheel (for the Rpath append)

I think NSE have documented something about this. For example here. May be we can extend / combine / inherit?

The newabi3 branch is a work in progress. It will reduce the number of hoc module shared libraries to just one hoc_abi3.so but because I haven't been able to get the cython rx3d files translatable to an abi3 compatible form, a separate .so for each of those is built for the -DNRN_PYTHON_DYNAMIC=semicolon;separated;list;of;pythons

I was using manulinux2010 just because I had docker recipe setup. We can definitely move to newer manylinux1.

There might be some confusion. manylinux1 is the good old (first?) manylinux image, based on centos5. I believe it still receives updates since it seems to be the most widespread form of manylinux, basically old pip's support it, and pretty much any "living" linux.

The newabi3 branch is a work in progress.

That sounds great! Good to know there is already work in that direction.

(Linking to issue #338 and the respective upstream Cython issue #2542)

There might be some confusion. manylinux1 is the good old (first?) manylinux image, based on centos5. I believe it still receives updates since it seems to be the most widespread form of manylinux, basically olf pip support it, and pretty much any "living" linux.

Oops, I thought it’s newer version! Anyway, once you build wheel we can test it on different platforms and as long it works, we should be fine!

I pushed wheels for 3.5-3.8

Btw my docker image is: docker.io/ferdonline/manylinux1_steroids_x86_64 You may want to create and mount a volume instead of working in the image, for the repo clone, etc...

@ferdonline : tested 3.6 and 3.7 and works great on BB5. Could we fix following?

bash: /root/dev/nrn/nrn/build/cmake_install/share/nrn/../../bin/nrnpyenv.sh: Permission denied
failed: bash /root/dev/nrn/nrn/build/cmake_install/share/nrn/../../bin/nrnpyenv.sh
oc>

You can reproduce by launching x86_64/special -python.

Testing NEURON wheel : Trail I

@ohm314 @jorblancoa @alexsavulescu @iomaganaris @ferdonline @nrnhines

Fernando has finished wheel and ready for first testing :star:

We have wheels created for Python 3.5, 3.6, 3.7 and 3.8. Goal is test on different Linux and OS X distributions (and not only BB5 because we have mostly tested it). So if you have local desktop, laptop, cluster node to test the compatibility, it would be great.

How to test?

First, make sure that python3 versions that you would like to test are added in $PATH.
Clone neuron repository and checkout branch python_wheel

git clone https://github.com/neuronsimulator/nrn.git -b python_wheel

If you are on OSX:

cd nrn
bash test.osx.sh

If you are on Linux:

cd nrn
bash test.linux.sh

On Ubuntu, you can have venv module as:

sudo apt-get install python3.6-venv python3.7-venv

I also saw error Python not available error when python-dev package was not installed:

++ ./x86_64/special -python -c 'import neuron; neuron.test(); neuron.test_rxd(); quit()'
NEURON -- VERSION 7.8.0-164-gca820e79+ python_wheel (ca820e79+) 2020-04-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from /home/kumbhar/nrn/x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
Python not available

@ferdonline @nrnhines : Is this expected? When I installed dev package, everything worked fine:

sudo apt-get install python3.7-dev

If you see any errors or warning, please report here.

My repository folder is named python_wheel on my linux machine. I have pythons managed by pyenv and the global version is Python 3.7.6. bash test.linux.sh fails. Note the folder for the pyenv shims is

2to3              f2py3     pip3              python2-config     python3-config
2to3-3.5          f2py3.5   pip3.5            python3            python-config
2to3-3.6          f2py3.6   pip3.6            python3.5          pyvenv
2to3-3.7          f2py3.7   pip3.7            python3.5-config   pyvenv-3.5
2to3-3.8          f2py3.8   pip3.8            python3.5m         pyvenv-3.6
bokeh             glia      pydoc             python3.5m-config  pyvenv-3.7
chardetect        idle      pydoc3            python3.6          smtpd.py
cygdb             idle3     pydoc3.5          python3.6-config   yaml_extract
cython            idle3.5   pydoc3.6          python3.6-gdb.py   yaml_get
cythonize         idle3.6   pydoc3.7          python3.6m         yaml_merge
easy_install      idle3.7   pydoc3.8          python3.6m-config  yaml_prune
easy_install-2.7  idle3.8   pygmentize        python3.7          yaml_put
easy_install-3.5  iptest    pytest            python3.7-config   yaml_rename
easy_install-3.6  iptest3   py.test           python3.7-gdb.py   yaml_transform
easy_install-3.7  ipython   python            python3.7m         yaml_validate
easy_install-3.8  ipython3  python2           python3.7m-config  yaml_walk
f2py              pip       python2.7         python3.8
f2py2             pip2      python2.7-config  python3.8-config
f2py2.7           pip2.7    python2.7-gdb.py  python3.8-gdb.py

The full output is:

+ set -e
+ export PATH=/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+ PATH=/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+ export PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+ PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+ for test_config in python3.8:wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl python3.7:wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl python3.6:wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl python3.5:wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
+ python_wheel=wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
+ python_exe=python3.8
++ command -v python3.8
+ '[' -x /home/hines/.pyenv/shims/python3.8 ']'
+ echo 'TESTING : python3.8 wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl'
TESTING : python3.8 wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
+ ./test_wheel.sh python3.8 wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
++ '[' 2 -ne 2 ']'
++ python_exe=python3.8
++ python_wheel=wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
++ venv_name=my_venv_python3.8
++ python3.8 -m venv my_venv_python3.8
++ . my_venv_python3.8/bin/activate
+++ deactivate nondestructive
+++ '[' -n '' ']'
+++ '[' -n '' ']'
+++ '[' -n /bin/bash -o -n '' ']'
+++ hash -r
+++ '[' -n '' ']'
+++ unset VIRTUAL_ENV
+++ '[' '!' nondestructive = nondestructive ']'
+++ VIRTUAL_ENV=/home/hines/neuron/python_wheel/my_venv_python3.8
+++ export VIRTUAL_ENV
+++ _OLD_VIRTUAL_PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+++ PATH=/home/hines/neuron/python_wheel/my_venv_python3.8/bin:/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+++ export PATH
+++ '[' -n '' ']'
+++ '[' -z '' ']'
+++ _OLD_VIRTUAL_PS1=
+++ '[' 'x(my_venv_python3.8) ' '!=' x ']'
+++ PS1='(my_venv_python3.8) '
+++ export PS1
+++ '[' -n /bin/bash -o -n '' ']'
+++ hash -r
++ pip install wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
Processing ./wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
Collecting numpy>=1.13.1 (from NEURON==7.8.11)
  Downloading https://files.pythonhosted.org/packages/2e/2f/5d2f9eb8ea6702966e31ca8a1f8515b34c240699fca389c6009fec919d56/numpy-1.18.2-cp38-cp38-manylinux1_x86_64.whl (20.6MB)
     |████████████████████████████████| 20.6MB 379kB/s 
Installing collected packages: numpy, NEURON
  Found existing installation: NEURON 7.8
    Not uninstalling neuron at /home/hines/neuron/nrnmpi/lib/python, outside environment /home/hines/neuron/python_wheel/my_venv_python3.8
    Can't uninstall 'NEURON'. No files were found to uninstall.
Successfully installed NEURON-7.8.11 numpy-1.18.2
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
++ '[' -d testcorenrn ']'
++ git clone https://github.com/nrnhines/testcorenrn.git
Cloning into 'testcorenrn'...
remote: Enumerating objects: 48, done.
remote: Total 48 (delta 0), reused 0 (delta 0), pack-reused 48
Unpacking objects: 100% (48/48), done.
++ python3.8 -c 'import neuron; neuron.test(); neuron.test_rxd()'
Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 106, in <module>
    import hoc
ModuleNotFoundError: No module named 'hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 110, in <module>
    import neuron.hoc
ModuleNotFoundError: No module named 'neuron.hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 112, in <module>
    exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1]))
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'neuron.hoc38'

@nrnhines : could you remove “set -e” and run tests again? i.e. run even on errors. I am curious if this is only case with python 3.8 or other versions as well.

I'm confused as to what the products are. So here is the output of bash test.linux.sh when set -e is commented out and starting from a clean repository (name is python_wheel) on branch python_wheel

hines@hines-T7500:~/neuron/python_wheel$ bash test.linux.sh
+ export PATH=/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+ PATH=/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+ export PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+ PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+ for test_config in python3.8:wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl python3.7:wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl python3.6:wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl python3.5:wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
+ python_wheel=wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
+ python_exe=python3.8
++ command -v python3.8
+ '[' -x /home/hines/.pyenv/shims/python3.8 ']'
+ echo 'TESTING : python3.8 wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl'
TESTING : python3.8 wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
+ ./test_wheel.sh python3.8 wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
++ '[' 2 -ne 2 ']'
++ python_exe=python3.8
++ python_wheel=wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
++ venv_name=my_venv_python3.8
++ python3.8 -m venv my_venv_python3.8
++ . my_venv_python3.8/bin/activate
+++ deactivate nondestructive
+++ '[' -n '' ']'
+++ '[' -n '' ']'
+++ '[' -n /bin/bash -o -n '' ']'
+++ hash -r
+++ '[' -n '' ']'
+++ unset VIRTUAL_ENV
+++ '[' '!' nondestructive = nondestructive ']'
+++ VIRTUAL_ENV=/home/hines/neuron/python_wheel/my_venv_python3.8
+++ export VIRTUAL_ENV
+++ _OLD_VIRTUAL_PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+++ PATH=/home/hines/neuron/python_wheel/my_venv_python3.8/bin:/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+++ export PATH
+++ '[' -n '' ']'
+++ '[' -z '' ']'
+++ _OLD_VIRTUAL_PS1=
+++ '[' 'x(my_venv_python3.8) ' '!=' x ']'
+++ PS1='(my_venv_python3.8) '
+++ export PS1
+++ '[' -n /bin/bash -o -n '' ']'
+++ hash -r
++ pip install wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
Processing ./wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl
Collecting numpy>=1.13.1 (from NEURON==7.8.11)
  Using cached https://files.pythonhosted.org/packages/2e/2f/5d2f9eb8ea6702966e31ca8a1f8515b34c240699fca389c6009fec919d56/numpy-1.18.2-cp38-cp38-manylinux1_x86_64.whl
Installing collected packages: numpy, NEURON
  Found existing installation: NEURON 7.8
    Not uninstalling neuron at /home/hines/neuron/nrnmpi/lib/python, outside environment /home/hines/neuron/python_wheel/my_venv_python3.8
    Can't uninstall 'NEURON'. No files were found to uninstall.
Successfully installed NEURON-7.8.11 numpy-1.18.2
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
++ '[' -d testcorenrn ']'
++ git clone https://github.com/nrnhines/testcorenrn.git
Cloning into 'testcorenrn'...
remote: Enumerating objects: 48, done.
remote: Total 48 (delta 0), reused 0 (delta 0), pack-reused 48
Unpacking objects: 100% (48/48), done.
++ python3.8 -c 'import neuron; neuron.test(); neuron.test_rxd()'
Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 106, in <module>
    import hoc
ModuleNotFoundError: No module named 'hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 110, in <module>
    import neuron.hoc
ModuleNotFoundError: No module named 'neuron.hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 112, in <module>
    exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1]))
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'neuron.hoc38'
+ for test_config in python3.8:wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl python3.7:wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl python3.6:wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl python3.5:wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
+ python_wheel=wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl
+ python_exe=python3.7
++ command -v python3.7
+ '[' -x /home/hines/.pyenv/shims/python3.7 ']'
+ echo 'TESTING : python3.7 wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl'
TESTING : python3.7 wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl
+ ./test_wheel.sh python3.7 wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl
++ '[' 2 -ne 2 ']'
++ python_exe=python3.7
++ python_wheel=wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl
++ venv_name=my_venv_python3.7
++ python3.7 -m venv my_venv_python3.7
++ . my_venv_python3.7/bin/activate
+++ deactivate nondestructive
+++ '[' -n '' ']'
+++ '[' -n '' ']'
+++ '[' -n /bin/bash -o -n '' ']'
+++ hash -r
+++ '[' -n '' ']'
+++ unset VIRTUAL_ENV
+++ '[' '!' nondestructive = nondestructive ']'
+++ VIRTUAL_ENV=/home/hines/neuron/python_wheel/my_venv_python3.7
+++ export VIRTUAL_ENV
+++ _OLD_VIRTUAL_PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+++ PATH=/home/hines/neuron/python_wheel/my_venv_python3.7/bin:/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+++ export PATH
+++ '[' -n '' ']'
+++ '[' -z '' ']'
+++ _OLD_VIRTUAL_PS1=
+++ '[' 'x(my_venv_python3.7) ' '!=' x ']'
+++ PS1='(my_venv_python3.7) '
+++ export PS1
+++ '[' -n /bin/bash -o -n '' ']'
+++ hash -r
++ pip install wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl
Processing ./wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl
Collecting numpy>=1.13.1 (from NEURON==7.8.11)
  Using cached https://files.pythonhosted.org/packages/b7/ce/d0b92f0283faa4da76ea82587ff9da70104e81f59ba14f76c87e4196254e/numpy-1.18.2-cp37-cp37m-manylinux1_x86_64.whl
Installing collected packages: numpy, NEURON
  Found existing installation: NEURON 7.8
    Not uninstalling neuron at /home/hines/neuron/nrnmpi/lib/python, outside environment /home/hines/neuron/python_wheel/my_venv_python3.7
    Can't uninstall 'NEURON'. No files were found to uninstall.
Successfully installed NEURON-7.8.11 numpy-1.18.2
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
++ '[' -d testcorenrn ']'
++ python3.7 -c 'import neuron; neuron.test(); neuron.test_rxd()'
testBytesize (neuron.tests.test_vector.VectorTestCase)
Test that Vector.__array_interface__ returns the proper bytesize (of a double) ... ok
testEndian (neuron.tests.test_vector.VectorTestCase)
Test that Vector.__array_interface__ returns the proper byteorder (endian) ... ok
testNumpyInteraction (neuron.tests.test_vector.VectorTestCase)
Testing numpy.array <=> hoc.Vector interaction ... ok
testPerformance (neuron.tests.test_vector.VectorTestCase)
Test performance of Vector<->list,array ... 

Executed "l = range(1000000)".  Elapsed = 0.000039 s
Executed "v = h.Vector(l)".  Elapsed = 0.232974 s
inplace:
Executed "v.from_python(l)".  Elapsed = 0.227108 s
Executed "a = numpy.array(v)".  Elapsed = 0.003660 s
inplace:
Executed "v.to_python(a)".  Elapsed = 0.002789 s
Executed "v2 = h.Vector(a)".  Elapsed = 0.005010 s
inplace:
Executed "v2.from_python(a)".  Elapsed = 0.002773 s
Executed "l2 = list(v2)".  Elapsed = 0.115118 s
inplace:
Executed "v.to_python(l2)".  Elapsed = 0.015055 s
Executed "v2 = h.Vector(a[::-1])".  Elapsed = 0.004451 s
Executed "a2 = numpy.array(v2)".  Elapsed = 0.003060 s
ok
testABI (neuron.tests.test_neuron.NeuronTestCase)
Test use of some  Py_LIMITED_API for python3. ... ok
testExtendedSection (neuron.tests.test_neuron.NeuronTestCase) ... test:  ok
testHClass (neuron.tests.test_neuron.NeuronTestCase)
Test subclass of hoc class. ... ok
testRxDexistence (neuron.tests.test_neuron.NeuronTestCase) ... scipy not available
ok
testpsection (neuron.tests.test_neuron.NeuronTestCase) ... ok

----------------------------------------------------------------------
Ran 9 tests in 4.198s

OK
test_ecs_diffusion_fixed_step (neuron.tests.test_rxd.RxDTestCase) ... ok
test_ecs_diffusion_variable_step_coarse (neuron.tests.test_rxd.RxDTestCase) ... ok
test_ecs_diffusion_variable_step_fine (neuron.tests.test_rxd.RxDTestCase) ... ok
test_rxd (neuron.tests.test_rxd.RxDTestCase) ... ok

----------------------------------------------------------------------
Ran 4 tests in 3.243s

OK
++ nrniv -c 'print "hello"'
NEURON -- VERSION 7.8.0-164-gca820e79+ python_wheel (ca820e79+) 2020-04-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

hello
++ '[' -d testcorenrn ']'
++ nrnivmodl testcorenrn
Creating x86_64 directory for .o files.

/home/hines/neuron/python_wheel
ls: cannot access 'testcorenrn/*.inc': No such file or directory
testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
Gfluct3.mod hhderiv.mod hhkin.mod hhwatch.mod nacum.mod
mkdir -p x86_64
 -> Compiling mod_func.c
gcc -O2   -I. -I..   -I/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/include   -fPIC -c mod_func.c -o mod_func.o
 -> NMODL Gfluct3.mod
MODLUNIT=/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
  /home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/bin/nocmodl Gfluct3.mod
 -> NMODL hhderiv.mod
MODLUNIT=/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
  /home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/bin/nocmodl hhderiv.mod
 -> NMODL hhkin.mod
MODLUNIT=/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
  /home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/bin/nocmodl hhkin.mod
Translating Gfluct3.mod into Gfluct3.c
Translating hhderiv.mod into hhderiv.c
Translating hhkin.mod into hhkin.c
Thread Safe
Thread Safe
Thread Safe
 -> NMODL hhwatch.mod
 -> NMODL nacum.mod
MODLUNIT=/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
  /home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/bin/nocmodl hhwatch.mod
 -> Compiling x86_64/Gfluct3.c
MODLUNIT=/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/share/nrn/lib/nrnunits.lib \
  /home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/bin/nocmodl nacum.mod
(cd .. ; gcc -O2   -I. -I..   -I/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/include   -fPIC -c x86_64/Gfluct3.c -o x86_64/Gfluct3.o)
Translating hhwatch.mod into hhwatch.c
Translating nacum.mod into nacum.c
Thread Safe
Thread Safe
 -> Compiling x86_64/hhderiv.c
(cd .. ; gcc -O2   -I. -I..   -I/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/include   -fPIC -c x86_64/hhderiv.c -o x86_64/hhderiv.o)
 -> Compiling x86_64/hhkin.c
(cd .. ; gcc -O2   -I. -I..   -I/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/include   -fPIC -c x86_64/hhkin.c -o x86_64/hhkin.o)
 -> Compiling x86_64/hhwatch.c
(cd .. ; gcc -O2   -I. -I..   -I/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/include   -fPIC -c x86_64/hhwatch.c -o x86_64/hhwatch.o)
 -> Compiling x86_64/nacum.c
(cd .. ; gcc -O2   -I. -I..   -I/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/include   -fPIC -c x86_64/nacum.c -o x86_64/nacum.o)
 => LINKING library x86_64/libnrnmech.0.0.so Mod files: Gfluct3.mod hhderiv.mod hhkin.mod hhwatch.mod nacum.mod
(cd .. ; g++ -O2 -DVERSION_INFO='7.8.11' -std=c++11 -shared -fPIC  -I /home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/include -o x86_64/libnrnmech.0.0.so -Wl,-soname,libnrnmech.0.0.so \
  x86_64/mod_func.o x86_64/Gfluct3.o x86_64/hhderiv.o x86_64/hhkin.o x86_64/hhwatch.o x86_64/nacum.o -L/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/lib -lnrniv -Wl,-rpath,/home/hines/neuron/python_wheel/my_venv_python3.7/lib/python3.7/site-packages/neuron/.data/lib    -lreadline)
(cd .. ; rm -f x86_64/.libs/libnrnmech.so ; mkdir -p x86_64/.libs ; ln -s ../../x86_64/libnrnmech.0.0.so x86_64/.libs/libnrnmech.so)
Successfully created x86_64/special
++ ./x86_64/special -python -c 'import neuron; neuron.test(); neuron.test_rxd(); quit()'
NEURON -- VERSION 7.8.0-164-gca820e79+ python_wheel (ca820e79+) 2020-04-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from /home/hines/neuron/python_wheel/x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
testBytesize (neuron.tests.test_vector.VectorTestCase)
Test that Vector.__array_interface__ returns the proper bytesize (of a double) ... ok
testEndian (neuron.tests.test_vector.VectorTestCase)
Test that Vector.__array_interface__ returns the proper byteorder (endian) ... ok
testNumpyInteraction (neuron.tests.test_vector.VectorTestCase)
Testing numpy.array <=> hoc.Vector interaction ... ok
testPerformance (neuron.tests.test_vector.VectorTestCase)
Test performance of Vector<->list,array ... 

Executed "l = range(1000000)".  Elapsed = 0.000049 s
Executed "v = h.Vector(l)".  Elapsed = 0.273958 s
inplace:
Executed "v.from_python(l)".  Elapsed = 0.269117 s
Executed "a = numpy.array(v)".  Elapsed = 0.003689 s
inplace:
Executed "v.to_python(a)".  Elapsed = 0.002844 s
Executed "v2 = h.Vector(a)".  Elapsed = 0.005253 s
inplace:
Executed "v2.from_python(a)".  Elapsed = 0.002895 s
Executed "l2 = list(v2)".  Elapsed = 0.122953 s
inplace:
Executed "v.to_python(l2)".  Elapsed = 0.014880 s
Executed "v2 = h.Vector(a[::-1])".  Elapsed = 0.004496 s
Executed "a2 = numpy.array(v2)".  Elapsed = 0.003044 s
ok
testABI (neuron.tests.test_neuron.NeuronTestCase)
Test use of some  Py_LIMITED_API for python3. ... ok
testExtendedSection (neuron.tests.test_neuron.NeuronTestCase) ... Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
ok
testHClass (neuron.tests.test_neuron.NeuronTestCase)
Test subclass of hoc class. ... ok
testRxDexistence (neuron.tests.test_neuron.NeuronTestCase) ... Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
FAIL
testpsection (neuron.tests.test_neuron.NeuronTestCase) ... Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
ok

======================================================================
FAIL: testRxDexistence (neuron.tests.test_neuron.NeuronTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/tests/test_neuron.py", line 119, in testRxDexistence
    assert(p.exitcode == 0)
AssertionError

----------------------------------------------------------------------
Ran 9 tests in 4.473s

FAILED (failures=1)
test_ecs_diffusion_fixed_step (neuron.tests.test_rxd.RxDTestCase) ... Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
FAIL
test_ecs_diffusion_variable_step_coarse (neuron.tests.test_rxd.RxDTestCase) ... Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
FAIL
test_ecs_diffusion_variable_step_fine (neuron.tests.test_rxd.RxDTestCase) ... Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
FAIL
test_rxd (neuron.tests.test_rxd.RxDTestCase) ... Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
FAIL

======================================================================
FAIL: test_ecs_diffusion_fixed_step (neuron.tests.test_rxd.RxDTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/tests/test_rxd.py", line 102, in test_ecs_diffusion_fixed_step
    assert(p.exitcode == 0)
AssertionError

======================================================================
FAIL: test_ecs_diffusion_variable_step_coarse (neuron.tests.test_rxd.RxDTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/tests/test_rxd.py", line 109, in test_ecs_diffusion_variable_step_coarse
    assert(p.exitcode == 0)
AssertionError

======================================================================
FAIL: test_ecs_diffusion_variable_step_fine (neuron.tests.test_rxd.RxDTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/tests/test_rxd.py", line 116, in test_ecs_diffusion_variable_step_fine
    assert(p.exitcode == 0)
AssertionError

======================================================================
FAIL: test_rxd (neuron.tests.test_rxd.RxDTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/tests/test_rxd.py", line 95, in test_rxd
    assert(p.exitcode == 0)
AssertionError

----------------------------------------------------------------------
Ran 4 tests in 1.197s

FAILED (failures=4)
++ ./x86_64/special -c 'print "hello"'
NEURON -- VERSION 7.8.0-164-gca820e79+ python_wheel (ca820e79+) 2020-04-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

loading membrane mechanisms from /home/hines/neuron/python_wheel/x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 testcorenrn/Gfluct3.mod testcorenrn/hhderiv.mod testcorenrn/hhkin.mod testcorenrn/hhwatch.mod testcorenrn/nacum.mod
hello
++ deactivate
++ '[' -n /gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin ']'
++ PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
++ export PATH
++ unset _OLD_VIRTUAL_PATH
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' '' = nondestructive ']'
++ unset -f deactivate
++ rm -rf my_venv_python3.7
+ for test_config in python3.8:wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl python3.7:wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl python3.6:wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl python3.5:wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
+ python_wheel=wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl
+ python_exe=python3.6
++ command -v python3.6
+ '[' -x /home/hines/.pyenv/shims/python3.6 ']'
+ echo 'TESTING : python3.6 wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl'
TESTING : python3.6 wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl
+ ./test_wheel.sh python3.6 wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl
++ '[' 2 -ne 2 ']'
++ python_exe=python3.6
++ python_wheel=wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl
++ venv_name=my_venv_python3.6
++ python3.6 -m venv my_venv_python3.6
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/home/hines/neuron/python_wheel/my_venv_python3.6/bin/python3.6', '-Im', 'ensurepip', '--upgrade', '--default-pip']

+ for test_config in python3.8:wheelhouse/NEURON-7.8.11-cp38-cp38-manylinux1_x86_64.whl python3.7:wheelhouse/NEURON-7.8.11-cp37-cp37m-manylinux1_x86_64.whl python3.6:wheelhouse/NEURON-7.8.11-cp36-cp36m-manylinux1_x86_64.whl python3.5:wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
+ python_wheel=wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
+ python_exe=python3.5
++ command -v python3.5
+ '[' -x /home/hines/.pyenv/shims/python3.5 ']'
+ echo 'TESTING : python3.5 wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl'
TESTING : python3.5 wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
+ ./test_wheel.sh python3.5 wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
++ '[' 2 -ne 2 ']'
++ python_exe=python3.5
++ python_wheel=wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
++ venv_name=my_venv_python3.5
++ python3.5 -m venv my_venv_python3.5
++ . my_venv_python3.5/bin/activate
+++ deactivate nondestructive
+++ '[' -n '' ']'
+++ '[' -n '' ']'
+++ '[' -n /bin/bash -o -n '' ']'
+++ hash -r
+++ '[' -n '' ']'
+++ unset VIRTUAL_ENV
+++ '[' '!' nondestructive = nondestructive ']'
+++ VIRTUAL_ENV=/home/hines/neuron/python_wheel/my_venv_python3.5
+++ export VIRTUAL_ENV
+++ _OLD_VIRTUAL_PATH=/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+++ PATH=/home/hines/neuron/python_wheel/my_venv_python3.5/bin:/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/bin:/opt/rh/rh-python36/root/bin/:/home/hines/.pyenv/plugins/pyenv-virtualenv/shims:/home/hines/.pyenv/shims:/home/hines/.pyenv/bin:/home/hines/soft/spack/bin:/home/hines/neuron/nrnmpi/x86_64/bin:/home/hines/bin:/home/hines/neuron/iv/x86_64/bin:/home/hines/soft/valgrind/bin:/usr/local/cuda-8.0/bin:/home/hines/bin:/opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/hines/soft/Python-3.8.0/install/bin:/home/hines/.local/bin
+++ export PATH
+++ '[' -n '' ']'
+++ '[' -z '' ']'
+++ _OLD_VIRTUAL_PS1=
+++ '[' 'x(my_venv_python3.5) ' '!=' x ']'
+++ PS1='(my_venv_python3.5) '
+++ export PS1
+++ '[' -n /bin/bash -o -n '' ']'
+++ hash -r
++ pip install wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
Processing ./wheelhouse/NEURON-7.8.11-cp35-cp35m-manylinux1_x86_64.whl
Collecting numpy>=1.13.1 (from NEURON==7.8.11)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/ff/18/c0b937e2f84095ae230196899e56d1d7d76c8e8424fb235ed7e5bb6d68af/numpy-1.18.2-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: numpy, NEURON
  Found existing installation: NEURON 7.8
    Not uninstalling neuron at /home/hines/neuron/nrnmpi/lib/python, outside environment /home/hines/neuron/python_wheel/my_venv_python3.5
Successfully installed NEURON-7.8.11 numpy-1.18.2
You are using pip version 9.0.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
++ '[' -d testcorenrn ']'
++ python3.5 -c 'import neuron; neuron.test(); neuron.test_rxd()'
Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 106, in <module>
    import hoc
ImportError: No module named 'hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 110, in <module>
    import neuron.hoc
ImportError: No module named 'neuron.hoc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/hines/neuron/nrnmpi/lib/python/neuron/__init__.py", line 112, in <module>
    exec("import neuron.hoc%d%d as hoc" % (sys.version_info[0], sys.version_info[1]))
  File "<string>", line 1, in <module>
ImportError: No module named 'neuron.hoc35'
+ echo --DONE--
--DONE--
hines@hines-T7500:~/neuron/python_wheel$

Ok - it seems like python 3.8 and 3.5 didn’t work at all. Python 3.7 and 3.6 partially worked.

Your one of the last fix https://github.com/neuronsimulator/nrn/pull/461 is not included in these wheels. I wonder if that will solve partial failures with 3.6 and 3.7.

For 3.8 and 3.5 we have to discuss a bit. Let’s do that next week.

Btw my docker image is: docker.io/ferdonline/manylinux1_steroids_x86_64

@ferdonline : could you also upload Dockerfile?

neuronsimulator / nrn

How to create platform portable wheels (manylinux) ? #329

Testing NEURON wheel : Trail I

How to test?