NEURON MPI is not enabled in Github Linux runner

kmilo9999 commented 3 weeks ago

Context

While setting up my GitHub Actions workflow to use NEURON with MPI support on a Linux runner, I am encountering issues when trying to run parallel simulations. Specifically, the NEURON MPI support does not seem to be enabled, and my scripts indicate that NEURON is not utilizing MPI for parallel processes.

Overview of the CI issue

This is a simplified version of the github workflow I am using

name: Unit tests

on:
  push:
    branches: ['**']

jobs:
  build:

    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-20.04]
        python-version: [3.8]

    steps:
      - uses: actions/checkout@v4
      - name: Install ubuntu dependencies
        shell: bash -el {0}
        run: |
          sudo DEBIAN_FRONTEND=noninteractive apt-get update
          sudo DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential wget
          sudo DEBIAN_FRONTEND=noninteractive apt-get install libopenmpi-dev openmpi-bin

      - uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: test
          python-version: ${{ matrix.python-version }}
          fetch-depth: 2
      - name: Install dependencies
        shell: bash -el {0}
        run: |
          conda activate test
          python -m pip install --upgrade pip
          conda install --yes -c conda-forge openmpi mpi4py neuron
      - name: Run MPI application with NEURON
        shell: bash -el {0}
        run: |
          conda activate test
          mpiexec -np 2 -verbose nrniv -python -mpi -nobanner /usr/share/miniconda/envs/test/bin/python ./simulation/mpi_test.py

This is my python script mpi_test.py:

from neuron import h
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

logger.info(f"Process {rank} out of {size} processors")
# Initialize NEURON with MPI support
h('''
objref pc
pc = new ParallelContext()
''')

# Check if NEURON MPI is enabled
is_mpi_enabled = int(h.pc.nhost() > 1)

if is_mpi_enabled:
    logger.info(f"NEURON MPI is enabled. Running on {int(h.pc.nhost())} processes")
else:
    logger.info("NEURON MPI is not enabled.")

if rank == 0:
    print("NEURON MPI test completed")

This is the output I get:

Warning: no DISPLAY environment variable.
Warning: detected user attempt to enable MPI, but MPI support was disabled at build time.
--No graphics will be displayed.
Warning: no DISPLAY environment variable.
--No graphics will be displayed.
Warning: detected user attempt to enable MPI, but MPI support was disabled at build time.
INFO:root:Process 0 out of 2 processors
INFO:root:Process 1 out of 2 processors
INFO:root:NEURON MPI is not enabled.
INFO:root:NEURON MPI is not enabled.
NEURON MPI test completed

Expected result/behavior

The same commands in my local environment throw the following output:

numprocs=2
INFO:root:Process 0 out of 2 processors
NEURON MPI test completed
INFO:root:NEURON MPI is enabled. Running on 2 processes
INFO:root:Process 1 out of 2 processors
INFO:root:NEURON MPI is enabled. Running on 2 processes

Testing the same workflow on the windows and mac runners produce the same result, so I think is related to Linux

nrnhines commented 3 weeks ago

Others with more understanding of GitHub runners may be able to immediately suggest a solution. I guess I would begin by asking whether the mpi on the runner is compatible with the mpi specified by cmake to create the neuron wheel. A way to approach this might be

nrniv -python
from neuron import h
print(h.nrnversion(6))

and see if there in any mention of -DNRN_MPI_DYNAMIC

By the way, I see that there is a bug in nrniv -python argument processing in that /usr/share/miniconda/envs/test/bin/python is completely ignored (not a hoc or py file and no error message). Did you mean for it to be the specific version of python to be dynamically loaded by nrniv? If so it should follow the argument -pyexe

kmilo9999 commented 3 weeks ago

I removed the conda dependency that installs neuron and mpi4py, and moved to python/pip. Now I am getting the expected results to run parallel simulations.

numprocs=2
INFO:root:Process 1 out of 2 processors
INFO:root:NEURON MPI is enabled. Running on 2 processes
INFO:root:Process 0 out of 2 processors
INFO:root:NEURON MPI is enabled. Running on 2 processes

So. maybe someone should check what version of neuron is conda installing??

nrnhines commented 3 weeks ago

It appears from anaconda.org (and searching for neuron) that there a a half-dozen or so artifacts, mostly associated with individuals. The most downloaded is [conda-forge](https://anaconda.org/conda-forge) / [neuron 8.2.4] for linux-64 and osx-64. Unfortunately I don't know the provenance of that (how our CI uploaded it) or the build process (Cmake?) that created it.

neuronsimulator / nrn