Investigate RXD failure on Ubuntu latest

[JCGoran]

For the past couple of days, the CI has been consistently failing on one RXD test, and only on Ubuntu latest (see for instance this CI run: https://github.com/neuronsimulator/nrn-build-ci/actions/runs/8933258624/job/24538369746), with the following trace:

 93: =================================== FAILURES ===================================
93: ___________________________ test_multi_gridding_mix ____________________________
93: 
93: neuron_instance = (<TopLevelHocInterpreter>, <module 'neuron.rxd' from '/__w/nrn-build-ci/nrn-build-ci/nrn/build/lib/python/neuron/rxd/__init__.py'>, {'data': [0.0, -65.0, -65.0, -65.0, -65.0, -65.0, ...], 'record_count': 4, 'rlen': 4075}, None)
93: 
93:     def test_multi_gridding_mix(neuron_instance):
93:         h, rxd, data, save_path = neuron_instance
93:         axon = h.Section(name="axon")
93:         axon.L = 2 * µm
93:         axon.diam = 2 * µm
93:         axon.nseg = 5
93:         axon.pt3dadd(5, 5, 0, 2)
93:         axon.pt3dadd(5, 5, 2, 2)
93:     
93:         axon2 = h.Section(name="axon2")
93:         axon2.L = 2 * µm
93:         axon2.diam = 2 * µm
93:         axon2.nseg = 5
93:         axon2.pt3dadd(0, 0, 10, 2)
93:         axon2.pt3dadd(0, 0, 12, 2)
93:     
93:         def my_initial(node):
93:             if node.x < 0.5:
93:                 return 1 * mM
93:             else:
93:                 return 0 * mM
93:     
93:         import random
93:     
93:         random.seed(1)
93:     
93:         def my_initial1(node):
93:             return random.random()
93:     
93:         cyt = rxd.Region([axon], nrn_region="i", name="cyt", dx=0.125)
93:         ca = rxd.Species(cyt, name="ca", charge=2, initial=my_initial1, d=1)
93:     
93:         cyt2 = rxd.Region([axon2], nrn_region="i", name="cyt2", dx=0.25)
93:         ca2 = rxd.Species(cyt2, name="ca2", charge=2, initial=my_initial, d=1)
93:         rate1 = rxd.Rate(ca, -ca * (1 - ca) * (0.2 - ca))
93:         rate2 = rxd.Rate(ca2, -ca2 * (1 - ca2) * (0.3 - ca2))
93:     
93:         rxd.set_solve_type(dimension=3)
93:     
93:         h.finitialize(-65 * mV)
93:         h.continuerun(0.1 * ms)
93:     
93:         if not save_path:
93:             max_err = compare_data(data)
93: >           assert max_err < tol
93: E           assert 0.9894138864909854 < 1e-10
93: 
93: test/rxd/3d/test_multi_gridding_mix.py:52: AssertionError

I've isolated the test in this PR (debugging via tmate is enabled): https://github.com/neuronsimulator/nrn-build-ci/pull/65

I did not manage to reproduce the failure locally due to certain container engine issues, but it should in principle be reproducible by using something like act (on x86_64 at least).

[JCGoran]

Further observations:

• the issue is reproducible in a Docker container with the ubuntu:latest image on both x86_64 and aarch64. Reproduction steps:
  • run docker run -ti --arch $(uname -m) --rm -v $(pwd):/neuron ubuntu:latest /bin/bash from the root dir of this repo to run the container
  • run bash scripts/install_debian.sh to install reqs
  • run python3 -m venv env && source env/bin/activate to setup venv
  • run pip install neuron-nightly pytest to get the Python packages
  • clone NEURON and run python -m pytest -k test_multi_gridding_mix. The test will fail with the exact error as above.
• the issue does not appear to be related to RNG since the produced numbers between Ubuntu 20.04 and 22.04 are identical
• the issue is not related to CVode since the test is not using it