RuntimeError: Unable to synchronously check link existence (addr overflow, addr = 2104, size = 328, eoa = 2048)

[FireworkZhao]

Dear fellow researchers:

I have faced with several difficulties while trying to follow the API: the Data recording and analysis from (https://steps.sourceforge.net/manual/API_2/STEPS_Tutorial_DataSaving.html), I used the script from the Git Repository (https://github.com/CNS-OIST/STEPS_Example/blob/master/user_manual/source/API_2/scripts/STEPS_Tutorial_DataSaving_box.py)

The result shows as follows: Model checking: No errors were found Traceback (most recent call last): File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/Diff_hdf5_Save.py", line 60, in sim.toDB(hdf, f'box_A{A}_B{B}_D{DCST}', A=A, B=B, DCST=DCST) File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/sim.py", line 1885, in toDB group, self._resultSelectors = dbh._newGroup(self, uid, self._resultSelectors, kwargs) File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 4080, in _newGroup group, selectors = super()._newGroup(sim, uid, selectors, kwargs) File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 3747, in _newGroup self._checkOpenFile(sim) File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 3570, in _checkOpenFile self._compObjHandler = _HDF5CompoundObjHandler( File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 2219, in init self._setUp() File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 2230, in _setUp if self._COMPOBJ_DSET_NAME not in self._group: File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/h5py/_hl/group.py", line 514, in contains return h5g._path_valid(self.id, self._e(name), self._lapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5g.pyx", line 531, in h5py.h5g._path_valid File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5l.pyx", line 212, in h5py.h5l.LinkProxy.exists RuntimeError: Unable to synchronously check link existence (addr overflow, addr = 2104, size = 328, eoa = 2048)

Although It says: no errors were found, I can only get the Brussselator_box.h5 file but not the box_A0.3_B3_D1e-13_Run0_rank0.xmf file. I assume the latter is needed for visualization in ParaView.

In summary, I wonder what was the possible reason to cause this situation? I would be much appreciate!

[jlallouette]

Dear @FireworkZhao,

This is most likely due to your HDF5 file being corrupted, presumably because the simulation was interrupted before it ended. You can delete the corrupted .h5 file and re-run the simulation. As mentioned in the user guide, this specific script takes some time to run. If you want to open the resulting files in Paraview, you can directly download the .h5 and .xmf files from https://github.com/CNS-OIST/STEPS_Example/raw/master/user_manual/source/data/DataSaving_HDF5_data.zip

[FireworkZhao]

Dear @FireworkZhao,

This is most likely due to your HDF5 file being corrupted, presumably because the simulation was interrupted before it ended. You can delete the corrupted .h5 file and re-run the simulation. As mentioned in the user guide, this specific script takes some time to run. If you want to open the resulting files in Paraview, you can directly download the .h5 and .xmf files from https://github.com/CNS-OIST/STEPS_Example/raw/master/user_manual/source/data/DataSaving_HDF5_data.zip

I deleted the '.h5' file and ran the script again， and found out this results:

Authorization required, but no authorization protocol specified Authorization required, but no authorization protocol specified Authorization required, but no authorization protocol specified [NF5468M6:2196678] Process received signal [NF5468M6:2196678] Signal: Segmentation fault (11) [NF5468M6:2196678] Signal code: Address not mapped (1) [NF5468M6:2196678] Failing at address: 0x440000e9 [NF5468M6:2196678] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fc48f242520] [NF5468M6:2196678] [ 1] /lib/x86_64-linux-gnu/libmpi.so.40(PMPI_Comm_set_errhandler+0x47)[0x7fc3ead786b7] [NF5468M6:2196678] [ 2] /home/zyh/STEPS/venv/lib/python3.10/site-packages/mpi4py/MPI.cpython-310-x86_64-linux-gnu.so(+0xa728f)[0x7fc48a03d28f] [NF5468M6:2196678] [ 3] /home/zyh/STEPS/venv/lib/python3.10/site-packages/mpi4py/MPI.cpython-310-x86_64-linux-gnu.so(+0x47e7c)[0x7fc489fdde7c] [NF5468M6:2196678] [ 4] /home/zyh/STEPS/venv/bin/python(PyModule_ExecDef+0x6f)[0x55965ea757ef] [NF5468M6:2196678] [ 5] /home/zyh/STEPS/venv/bin/python(+0x2370c0)[0x55965ea760c0] [NF5468M6:2196678] [ 6] /home/zyh/STEPS/venv/bin/python(+0x159b34)[0x55965e998b34] [NF5468M6:2196678] [ 7] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x2a27)[0x55965e9845d7] [NF5468M6:2196678] [ 8] /home/zyh/STEPS/venv/bin/python(_PyFunction_Vectorcall+0x7c)[0x55965e9999fc] [NF5468M6:2196678] [ 9] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x614a)[0x55965e987cfa] [NF5468M6:2196678] [10] /home/zyh/STEPS/venv/bin/python(_PyFunction_Vectorcall+0x7c)[0x55965e9999fc] [NF5468M6:2196678] [11] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x8ac)[0x55965e98245c] [NF5468M6:2196678] [12] /home/zyh/STEPS/venv/bin/python(_PyFunction_Vectorcall+0x7c)[0x55965e9999fc] [NF5468M6:2196678] [13] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x6bd)[0x55965e98226d] [NF5468M6:2196678] [14] /home/zyh/STEPS/venv/bin/python(_PyFunction_Vectorcall+0x7c)[0x55965e9999fc] [NF5468M6:2196678] [15] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x6bd)[0x55965e98226d] [NF5468M6:2196678] [16] /home/zyh/STEPS/venv/bin/python(_PyFunction_Vectorcall+0x7c)[0x55965e9999fc] [NF5468M6:2196678] [17] /home/zyh/STEPS/venv/bin/python(+0x159dc9)[0x55965e998dc9] [NF5468M6:2196678] [18] /home/zyh/STEPS/venv/bin/python(_PyObject_CallMethodIdObjArgs+0xff)[0x55965ea797cf] [NF5468M6:2196678] [19] /home/zyh/STEPS/venv/bin/python(PyImport_ImportModuleLevelObject+0x4b3)[0x55965e9ad7e3] [NF5468M6:2196678] [20] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x85a5)[0x55965e98a155] [NF5468M6:2196678] [21] /home/zyh/STEPS/venv/bin/python(_PyFunction_Vectorcall+0x7c)[0x55965e9999fc] [NF5468M6:2196678] [22] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x8ac)[0x55965e98245c] [NF5468M6:2196678] [23] /home/zyh/STEPS/venv/bin/python(_PyFunction_Vectorcall+0x7c)[0x55965e9999fc] [NF5468M6:2196678] [24] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x8ac)[0x55965e98245c] [NF5468M6:2196678] [25] /home/zyh/STEPS/venv/bin/python(_PyFunction_Vectorcall+0x7c)[0x55965e9999fc] [NF5468M6:2196678] [26] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x8ac)[0x55965e98245c] [NF5468M6:2196678] [27] /home/zyh/STEPS/venv/bin/python(+0x1687f1)[0x55965e9a77f1] [NF5468M6:2196678] [28] /home/zyh/STEPS/venv/bin/python(_PyEval_EvalFrameDefault+0x614a)[0x55965e987cfa] [NF5468M6:2196678] [29] /home/zyh/STEPS/venv/bin/python(+0x1687f1)[0x55965e9a77f1] [NF5468M6:2196678] End of error message

This has happened before I raised the first issue. I thought it might have sth. to do with whether I logged in Linux server as root or not. So I changed the user to root and ran the same script again without deleting the existing '.h5' file, hence causing the former issue. After your suggestion, I deleted the file while remaining logging in as root. But the same problem occurred again as the result shows above.

Also I have downloaded the zip file and tried to adjust it in Paraview. Still, I would like to make my own simulation with other mesh models, so I think I need to learn how to record the data. Is there any way to solve this so that I can visualize other diffusion results?

[jlallouette]

I would need more information: Can you run other STEPS scripts or do you get similar errors for all of them? If the error is specific to this one, was the script modified? If so, how? Which command are you using to run the script? If you are using something like python STEPS_Tutorial_DataSaving_box.py, can you try to run mpirun -n 1 python STEPS_Tutorial_DataSaving_box.py? How much memory do you have available on your Linux server?

[FireworkZhao]

Sorry for taking so long. 1) I have run other diffuison related scripts and certain surface diffusion and volume diffusion simulation can be carried out without error.

2) The script wasn't modified. So far, I have tried both of the command. The first one has the same result as I pasted above. The second command resulted similar error with slightly difference that it only had two rows of "Authorization required, but no authorization protocol specified" instead of three. Then I slightly modified the script in last few lines, but resulted the same.

    with XDMFHandler('Brusselator_box', hdf5DatasetKwArgs=options) as hdf:
        sim.toDB(hdf, f'box_A{A}_B{B}_D{DCST}', A=A, B=B, DCST=DCST)
        try:

            sim.newRun()

            sim.comp.X.Conc = 5e-6
            sim.comp.Y.Conc = 1.6e-4

            sim.run(ENDT)
        except Exception as e:
            print(e)
            raise e

3) the server has a memory of 516G.

[jlallouette]

Can you try to run the following script:

import mpi4py.MPI
a = mpi4py.MPI.COMM_WORLD.bcast(1, root=0)

Try to run it with both python script.py and mpirun -n 1 python script.py. If it also crashes, there might be a problem with your mpi4py or with your openMPI installation.

[FireworkZhao]

Can you try to run the following script:

import mpi4py.MPI
a = mpi4py.MPI.COMM_WORLD.bcast(1, root=0)

Try to run it with both python script.py and mpirun -n 1 python script.py. If it also crashes, there might be a problem with your mpi4py or with your openMPI installation.

I ran the scripy with both commands. The first on seems to be fine. It came back with no error. But the second turnd out to have the same problem:

Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified

Does it still means that I need to reinstall mpi4py?

[jlallouette]

But the second turnd out to have the same problem:

Did you only get the Authorization required ... message, or did it also crash with a Signal: Segmentation fault (11)?

The message is probably linked to problems with ssh and the X11 server (see https://github.com/open-mpi/ompi/issues/11553). Can you try to run export HWLOC_COMPONENTS=-gl before running the script?

[FireworkZhao]

1) I only get two lines of authorinzation required

2) Then I do the export HWLOC_COMPONENTS=-gl command and run the script in both ways, which both came out with no errors.

3) Then I do this export HWLOC_COMPONENTS=-gl command and try to run the STEPS_Tutorial_DataSaving_box.py script with both command. (I rename the sricpt as Diff_hdf5_Save but didn't alter the context) The result of python Diff_hdf5_Save.py is: The result of mpirun -n 1 python Diff_hdf5_Save.py is: Both of which seems to be incorrect but the errors turn out to be different.

[jlallouette]

Could you try adding the following code at the top of the STEPS_Tutorial_DataSaving_box.py script:

class Stub:
    def __getattr__(self, n):
        return Stub()
    def bcast(self, a, **_):
        return a
import sys
sys.modules['mpi4py'] = Stub()

If it prevents the segmentation fault, I would tend to think that something is wrong with your mpi4py or open MPI installation.

[FireworkZhao]

Could you try adding the following code at the top of the STEPS_Tutorial_DataSaving_box.py script:

class Stub:
    def __getattr__(self, n):
        return Stub()
    def bcast(self, a, **_):
        return a
import sys
sys.modules['mpi4py'] = Stub()

If it prevents the segmentation fault, I would tend to think that something is wrong with your mpi4py or open MPI installation.

Thank you for your help. I altered the script as instructed, and it did seem to be an MPI problem because I get the following result:

Traceback (most recent call last):
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/Diff_hdf5_Save.py", line 73, in <module>
    sim.newRun()
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/sim.py", line 1815, in newRun
    self._newRun()
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/sim.py", line 1977, in _newRun
    rs._newRun()
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 470, in _newRun
    self._dataHandler._newRun()
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 2368, in _newRun
    self._dbh._newRun(self._runId)
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 4472, in _newRun
    self._writeGrid(refCls, i, 0, 0)
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/saving.py", line 4577, in _writeGrid
    import mpi4py.MPI
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1439, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1405, in _get_spec
TypeError: 'Stub' object is not iterable
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

I will try to reinstall it first.

[jlallouette]

Can you try adding sys.modules['mpi4py.MPI'] = Stub() below sys.modules['mpi4py'] = Stub()?

[FireworkZhao]

Can you try adding sys.modules['mpi4py.MPI'] = Stub() below sys.modules['mpi4py'] = Stub()?

Thank you. I added the code and rerun the script. No error seems to occur. This time I have got both of the file of "xmf" and "h5" but when I try to open "xmf" in ParaView, I can't find the color map same in the instruction. I compared properties with my "xmf" file and the downloaded "xmf" file. They seem to have the same file size. Does this mean the simulation goes wrong or the problem still lies in open mpi?

[jlallouette]

Did you click on the "Apply" button (the green one on the left side of the screen) after opening the file? If you have both h5 and xmf, the simulation probably worked. But if you need MPI for future simulations with the TetOpSplit or DistTetOpSplit solvers, I guess you might run into the same issue unless you fix your mpi4py / openMPI installation.

[FireworkZhao]

Did you click on the "Apply" button (the green one on the left side of the screen) after opening the file? If you have both h5 and xmf, the simulation probably worked. But if you need MPI for future simulations with the TetOpSplit or DistTetOpSplit solvers, I guess you might run into the same issue unless you fix your mpi4py / openMPI installation.

Yes, I did. The cubit didn't appear on the Layout neither before nor after clicking. However, when I was testing the download file, the cubit would showed up with the right color map pannel after clicking apply. Will there be any preprocessing steps that I missed to cause this problem? I am also a little bit spectical about whether I've generated the right 'xmf' file.

[jlallouette]

There shouldn't be any preprocessing steps. Is the .h5 file located in the same folder as your .xmf file? Is it (roughly) the same size as the downloaded .h5 file?

[FireworkZhao]

There shouldn't be any preprocessing steps. Is the .h5 file located in the same folder as your .xmf file? Is it (roughly) the same size as the downloaded .h5 file?

The generated h5 file is almost twice bigger than the download file.

[jlallouette]

The .xmf file contains references to the .h5 file. Since your .h5 file seems to have been renamed with a (1) at the end, it's possible that Paraview cannot read the data. However, if the .xmf file actually contains references to Brusselator_box (1).h5, there must be some other issue.

[FireworkZhao]

The .xmf file contains references to the .h5 file. Since your .h5 file seems to have been renamed with a (1) at the end, it's possible that Paraview cannot read the data. However, if the .xmf file actually contains references to Brusselator_box (1).h5, there must be some other issue.

Sorry for the delay. I've make several attempts these days. The previous "not showing visualization result in ParaView" problem did related to how the file is named, thus by renaming .h5 , I can see the result of the brusselator reaction on the cubit now. The thing is, I found it difficult to reproduce the same diffusion effect on other model. I've got several questions: 1) I slightly alter the original script https://github.com/CNS-OIST/STEPS_Example/blob/master/user_manual/source/API_2/scripts/STEPS_Tutorial_DataSaving_box.py I changed the original mesh: mesh = TetMesh.LoadGmsh('./meshes/box_110k.msh', scale=1e-6) to: mesh = TetMesh.LoadOBJ('./meshes/101711-10_4p5-of-16_initial.CNG.swc_mesh.obj', scale=1e-6) But the result says AttributeError: type object 'TetMesh' has no attribute 'LoadOBJ'; Would you please tell me which formats does it support?

2) I changed the input mesh to: mesh = TetMesh.LoadGmsh('./meshes/convert_to_msh2.msh', scale=1e-6) The.mshfile was obtained by converting the previous .obj file to .msh using the following script:

import meshio 

in_file = "/path/to/101711-10_4p5-of-16_initial.CNG.swc_mesh.obj"
out_file = "/path/to/convert_to_msh2.msh"
mesh = meshio.read(in_file, file_format="obj")  
mesh.write(out_file,  mesh , file_format="gmsh") 

The result shows: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 88: invalid continuation byte Did the requisted input msh file has special format and how can I tell if the mesh model prepared fits the requirement?

3) I changed the input mesh to: mesh = TetMesh.LoadAbaqus('./meshes/convert_to_msh2.inp', scale=1e-6) The.inpfile was obtained by converting the previous .obj file to .inp using the following script:

import meshio 
in_file = "/path/to/101711-10_4p5-of-16_initial.CNG.swc_mesh.obj"
out_file = "/path/to/convert_to_msh2.inp"
mesh = meshio.read(in_file, file_format="obj")  
mesh.write(out_file,  mesh , file_format="abaqus") 

The result shows: [2024-06-05 15:33:28,944][ERROR][/home/zyh/STEPS/src/steps/geom/tetmesh.cpp:96][steps::tetmesh::Tetmesh::Tetmesh(const std::vector&, const std::vector&, const std::vector&)]: ArgErr: Empty tets list [Error Condition] !pTetsN Traceback (most recent call last): File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/Diff_hdf5_Save.py", line 50, in mesh = TetMesh.LoadAbaqus('./meshes/convert_to_msh2.inp', scale=1e-6) File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/geom.py", line 1539, in LoadAbaqus stepsMesh, ndprx, tetprx, triprx = smeshio.importAbaqus( File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_1/utilities/meshio.py", line 762, in importAbaqus mesh = stetmesh.Tetmesh(nodedata, tetdata, tridata) File "cysteps_geom.pyx", line 845, in cysteps_mpi._py_Tetmesh.init RuntimeError: ArgErr: Empty tets list [Error Condition] !pTetsN

To rule out the possibilities of the script didn't support inp format, I then tried to change the model into the cylinder same as https://github.com/CNS-OIST/STEPS_Example/blob/master/user_manual/source/API_2/scripts/STEPS_Tutorial_Diffusion.py used mesh = TetMesh.LoadAbaqus('./meshes/cylinder_86k.inp', scale=1e-6) which did generate the xmf file and h5 file that I wanted. Thus, the question remains: **1. Did I get the right inp input file after converting? Did it require special format that I am missing?

2. By getting the xmf and h5 file, does this mean the input can be in inp format?
3. When I further examine the generated xmf file, the cylinder can't quit exhibit the same result as the cubit do in ParaView? Does this mean the simulation is guaranteed to fail in other model, if so what's the adjustment that I need to do to prevent that?**

[WeiliangChenOIST]

.obj is surface mesh, you will need to generate a tetrahedral mesh from it before using the morphology in STEPS.

Something like https://gmsh.info/ can help you to do it. Here is a collection of meshing tools I usually use for STEPS projects. https://github.com/CNS-OIST/meshing

[jlallouette]

Regarding 1: You can see the available loading methods in the API reference: https://steps.sourceforge.net/manual/API_2/API_geom.html#steps.API_2.geom.TetMesh

Regarding 3: The cylinder_86k.inp has a higher volume, you can either rescale the mesh so that it has the same volume as box_110k.msh (i.e use scale=1e-6/(6.279 ** (1/3)) in TetMesh.LoadAbaqus), or let the simulation run for longer. As an indication, you can look at the well-mixed dynamics (in https://steps.sourceforge.net/manual/API_2/STEPS_Tutorial_DataSaving.html#Reading-saved-data). The period of the oscillation is around 5s. The wave starts propagating when the local concentration of X and Y is high enough in a tetrahedron, if you do not want to let the simulation run for 5s, you can change the initial concentration of Y to be higher (2e-4 should be enough without needing to rescale the mesh).

[FireworkZhao]

.obj is surface mesh, you will need to generate a tetrahedral mesh from it before using the morphology in STEPS.

Something like https://gmsh.info/ can help you to do it. Here is a collection of meshing tools I usually use for STEPS projects. https://github.com/CNS-OIST/meshing

Sorry to bother again. I have tried the https://gmsh.info/ toolkit to convert the mesh input and succeeded to get both the inp and msh output files from the same .obj model. But when I tried to used it in the script eg:

mesh = TetMesh.LoadGmsh('./meshes/convert_to_mshsix.msh', scale=1e-6)

or

mesh = TetMesh.LoadAbaqus('./meshes/convert_to_msh4.inp', scale=1e-6)

The first comes back with KeyError: 6 and the second comes back with the same result as last week:

[Error Condition] !pTetsN```
Does the input file require manually editting so that it can be used?

[WeiliangChenOIST]

can you upload the mesh please?

[FireworkZhao]

can you upload the mesh please?

convert_to_msh6.zip convert_to_msh4.zip there are the files I used, thank you for checking it up

[WeiliangChenOIST]

there is no tetrahedrons in your mesh file though. What is the original reconstruction format? From my experience it looks like a LM reconstruction in .swc or .nrn, if so you can try to use https://github.com/CNS-OIST/SWCTetMesher to generate the tetrahedral mesh.

[FireworkZhao]

there is no tetrahedrons in your mesh file though. What is the original reconstruction format? From my experience it looks like a LM reconstruction in .swc or .nrn, if so you can try to use https://github.com/CNS-OIST/SWCTetMesher to generate the tetrahedral mesh.

I got the ".obj" file originally. So I have tired to convert the original file to msh format using "meshio" which leads to the above problem with no tetrahedral. Then I tried a different method. I used the tool "tetgen" and convert it into tetgen format which contains four generated files of '.edge' '.ele' '.face' '.node'. Following the instructions of https://steps.sourceforge.net/manual/API_2/API_geom.html#steps.API_2.geom.TetMesh, I put three required files '.ele' '.face' '.noed' into a folder and tried to load this tetgen mesh with the code

mesh = TetMesh.LoadTetGen('./meshes/tetmesh1', scale=1e-6)

This leads to the following error:

./meshes/tetmesh1.node
Traceback (most recent call last):
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/Diff_hdf5_Save.py", line 56, in <module>
    mesh = TetMesh.LoadTetGen('./meshes/tetmesh1', scale=1e-6)
  File "/home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/API_2/geom.py", line 1631, in LoadTetGen
    stepsMesh, ndprx, tetprx, triprx = smeshio.importTetGen(pathroot, scale)
TypeError: cannot unpack non-iterable NoneType object

Does this means this work didn't support tetgen loading methods?

[WeiliangChenOIST]

can you upload the original .obj and the tetgen files please? If it is from neuromorpho etc please send me the original link.

[FireworkZhao]

can you upload the original .obj and the tetgen files please? If it is from neuromorpho etc please send me the original link.

Yes, the original file was initial_mesh.zip

[WeiliangChenOIST]

From the file name, I figured your morphology is this, so I fetched the standardized swc file and generated the mesh for you using the tool I mentioned. You can download the mesh from here.

But the mesh has 1.3e6 tetrahedrons, STEPS simulations with such large size mesh usually requires HPC cluster with hundreds to thousands cores, I am not sure if that's what you prepared for.

[FireworkZhao]

From the file name, I figured your morphology is this, so I fetched the standardized swc file and generated the mesh for you using the tool I mentioned. You can download the mesh from here.

But the mesh has 1.3e6 tetrahedrons, STEPS simulations with such large size mesh usually requires HPC cluster with hundreds to thousands cores, I am not sure if that's what you prepared for.

I see. what if I make a segmetation of the original neural mesh to reduce the number of terahedrons? what the maximum number of terahedrons it can handle? Also, I am sorry that I missed uploading the tetgen file I used and I will make a supplement below tetgenmeshexample.zip because I used the tool 'tetgen' to conver the original mesh. It came out with 5 outputs. But I only put '.ele' '.face' '.node' in the folder 'tetmesh1' when I run the script as the instruction said. Was the code mesh = TetMesh.LoadTetGen('./meshes/tetmesh1', scale=1e-6) wrongfully used to cause such error? I also used Gmsh to generate the coresponding volume to make sure the converted 'msh' file would contain hetrahedrons. I could provide it. Badea.zip It apperas to have a small number of hetrahedrons but the simulation still comes back with a KeyError:1. How can I avoid this similiar situation? Thank you so much for you time!

[jlallouette]

Regarding the tetgen files, we can load them without any issues, you need to also provide the name of the mesh when you load it: mesh = TetMesh.LoadTetGen('./meshes/tetmesh1/Badea2011Brn3bLED.CNG.1', scale=1e-6).

Regarding the gmsh file, it contains a $ParameterizedNodes section that STEPS does not support, you can replace it by a $Node section by opening the mesh with gmsh GUI, exporting to .msh version 2 ASCII but do not check the Save parametric coordinates checkbox.

[FireworkZhao]

Regarding the tetgen files, we can load them without any issues, you need to also provide the name of the mesh when you load it: mesh = TetMesh.LoadTetGen('./meshes/tetmesh1/Badea2011Brn3bLED.CNG.1', scale=1e-6).

Regarding the gmsh file, it contains a $ParameterizedNodes section that STEPS does not support, you can replace it by a $Node section by opening the mesh with gmsh GUI, exporting to .msh version 2 ASCII but do not check the Save parametric coordinates checkbox.

I changed the code and msh file and tested them seperately. I found out that both tetgen file and msh file came back with

Model checking:
No errors were found
[NF5468M6:170272] *** Process received signal ***
[NF5468M6:170272] Signal: Segmentation fault (11)
[NF5468M6:170272] Signal code: Address not mapped (1)
[NF5468M6:170272] Failing at address: (nil)
[NF5468M6:170272] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x763ef9642520]
[NF5468M6:170272] [ 1] /home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/cysteps_mpi.so(+0x832767)[0x763ef8274767]
[NF5468M6:170272] [ 2] /home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/cysteps_mpi.so(_ZN5steps8tetexact8Tetexact17_setCompSpecCountENS_4util9strong_idIjNS_6solver20comp_global_id_traitESt9enable_ifILb1EvEEENS3_IjNS4_20spec_global_id_traitES7_EEd+0x56)[0x763ef82749a6]
[NF5468M6:170272] [ 3] /home/zyh/STEPS/venv/lib/python3.10/site-packages/steps/cysteps_mpi.so(+0x6ecb01)[0x763ef812eb01]
[NF5468M6:170272] [ 4] python(+0x168c87)[0x5659fa2b7c87]
[NF5468M6:170272] [ 5] python(_PyEval_EvalFrameDefault+0x2a27)[0x5659fa2945d7]
[NF5468M6:170272] [ 6] python(_PyFunction_Vectorcall+0x7c)[0x5659fa2a99fc]
[NF5468M6:170272] [ 7] python(_PyEval_EvalFrameDefault+0x8ac)[0x5659fa29245c]
[NF5468M6:170272] [ 8] python(+0x1c2afe)[0x5659fa311afe]
[NF5468M6:170272] [ 9] python(+0xc75ef)[0x5659fa2165ef]
[NF5468M6:170272] [10] python(_PyObject_GenericSetAttrWithDict+0x73b)[0x5659fa27ceab]
[NF5468M6:170272] [11] python(+0x1c883c)[0x5659fa31783c]
[NF5468M6:170272] [12] python(_PyObject_MakeTpCall+0x25b)[0x5659fa29fa7b]
[NF5468M6:170272] [13] python(_PyEval_EvalFrameDefault+0x6a79)[0x5659fa298629]
[NF5468M6:170272] [14] python(+0x1c2afe)[0x5659fa311afe]
[NF5468M6:170272] [15] python(+0x1c60a8)[0x5659fa3150a8]
[NF5468M6:170272] [16] python(PyObject_SetAttr+0x10b)[0x5659fa27c6bb]
[NF5468M6:170272] [17] python(_PyEval_EvalFrameDefault+0x1041)[0x5659fa292bf1]
[NF5468M6:170272] [18] python(+0x13f9c6)[0x5659fa28e9c6]
[NF5468M6:170272] [19] python(PyEval_EvalCode+0x86)[0x5659fa384256]
[NF5468M6:170272] [20] python(+0x260108)[0x5659fa3af108]
[NF5468M6:170272] [21] python(+0x2599cb)[0x5659fa3a89cb]
[NF5468M6:170272] [22] python(+0x25fe55)[0x5659fa3aee55]
[NF5468M6:170272] [23] python(_PyRun_SimpleFileObject+0x1a8)[0x5659fa3ae338]
[NF5468M6:170272] [24] python(_PyRun_AnyFileObject+0x43)[0x5659fa3adf83]
[NF5468M6:170272] [25] python(Py_RunMain+0x2be)[0x5659fa3a0a5e]
[NF5468M6:170272] [26] python(Py_BytesMain+0x2d)[0x5659fa37702d]
[NF5468M6:170272] [27] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x763ef9629d90]
[NF5468M6:170272] [28] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x763ef9629e40]
[NF5468M6:170272] [29] python(_start+0x25)[0x5659fa376f25]
[NF5468M6:170272] *** End of error message ***
Segmentation fault (core dumped)

Does this means the tetgen and msh files are correct but something else has gone wrong?

[WeiliangChenOIST]

could you upload your simulation script and the mesh please?

[FireworkZhao]

yes. the script i used is Diff_hdf5_Save.zip and the tetgen file I put in the folder is tetmesh2.zip the msh file is Badea2.zip

[FireworkZhao]

Regarding 1: You can see the available loading methods in the API reference: https://steps.sourceforge.net/manual/API_2/API_geom.html#steps.API_2.geom.TetMesh

Regarding 3: The cylinder_86k.inp has a higher volume, you can either rescale the mesh so that it has the same volume as box_110k.msh (i.e use scale=1e-6/(6.279 ** (1/3)) in TetMesh.LoadAbaqus), or let the simulation run for longer. As an indication, you can look at the well-mixed dynamics (in https://steps.sourceforge.net/manual/API_2/STEPS_Tutorial_DataSaving.html#Reading-saved-data). The period of the oscillation is around 5s. The wave starts propagating when the local concentration of X and Y is high enough in a tetrahedron, if you do not want to let the simulation run for 5s, you can change the initial concentration of Y to be higher (2e-4 should be enough without needing to rescale the mesh).

I have tired several models to run the simulation these days. The segmentation fault seems to be concerned with https://github.com/CNS-OIST/STEPS/issues/42#issuecomment-2178027499. I have found other ways to alter the mesh models. But I am having trouble to understand the equation to rescale the input mesh. Would you show me more details please?

[iahepburn]

Regarding 1: You can see the available loading methods in the API reference: https://steps.sourceforge.net/manual/API_2/API_geom.html#steps.API_2.geom.TetMesh Regarding 3: The cylinder_86k.inp has a higher volume, you can either rescale the mesh so that it has the same volume as box_110k.msh (i.e use scale=1e-6/(6.279 ** (1/3)) in TetMesh.LoadAbaqus), or let the simulation run for longer. As an indication, you can look at the well-mixed dynamics (in https://steps.sourceforge.net/manual/API_2/STEPS_Tutorial_DataSaving.html#Reading-saved-data). The period of the oscillation is around 5s. The wave starts propagating when the local concentration of X and Y is high enough in a tetrahedron, if you do not want to let the simulation run for 5s, you can change the initial concentration of Y to be higher (2e-4 should be enough without needing to rescale the mesh).

I have tired several models to run the simulation these days. The segmentation fault seems to be concerned with #42 (comment). I have found other ways to alter the mesh models. But I am having trouble to understand the equation to rescale the input mesh. Would you show me more details please?

Hi @FireworkZhao . I could reproduce your issue with a scale of 1e-6, but the issue goes away with a scale of 1e-9. This implies that the volumes are very large (relatively speaking) for a stochastic simulator, so please check the geometry dimensions. Since STEPS works with discrete numbers meaning unsigned integers, we can get overflow issues with large geometries such as this.

[WeiliangChenOIST]

For Badea2.msh, setting the scale to 1e-9 results in the following molecule counts

X count  48.0
Y count  1931.0

which seems reasonable for the morphology with several dendritic branches. I suggest check the reconstruction scale (um or nm) of your original data, for um uses 1e-6 when importing to STEPS, and 1e-9 for nm.

[iahepburn]

Regarding 1: You can see the available loading methods in the API reference: https://steps.sourceforge.net/manual/API_2/API_geom.html#steps.API_2.geom.TetMesh Regarding 3: The cylinder_86k.inp has a higher volume, you can either rescale the mesh so that it has the same volume as box_110k.msh (i.e use scale=1e-6/(6.279 ** (1/3)) in TetMesh.LoadAbaqus), or let the simulation run for longer. As an indication, you can look at the well-mixed dynamics (in https://steps.sourceforge.net/manual/API_2/STEPS_Tutorial_DataSaving.html#Reading-saved-data). The period of the oscillation is around 5s. The wave starts propagating when the local concentration of X and Y is high enough in a tetrahedron, if you do not want to let the simulation run for 5s, you can change the initial concentration of Y to be higher (2e-4 should be enough without needing to rescale the mesh).

I have tired several models to run the simulation these days. The segmentation fault seems to be concerned with #42 (comment). I have found other ways to alter the mesh models. But I am having trouble to understand the equation to rescale the input mesh. Would you show me more details please?

Hi @FireworkZhao . I could reproduce your issue with a scale of 1e-6, but the issue goes away with a scale of 1e-9. This implies that the volumes are very large (relatively speaking) for a stochastic simulator, so please check the geometry dimensions. Since STEPS works with discrete numbers meaning unsigned integers, we can get overflow issues with large geometries such as this.

With a scale of 1e-6 the concentrations result in molecular numbers of approximately 1.9 trillion and 48 billion- STEPS simple can't cope with numbers as high as this, and for practical reasons you wouldn't want to run a stochastic simulation with these numbers anyway because the results would take a long, long time.

[FireworkZhao]

Regarding 1: You can see the available loading methods in the API reference: https://steps.sourceforge.net/manual/API_2/API_geom.html#steps.API_2.geom.TetMesh Regarding 3: The cylinder_86k.inp has a higher volume, you can either rescale the mesh so that it has the same volume as box_110k.msh (i.e use scale=1e-6/(6.279 ** (1/3)) in TetMesh.LoadAbaqus), or let the simulation run for longer. As an indication, you can look at the well-mixed dynamics (in https://steps.sourceforge.net/manual/API_2/STEPS_Tutorial_DataSaving.html#Reading-saved-data). The period of the oscillation is around 5s. The wave starts propagating when the local concentration of X and Y is high enough in a tetrahedron, if you do not want to let the simulation run for 5s, you can change the initial concentration of Y to be higher (2e-4 should be enough without needing to rescale the mesh).

I have tired several models to run the simulation these days. The segmentation fault seems to be concerned with #42 (comment). I have found other ways to alter the mesh models. But I am having trouble to understand the equation to rescale the input mesh. Would you show me more details please?

Hi @FireworkZhao . I could reproduce your issue with a scale of 1e-6, but the issue goes away with a scale of 1e-9. This implies that the volumes are very large (relatively speaking) for a stochastic simulator, so please check the geometry dimensions. Since STEPS works with discrete numbers meaning unsigned integers, we can get overflow issues with large geometries such as this.

With a scale of 1e-6 the concentrations result in molecular numbers of approximately 1.9 trillion and 48 billion- STEPS simple can't cope with numbers as high as this, and for practical reasons you wouldn't want to run a stochastic simulation with these numbers anyway because the results would take a long, long time.

Is it possible to calculate the molecular numbers according to the mesh scale? I can't quite understand the link between them. Also, is the equation scale=1e-6/(6.279 ** (1/3)) using any specific method so that the rescaling is more accurate?

[jlallouette]

If you want to rescale your mesh to have the same volume as another mesh (for example the one used in the user guide, box_110k.msh'), you need to use scale=(desiredVolume / currentVolume) ** (1/3) with desiredVolume being the (scaled) volume that you want for your simulation, and currentVolume the (unscaled) volume of your mesh. The ** (1/3) is for transforming a volume ratio to a length ratio. The 6.279 value from the above scale comes from the volume of the mesh you were trying to run the simulation on (cylinder_86k.msh). To get the volume of a loaded mesh: https://steps.sourceforge.net/manual/API_2/API_geom.html#steps.API_2.geom.TetMesh.Vol

[iahepburn]

and to calculate the number of molecules from a molar concentration, simply multiply the concentration by the compartment volume (in litres) and Avogadro's number.