search

mikaem / Oasis

55 stars 48 forks source link

Example problems hang inside initialize loop when run with MPI #25

Closed benzwick closed 5 years ago

benzwick commented 5 years ago

The DrivenCavity example runs fine in serial:

$ python NSfracStep.py problem=DrivenCavity

but when run with MPI:

$ mpirun -n 4 python NSfracStep.py problem=DrivenCavity

it hangs at the following line: https://github.com/mikaem/Oasis/blob/3de552ca7271b0f91bed51322bab4fce06e63a99/oasis/problems/NSfracStep/DrivenCavity.py#L48

The following example problems have similar issues:

For example, TaylorGreen3D hangs here: https://github.com/mikaem/Oasis/blob/3de552ca7271b0f91bed51322bab4fce06e63a99/oasis/problems/NSfracStep/TaylorGreen3D.py#L87

mikaem commented 5 years ago

I cannot reproduce this. Works fine for me on OSX, with fenics installed from conda-forge.

benzwick commented 5 years ago

EDIT: I introduced the following errors while trying to debug the original issue with MPI. The examples run with MPI in both the singularity container and with fenics installed from conda-forge.

It must be an issue with my FEniCS installation. However, I tried running the above in a Singularity container created using the Docker images but now I get errors like this:

Cylinder:

$ mpirun -n 4 python3 NSfracStep.py problem=Cylinder
Traceback (most recent call last):
  File "NSfracStep.py", line 156, in <module>
    vars().update(setup(**vars()))
  File "/home/ben/projects/oasis/Oasis-BZ/oasis/solvers/NSfracStep/IPCS_ABCN.py", line 48, in setup
    method=velocity_update_solver)
  File "/home/ben/projects/oasis/Oasis-BZ/oasis/common/utilities.py", line 235, in __init__
    preconditioner_type=preconditioner_type)
  File "/home/ben/projects/oasis/Oasis-BZ/oasis/common/utilities.py", line 97, in __init__
    self.rhs = Vector(self.vector())
  File "/usr/local/lib/python3.6/dist-packages/dolfin/function/function.py", line 386, in vector
    return self._cpp_object.vector()
AttributeError: 'DivFunction' object has no attribute '_cpp_object'

TaylorGreen3D:

$ mpirun -n 4 python3 NSfracStep.py problem=TaylorGreen3D
Traceback (most recent call last):
  File "NSfracStep.py", line 138, in <module>
    initialize(**vars())
  File "/home/ben/projects/oasis/Oasis-BZ/oasis/problems/NSfracStep/TaylorGreen3D.py", line 86, in initialize
    element=VV[ui].ufl_element()), VV[ui])
  File "/home/ben/projects/oasis/Oasis-BZ/oasis/common/utilities.py", line 97, in __init__
    self.rhs = Vector(self.vector())
  File "/usr/local/lib/python3.6/dist-packages/dolfin/function/function.py", line 386, in vector
    return self._cpp_object.vector()
AttributeError: 'OasisFunction' object has no attribute '_cpp_object'

DrivenCavity:

$ mpirun -n 4 python3 NSfracStep.py problem=DrivenCavity
Traceback (most recent call last):
  File "NSfracStep.py", line 156, in <module>
    vars().update(setup(**vars()))
  File "/home/ben/projects/oasis/Oasis-BZ/oasis/solvers/NSfracStep/IPCS_ABCN.py", line 48, in setup
    method=velocity_update_solver)
  File "/home/ben/projects/oasis/Oasis-BZ/oasis/common/utilities.py", line 235, in __init__
    preconditioner_type=preconditioner_type)
  File "/home/ben/projects/oasis/Oasis-BZ/oasis/common/utilities.py", line 97, in __init__
    self.rhs = Vector(self.vector())
  File "/usr/local/lib/python3.6/dist-packages/dolfin/function/function.py", line 386, in vector
    return self._cpp_object.vector()
AttributeError: 'DivFunction' object has no attribute '_cpp_object'
benzwick commented 5 years ago

Upgrading Python to 3.6.7 (from 3.5.3, the Debian Stretch 9.6 default) and reinstalling FEniCS 2018.1.0 from source with pybind11 version 2.2.3 also fixes this issue.