BUG: Mesh parallel error

zhyxue0 commented 9 months ago

Hello everyone.

I try to use project on two different meshes. The code worked fine when running serially, but when I ran it in parallel, it prompted: AssertionError: Whoever made mesh_B should explicitly mark mesh_A as having a compatible parallel layout.

Here is the code:

from firedrake import *

Lx = 2 
Ly = 2

Nx = 4
Ny = Nx

mesh_1 = PeriodicRectangleMesh(Nx, Ny, Lx, Ly)
V_1 = VectorFunctionSpace(mesh_1, "DG", 1)
u_1 = Function(V_1)

mesh_2 = PeriodicRectangleMesh(2*Nx, 2*Ny, Lx, Ly)
V_2 = VectorFunctionSpace(mesh_2, "DG", 1)

u_2 = project(u_1, V_2)

Here is the report:

Traceback (most recent call last):
  File "/home/text_1.py", line 18, in <module>
    u_2 = project(u_1, V_2)
  File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
Traceback (most recent call last):
  File "/home/text_1.py", line 18, in <module>
    u_2 = project(u_1, V_2)
  File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "/home/firedrake/firedrake/src/firedrake/firedrake/adjoint/projection.py", line 34, in wrapper
  File "/home/firedrake/firedrake/src/firedrake/firedrake/adjoint/projection.py", line 34, in wrapper
    output = project(*args, **kwargs)
  File "/home/firedrake/firedrake/src/firedrake/firedrake/projection.py", line 78, in project
    output = project(*args, **kwargs)
  File "/home/firedrake/firedrake/src/firedrake/firedrake/projection.py", line 78, in project
    use_slate_for_inverse=use_slate_for_inverse).project()
  File "/home/firedrake/firedrake/src/firedrake/firedrake/projection.py", line 181, in project
    use_slate_for_inverse=use_slate_for_inverse).project()
  File "/home/firedrake/firedrake/src/firedrake/firedrake/projection.py", line 181, in project
    self.apply_massinv(self.target, self.rhs)
  File "/home/firedrake/firedrake/src/firedrake/firedrake/projection.py", line 240, in rhs
    self.apply_massinv(self.target, self.rhs)
  File "/home/firedrake/firedrake/src/firedrake/firedrake/projection.py", line 240, in rhs
    self.mixed_mass.mult(u, v)
  File "/home/firedrake/firedrake/src/PyOP2/pyop2/utils.py", line 62, in __get__
    self.mixed_mass.mult(u, v)
  File "/home/firedrake/firedrake/src/PyOP2/pyop2/utils.py", line 62, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/home/firedrake/firedrake/src/firedrake/firedrake/projection.py", line 234, in mixed_mass
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/home/firedrake/firedrake/src/firedrake/firedrake/projection.py", line 234, in mixed_mass
    return assemble_mixed_mass_matrix(self.source.function_space(),
  File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
    return assemble_mixed_mass_matrix(self.source.function_space(),
  File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "/home/firedrake/firedrake/src/firedrake/firedrake/supermeshing.py", line 97, in assemble_mixed_mass_matrix
  File "/home/firedrake/firedrake/src/firedrake/firedrake/supermeshing.py", line 97, in assemble_mixed_mass_matrix
    intersections = intersection_finder(mesh_A, mesh_B)
  File "firedrake/cython/supermeshimpl.pyx", line 132, in firedrake.cython.supermeshimpl.intersection_finder
    intersections = intersection_finder(mesh_A, mesh_B)
  File "firedrake/cython/supermeshimpl.pyx", line 132, in firedrake.cython.supermeshimpl.intersection_finder
AssertionError: Whoever made mesh_B should explicitly mark mesh_A as having a compatible parallel layout.
application called MPI_Abort(PYOP2_COMM_WORLD, 1) - process 0
AssertionError: Whoever made mesh_B should explicitly mark mesh_A as having a compatible parallel layout.
application called MPI_Abort(PYOP2_COMM_WORLD, 1) - process 1

connorjward commented 9 months ago

This is a known limitation of projection between mismatching meshes (supermeshing). It will only work if the meshes have a matching parallel decomposition which we cannot guarantee here I believe.

We do support interpolation across meshes into particular function spaces. This is however not quite the same as projection so may not be suitable for you.

colinjcotter commented 9 months ago

Those two meshes are nested so project and interpolate have the same result.

From: Connor Ward @.***> Sent: 27 November 2023 09:31 To: firedrakeproject/firedrake Cc: Subscribed Subject: Re: [firedrakeproject/firedrake] BUG: Mesh parallel error (Issue #3265)

This is a known limitation of projection between mismatching meshes (supermeshing). It will only work if the meshes have a matching parallel decomposition which we cannot guarantee here I believe.

We do support interpolation across mesheshttps://www.firedrakeproject.org/interpolation.html#interpolation-across-meshes into particular function spaces. This is however not quite the same as projection so may not be suitable for you.

— Reply to this email directly, view it on GitHubhttps://github.com/firedrakeproject/firedrake/issues/3265#issuecomment-1827462564, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOSV4TRJN2NOEGBK6STDVLYGRMY3AVCNFSM6AAAAAA72TB3FCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRXGQ3DENJWGQ. You are receiving this because you are subscribed to this thread.Message ID: @.***>

firedrakeproject / firedrake

BUG: Mesh parallel error #3265