Submesh creation fails on 6 processes with ghostmode shared facet

[jorgensd]

Summarize the issue

Minimal example below. Runs with less than 6 processes, and if ghostmode is none

How to reproduce the bug

Runing test below.

Minimal Example (Python)

import dolfinx
import numpy as np
from mpi4py import MPI

mesh = dolfinx.mesh.create_unit_square(
    MPI.COMM_WORLD, 4, 2, ghost_mode=dolfinx.mesh.GhostMode.shared_facet)

# Create connectivities required for defining integration entities
tdim = mesh.topology.dim

# Get number of cells on process
cell_map = mesh.topology.index_map(tdim)
num_cells = cell_map.size_local + cell_map.num_ghosts

# Create markers for each size of the interface
cell_values = np.ones(num_cells, dtype=np.int32)
cell_values[dolfinx.mesh.locate_entities(
    mesh, tdim, lambda x: x[0] <= 0.5 + 1e-13)] = 2
ct = dolfinx.mesh.meshtags(mesh, tdim, np.arange(
    num_cells, dtype=np.int32), cell_values)

submesh, sub_cell_to_parent, sub_vertex_to_parent, _ = dolfinx.mesh.create_submesh(mesh, ct.dim, ct.find(2))

Output (Python)

No response

Version

main branch

DOLFINx git commit

No response

Installation

No response

Additional information

No response

[jpdean]

On my old branches, this runs fine. I'll have a look into it

[jpdean]

This problem was caused when NBX was removed from create_sub_index_map. The submap destination ranks are now being computed incorrectly in certain corner cases, which causes MPI to hang. I'll make a PR with a fix soon.

[garth-wells]

Fixed in #3099