search

OP2 / PyOP2

Framework for performance-portable parallel computations on unstructured meshes
https://op2.github.io/PyOP2
Other
80 stars 35 forks source link

Rework initialisation #459

Open wence- opened 9 years ago

wence- commented 9 years ago

Recent issues on HPC systems to do with forking show how fragile our "automatic" initialisation is. There's a subtle interplay between MPI init and enabling the fork server for compilation, along with when the forked child is allowed to "import" anything.

The issues are:

  1. After MPI_Init we should not fork (either for compilation, or getting a version or anything else)

To address this we use Andreas Kloeckner's prefork.py to fork (before MPI_Init) a "fork server".

  1. During python "import", a fork()ed child may not also call "import", Python complains about import locks.

Plausibly the "prefork-everywhere" branch of PyOP2 addresses these problems. However, it's pretty fragile.

Problems are:

from mpi4py import MPI
# or
from petsc4py import PETSc

initialize MPI automagically. Therefore, we need to make sure we've set up the fork server before either of these are imported.

Given that we don't have a top-level "init" method the user has to call after importing firedrake, we're in a bit of a bind. Suggestions?

miklos1 commented 9 years ago

Can't we just force from firedrake import * to call the prefork? Or is it the problem that that line itself is an import, so if we fork there then the fork server cannot import any more?

wence- commented 9 years ago

Exactly. Until that line completes, you're "importing"

wence- commented 9 years ago

Note that one way out of this is to move all the MPI/petsc initialisation into op2.init, which is currently called automagically for the user when they create a firedrake object.

We should then expose it in the API and document when one needs to do things by hand.