search

firedrakeproject / firedrake

Firedrake is an automated system for the portable solution of partial differential equations using the finite element method (FEM)
https://firedrakeproject.org
Other
498 stars 157 forks source link

Patch PC + fieldsplit produces compliation errors #1356

Closed colinjcotter closed 5 years ago

colinjcotter commented 5 years ago

The following tries to use patch PC with a fieldsplit:

from firedrake import *

ai = Constant(1)

R = 6371220.
H = Constant(5960.)

mesh = IcosahedralSphereMesh(radius=R, refinement_level=3, degree=3)
x = SpatialCoordinate(mesh)
mesh.init_cell_orientations(x)

x, y, z = SpatialCoordinate(mesh)

outward_normals = CellNormal(mesh)
perp = lambda u: cross(outward_normals, u)

V1 = FunctionSpace(mesh, "BDM", 2)
V2 = FunctionSpace(mesh, "DG", 1)
W = MixedFunctionSpace((V1, V2))

u, h = TrialFunctions(W)
v, phi = TestFunctions(W)

Omega = Constant(7.292e-5)  # rotation rate
R = Constant(R)
f = 2*Omega*z/R  # Coriolis parameter
g = Constant(9.8)  # Gravitational constant
tau = Constant(60*60)

a = (
    inner(ai*u,v) - tau*inner(f*perp(u),v) + 
    tau*g*inner(h,div(v))
    +inner(ai*h,phi) - tau*H*inner(div(u),phi)
)*dx

aP = (inner(ai*u,v) + tau**2*g*H*inner(div(u)/ai,div(v))
      - tau*inner(f*perp(u),v)
      + inner(ai*h,phi))*dx

import math
f1 = exp((x+y+z)/R)*x*y*z/R**3
F = inner(f1,phi)*dx

params = {
    "ksp_type": "gmres",
    "ksp_rtol": 1e-8,
    "pc_type": "fieldsplit",
    "pc_fieldsplit_type": "schur",
    "pc_fieldsplit_0_fields":"1",
    "pc_fieldsplit_1_fields":"0",
    "pc_fieldsplit_schur_fact_type": "full",
    "pc_fieldsplit_off_diag_use_amat": True,
    "fieldsplit_0_ksp_type": "preonly",
    "fieldsplit_0_pc_type": "bjacobi",
    "fieldsplit_0_pc_sub_type": "ilu",
    "fieldsplit_1_pc_type": "python",
    "fieldsplit_1_pc_python_type": "firedrake.PatchPC",
    "fieldsplit_1_ksp_type": "preonly",
    "fieldsplit_1_patch_pc_patch_save_operators": True,
    "fieldsplit_1_patch_pc_patch_partition_of_unity": False,
    "fieldsplit_1_patch_pc_patch_construct_type": "star",
    "fieldsplit_1_patch_pc_patch_construct_dim": 0,
    "fieldsplit_1_patch_pc_patch_sub_mat_type": "seqdense",
    "fieldsplit_1_patch_sub_ksp_type": "preonly",
    "fieldsplit_1_patch_sub_pc_type": "lu",
    "ksp_monitor": True
}

w = Function(W)

Prob = LinearVariationalProblem(a, F, w, aP=aP)
Solver = LinearVariationalSolver(Prob, solver_parameters=params)

Solver.solve()
colinjcotter commented 5 years ago

Error is:

Traceback (most recent call last):
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 247, in get_so
    return ctypes.CDLL(soname)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f.so, 6): image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 278, in get_so
    stdout=log)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mpicc', '-std=c99', '-fPIC', '-Wall', '-framework', 'Accelerate', '-march=native', '-O3', '-ffast-math', '-I/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/include', '-I/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2', '-msse', '-o', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.so.tmp', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.c', '-dynamiclib', '-L/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-Wl,-rpath,/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-lpetsc', '-lm']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "libpetsc4py/libpetsc4py.pyx", line 1253, in libpetsc4py.PCSetUp_Python
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/base.py", line 45, in setUp
    self.initialize(pc)
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/patch.py", line 277, in initialize
    funptr, kinfo = matrix_funptr(J)
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/patch.py", line 132, in matrix_funptr
    mod = JITModule(kinfo.kernel, iterset, *args)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/caching.py", line 195, in __new__
    return make_obj()
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/caching.py", line 189, in make_obj
    obj.__init__(*args, **kwargs)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/sequential.py", line 756, in __init__
    self.compile()
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/sequential.py", line 842, in compile
    comm=self.comm)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 442, in load
    dll = compiler.get_so(src, extension)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 284, in get_so
    Compile errors in %s""" % (e.cmd, e.returncode, logfile, errfile))
pyop2.exceptions.CompilationError: Command "['mpicc', '-std=c99', '-fPIC', '-Wall', '-framework', 'Accelerate', '-march=native', '-O3', '-ffast-math', '-I/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/include', '-I/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2', '-msse', '-o', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.so.tmp', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.c', '-dynamiclib', '-L/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-Wl,-rpath,/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-lpetsc', '-lm']" return error status 1.
Unable to compile code
Compile log in /Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.log
Compile errors in /Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.err

The above exception was the direct cause of the following exception:

SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
Exception ignored in: 'petsc4py.PETSc.traceback'
SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
Traceback (most recent call last):
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 247, in get_so
    return ctypes.CDLL(soname)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f.so, 6): image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 278, in get_so
    stdout=log)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mpicc', '-std=c99', '-fPIC', '-Wall', '-framework', 'Accelerate', '-march=native', '-O3', '-ffast-math', '-I/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/include', '-I/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2', '-msse', '-o', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.so.tmp', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.c', '-dynamiclib', '-L/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-Wl,-rpath,/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-lpetsc', '-lm']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "libpetsc4py/libpetsc4py.pyx", line 1253, in libpetsc4py.PCSetUp_Python
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/base.py", line 45, in setUp
    self.initialize(pc)
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/patch.py", line 277, in initialize
    funptr, kinfo = matrix_funptr(J)
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/patch.py", line 132, in matrix_funptr
    mod = JITModule(kinfo.kernel, iterset, *args)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/caching.py", line 195, in __new__
    return make_obj()
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/caching.py", line 189, in make_obj
    obj.__init__(*args, **kwargs)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/sequential.py", line 756, in __init__
    self.compile()
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/sequential.py", line 842, in compile
    comm=self.comm)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 442, in load
    dll = compiler.get_so(src, extension)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 284, in get_so
    Compile errors in %s""" % (e.cmd, e.returncode, logfile, errfile))
pyop2.exceptions.CompilationError: Command "['mpicc', '-std=c99', '-fPIC', '-Wall', '-framework', 'Accelerate', '-march=native', '-O3', '-ffast-math', '-I/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/include', '-I/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2', '-msse', '-o', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.so.tmp', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.c', '-dynamiclib', '-L/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-Wl,-rpath,/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-lpetsc', '-lm']" return error status 1.
Unable to compile code
Compile log in /Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.log
Compile errors in /Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.err

The above exception was the direct cause of the following exception:

SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
Exception ignored in: 'petsc4py.PETSc.traceback'
SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
Traceback (most recent call last):
  File "libpetsc4py/libpetsc4py.pyx", line 1344, in libpetsc4py.PCApply_Python
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/patch.py", line 350, in apply
    self.patch.apply(x, y)
AttributeError: 'PatchPC' object has no attribute 'patch'

The above exception was the direct cause of the following exception:

SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
Exception ignored in: 'petsc4py.PETSc.traceback'
SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
    Residual norms for firedrake_0_ solve.
    0 KSP Residual norm 1.333507165981e+01 
Traceback (most recent call last):
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 247, in get_so
    return ctypes.CDLL(soname)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f.so, 6): image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 278, in get_so
    stdout=log)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mpicc', '-std=c99', '-fPIC', '-Wall', '-framework', 'Accelerate', '-march=native', '-O3', '-ffast-math', '-I/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/include', '-I/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2', '-msse', '-o', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.so.tmp', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.c', '-dynamiclib', '-L/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-Wl,-rpath,/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-lpetsc', '-lm']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "libpetsc4py/libpetsc4py.pyx", line 1253, in libpetsc4py.PCSetUp_Python
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/base.py", line 45, in setUp
    self.initialize(pc)
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/patch.py", line 277, in initialize
    funptr, kinfo = matrix_funptr(J)
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/patch.py", line 132, in matrix_funptr
    mod = JITModule(kinfo.kernel, iterset, *args)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/caching.py", line 195, in __new__
    return make_obj()
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/caching.py", line 189, in make_obj
    obj.__init__(*args, **kwargs)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/sequential.py", line 756, in __init__
    self.compile()
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/sequential.py", line 842, in compile
    comm=self.comm)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 442, in load
    dll = compiler.get_so(src, extension)
  File "/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2/compilation.py", line 284, in get_so
    Compile errors in %s""" % (e.cmd, e.returncode, logfile, errfile))
pyop2.exceptions.CompilationError: Command "['mpicc', '-std=c99', '-fPIC', '-Wall', '-framework', 'Accelerate', '-march=native', '-O3', '-ffast-math', '-I/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/include', '-I/Users/colincotter/firedrake-complex/firedrake/src/PyOP2/pyop2', '-msse', '-o', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.so.tmp', '/Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.c', '-dynamiclib', '-L/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-Wl,-rpath,/Users/colincotter/firedrake-complex/firedrake/lib/python3.6/site-packages/petsc/lib', '-lpetsc', '-lm']" return error status 1.
Unable to compile code
Compile log in /Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.log
Compile errors in /Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.err

The above exception was the direct cause of the following exception:

SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
Exception ignored in: 'petsc4py.PETSc.traceback'
SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
Traceback (most recent call last):
  File "libpetsc4py/libpetsc4py.pyx", line 1344, in libpetsc4py.PCApply_Python
  File "/Users/colincotter/firedrake-complex/firedrake/src/firedrake/firedrake/preconditioners/patch.py", line 350, in apply
    self.patch.apply(x, y)
AttributeError: 'PatchPC' object has no attribute 'patch'

The above exception was the direct cause of the following exception:

SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
Exception ignored in: 'petsc4py.PETSc.traceback'
SystemError: <built-in method insert of list object at 0x116127588> returned a result with an error set
    1 KSP Residual norm 2.359487599170e-14
pefarrell commented 5 years ago

What are the contents of

Compile log in /Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.log
Compile errors in /Users/colincotter/firedrake-complex/firedrake/.cache/pyop2/256f446535eab788b7de04381a03533f_p39831.err

?

wence- commented 5 years ago

The problem is that we don't handle cell orientations in the hacked up patch "assembler" right now. Perhaps like this?

diff --git a/firedrake/preconditioners/patch.py b/firedrake/preconditioners/patch.py
index b34ae434..affffe15 100644
--- a/firedrake/preconditioners/patch.py
+++ b/firedrake/preconditioners/patch.py
@@ -118,6 +118,11 @@ def matrix_funptr(form):
     arg = mesh.coordinates.dat(op2.READ, mesh.coordinates.cell_node_map()[op2.i[0]])
     arg.position = 1
     args.append(arg)
+    if kinfo.oriented:
+        c = mesh.cell_orientations()
+        arg = c.dat(op2.READ, c.cell_node_map())
+        arg.position = len(args)
+        args.append(arg)
     for n in kinfo.coefficient_map:
         c = form.coefficients()[n]
         for (i, c_) in enumerate(c.split()):
@@ -295,6 +300,8 @@ class PatchPC(PCBase):
             global_bc_nodes = numpy.empty(0, dtype=PETSc.IntType)

         op_coeffs = [mesh.coordinates]
+        if kinfo.oriented:
+            op_coeffs.append(mesh.cell_orientations())
         for n in kinfo.coefficient_map:
             op_coeffs.append(J.coefficients()[n])

Really we (I?) need to refactor the assemble logic so we can do this without duplicating code.

wence- commented 5 years ago

Cell orientations are now supported (that was #1432).