Adjoint issues after updating firedrake / leaving pyroteus

[ddundo]

Hey @jwallwork23 and @stephankramer, could you please help me debug the errors I get when solving the adjoint on the new firedrake installation? Everything worked well on my previous installation (Using pyroteus and Joe's parmmg-metric-based firedrake branch, which I last updated some 6 months ago). Now I am on the firedrake master branch and using animate and goalie.

I get this error in my glacier experiments, but also when running the demos (e.g. burgers-ee.py and point_discharge2d-goal_oriented.py): ValueError: Looks like no solves were written to tape! Does the solution depend on the initial condition?. Here's the debug and traceback from burgers_ee.py:

WARNING GoalOrientedMeshSeq: Tape has no blocks!
WARNING GoalOrientedMeshSeq: Tape has no blocks!
WARNING GoalOrientedMeshSeq: No block has an adjoint solution. Has the adjoint equation been solved?
Traceback (most recent call last):
  File "/home/ubuntu/software/firedrake-sep23/src/goalie/demos/burgers_ee.py", line 139, in <module>
    solutions, indicators = mesh_seq.indicate_errors(
  File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "/home/ubuntu/software/firedrake-sep23/src/goalie/goalie/go_mesh_seq.py", line 141, in indicate_errors
    sols = self.solve_adjoint(**adj_kwargs)
  File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
  File "/home/ubuntu/software/firedrake-sep23/src/goalie/goalie/adjoint.py", line 323, in solve_adjoint
    raise ValueError(
ValueError: Looks like no solves were written to tape! Does the solution depend on the initial condition?

I made some importing changes inside goalie. Here they are:

1. goalie currently imports firedrake_adjoint which gave me a warning FutureWarning: The firedrake_adjoint module is deprecated. Instead, use the firedrake.adjoint module and explicitly start taping by calling continue_annotation(). So I changed all such imports to firedrake.adjoint, but I did not call this continue_annotation(). I wasn't sure where to call this. Before I did that, by still importing firedrake_adjoint, I would get the following error:

   
   Traceback (most recent call last):
   File "/home/ubuntu/software/firedrake-sep23/src/PyOP2/pyop2/caching.py", line 205, in __new__
   return cls._cache_lookup(key)
   File "/home/ubuntu/software/firedrake-sep23/src/PyOP2/pyop2/caching.py", line 213, in _cache_lookup
   return cls._cache[key]
   KeyError: (<class 'pyop2.global_kernel.GlobalKernel'>, ('0267ee8ad062db3cfcadfaa6e6454f87', (<Access.WRITE: 2>, <Access.READ: 1>, <Access.READ: 1>, <Access.READ: 1>, <Acc
   ess.READ: 1>, <Access.READ: 1>, <Access.READ: 1>, <Access.READ: 1>), (np:dtype('float64'), np:dtype('float64'), np:dtype('float64'), np:dtype('float64'), np:dtype('float6
   4'), np:dtype('float64'), np:dtype('float64'))), ('extruded', False), ('extruded_periodic', False), ('constant_layers', False), ('subset', False), 4, (<class 'pyop2.globa
   l_kernel.MatKernelArg'>, (((1, 1),),), ((<class 'pyop2.global_kernel.MapKernelArg'>, 3, None, None), (<class 'pyop2.global_kernel.MapKernelArg'>, 3, None, None)), False),
   (<class 'pyop2.global_kernel.DatKernelArg'>, (1,), (<class 'pyop2.global_kernel.MapKernelArg'>, 3, None, None), None), (<class 'pyop2.global_kernel.DatKernelArg'>, (1,),
   (<class 'pyop2.global_kernel.MapKernelArg'>, 3, None, None), None), (<class 'pyop2.global_kernel.DatKernelArg'>, (1,), (<class 'pyop2.global_kernel.MapKernelArg'>, 3, No
   ne, None), None), (<class 'pyop2.global_kernel.GlobalKernelArg'>, (1,)), (<class 'pyop2.global_kernel.GlobalKernelArg'>, (1,)), (<class 'pyop2.global_kernel.GlobalKernelA
   rg'>, (1,)), (<class 'pyop2.global_kernel.GlobalKernelArg'>, (1,)), 0, 0, 0, 0, 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data/meshadapt/Ice1-27sep2023/./timedep_goal.py", line 445, in msq.fixed_point_iteration(adaptor) File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "/home/ubuntu/software/firedrake-sep23/src/goalie/goalie/go_mesh_seq.py", line 283, in fixed_point_iteration sols, indicators = self.indicate_errors( File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "/home/ubuntu/software/firedrake-sep23/src/goalie/goalie/go_mesh_seq.py", line 141, in indicate_errors sols = self.solve_adjoint(adj_kwargs) File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "/home/ubuntu/software/firedrake-sep23/src/goalie/goalie/adjoint.py", line 315, in solve_adjoint tape.evaluate_adj(markings=True) File "/home/ubuntu/software/firedrake-sep23/src/pyadjoint/pyadjoint/tape.py", line 228, in evaluate_adj self._blocks[i].evaluate_adj(markings=markings) File "/home/ubuntu/software/firedrake-sep23/src/pyadjoint/pyadjoint/tape.py", line 105, in wrapper return function(*args, *kwargs) File "/home/ubuntu/software/firedrake-sep23/src/pyadjoint/pyadjoint/block.py", line 131, in evaluate_adj adj_output = self.evaluate_adj_component(inputs, File "/home/ubuntu/software/firedrake-sep23/src/firedrake/firedrake/adjoint_utils/blocks/function.py", line 335, in evaluate_adj_component return self.backend.Interpolator(dJdm, self.V).interpolate(adj_inputs[0], output=output, transpose=True) File "/home/ubuntu/software/firedrake-sep23/src/firedrake/firedrake/interpolation.py", line 621, in init self.callable, arguments = make_interpolator(expr, V, subset, access, bcs=bcs) File "petsc4py/PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "petsc4py/PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "/home/ubuntu/software/firedrake-sep23/src/firedrake/firedrake/interpolation.py", line 801, in make_interpolator loops.extend(_interpolator(V, tensor, expr, subset, arguments, access, bcs=bcs)) File "", line 2, in _interpolator File "/home/ubuntu/software/firedrake-sep23/src/firedrake/firedrake/utils.py", line 75, in wrapper return f(args, kwargs) File "/home/ubuntu/software/firedrake-sep23/src/firedrake/firedrake/interpolation.py", line 985, in _interpolator parloop = op2.ParLoop(parloop_args) File "/home/ubuntu/software/firedrake-sep23/src/PyOP2/pyop2/parloop.py", line 606, in ParLoop return LegacyParloop(args, *kwargs) File "/home/ubuntu/software/firedrake-sep23/src/PyOP2/pyop2/parloop.py", line 635, in LegacyParloop global_knl = GlobalKernel(local_knl, global_knl_args, File "/home/ubuntu/software/firedrake-sep23/src/PyOP2/pyop2/caching.py", line 207, in new obj = make_obj() File "/home/ubuntu/software/firedrake-sep23/src/PyOP2/pyop2/caching.py", line 197, in make_obj obj.init(args, **kwargs) File "/home/ubuntu/software/firedrake-sep23/src/PyOP2/pyop2/global_kernel.py", line 287, in init for larg, garg in zip(local_kernel.arguments, arguments) File "/home/ubuntu/software/firedrake-sep23/src/PyOP2/pyop2/local_kernel.py", line 144, in arguments assert len(self.accesses) == len(self.dtypes) AssertionError

2. Changed import in L6 in goalie.mesh_seq.py from `from firedrake.adjoint.solving import get_solve_blocks` to `from firedrake.adjoint_utils.solving import get_solve_blocks` (because `firedrake.adjoint.solving` doesn't exist on the master branch). This seems like an obvious issue based on the warning above `WARNING GoalOrientedMeshSeq: Tape has no blocks!`
3. Other small import changes which I don't think are important here. E.g. in goalie.options.py changed `from firedrake.meshadapt import RiemannianMetric` to `from animate.adapt import RiemannianMetric`.

[jwallwork23]

Thanks for flagging this, @ddundo. I'm working on it now and hope to update in the meeting.

[jwallwork23]

Other bugs were fixed but there is still a bug related to interpolation which gives the error above. I can reproduce this.

[jwallwork23]

I got it down to this minimal failing example:

from firedrake import *
from firedrake.adjoint import pyadjoint

pyadjoint.continue_annotation()

mesh = UnitSquareMesh(1, 1)
V = FunctionSpace(mesh, "CG", 1)
# R = FunctionSpace(mesh, "R", 0)
# Constant = lambda x: Function(R).assign(x)
u0 = Function(V).assign(1.0)
u = Function(V).assign(u0)
a = interpolate(Constant(1) + u, V)
F = (u - a) * TestFunction(V) * dx
solve(F == 0, u)
J = assemble(u * dx)

pyadjoint.pause_annotation()
pyadjoint.compute_gradient(J, pyadjoint.Control(u0))

It's fixed by uncommenting those two lines, i.e. using a Function from Real space instead of a Constant. I thought I'd suggested doing this before? Did you try it? When I put acc_rate in R-space your code doesn't raise the error above.

[jwallwork23]

When I put acc_rate in R-space your code doesn't raise the error above.

Sorry, I meant changing the Constant inside it to a R-space Function, i.e.

acc_rate = Function(FunctionSpace(mesh_seq[index], "R", 0)).assign(0.3)
a.interpolate(acc_rate - melt)

[ddundo]

Sorry, you are right - I completely forgot about this change with not using Constant! This indeed works :) Thanks Joe!