search

thetisproject / thetis

Finite element flow solver for simulating coastal and estuarine flows.
Other
70 stars 28 forks source link

Error with new checkpointing #336

Closed jhill1 closed 1 year ago

jhill1 commented 1 year ago

If I run a script for a short time; with hdf5 output. Then continue from one of the checkpoints, I receive:

(firedrake) jh1889@envpc319 ~/work/projects/Tsunamis/thetis_runs/rectangle_test $ python run_wd_cont.py Using constant wetting and drying parameter (value 1.00) /home/jh1889/firedrake/src/firedrake/firedrake/function.py:317: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning) /home/jh1889/firedrake/src/firedrake/firedrake/functionspaceimpl.py:125: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning) Element family: dg-dg, degree: 1 2D cell type: triangle 2D mesh: 6704 vertices, 13106 elements Horizontal element size: 337.93 ... 829.43 m Number of 2D elevation DOFs: 39318 Number of 2D velocity DOFs: 78636 Number of cores: 1 Elevation DOFs per core: ~39318.0 dt = 10 Using time integrator: CrankNicolson Traceback (most recent call last): File "/home/jh1889/work/projects/Tsunamis/thetis_runs/rectangle_test/run_wd_cont.py", line 78, in solverObj.load_state(checkpoint, outputdir=output_dir) File "PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "/home/jh1889/firedrake/src/thetis/thetis/solver2d.py", line 840, in load_state e.exporters['uv_2d'].load(i_stored, self.fields.uv_2d) File "PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "/home/jh1889/firedrake/src/thetis/thetis/exporter.py", line 224, in load function.assign(g) File "PETSc/Log.pyx", line 115, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "PETSc/Log.pyx", line 116, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func File "/home/jh1889/firedrake/src/firedrake/firedrake/adjoint/function.py", line 135, in wrapper ret = assign(self, other, *args, **kwargs) File "/home/jh1889/firedrake/src/firedrake/firedrake/function.py", line 404, in assign Assigner(self, expr, subset).assign() File "/home/jh1889/firedrake/src/firedrake/firedrake/assign.py", line 141, in init raise ValueError("All functions in the expression must use the same " ValueError: All functions in the expression must use the same mesh as the assignee

The mesh coming from the checkpoint is not identical according to UFL:

From assign.py in firedrake:

                if coeff.ufl_element() != assignee.ufl_element():
                    raise ValueError("All functions in the expression must have the same "
                                     "element as the assignee")
                if coeff.ufl_domain() != assignee.ufl_domain():
                    raise ValueError("All functions in the expression must use the same "
                                     "mesh as the assignee")

Version is as of 1st Feb: jh1889@envpc319 ~/firedrake/src/firedrake $ git log commit c0b45ce2123fdeadf358df1d5655ce42f3b3d74b (HEAD -> master, origin/master, origin/HEAD) Author: Uygar 83709209+uygarkov@users.noreply.github.com Date: Wed Feb 1 17:16:40 2023 +0100

The code to reproduce can be found here.

stephankramer commented 1 year ago

Yeah, not entirely sure what the right fix is: we could change https://github.com/thetisproject/thetis/blob/3437bbd2853f69568a8a996f9235fa4795b1bb2b/thetis/exporter.py#L223 to:

function = f.load_function(function.function_space().mesh(), function.name())

i.e. not read the mesh at all from the checkpoint and directly load into function. But this only works if the user provided mesh is read from a checkpoint (as in your example) and I'm also not entirely sure this is safe if that checkpoint wasn't created in the same run.

Or we could do a nasty copy instead of the assign

function.dat.data[:] = g.dat.data[:]

but again I don't really know how bad that is....

I've opened https://github.com/firedrakeproject/firedrake/discussions/3053 regarding this