Open w3168 opened 3 weeks ago
stephankramer
commented
3 weeks ago So I suspect it's because the A11 block is empty: we don't have any pressure dependence in the continuity equation. So maybe just put a check for that?
if A11.getSize()[1]>0:
self.schur_plus.aypx(-1.0, A11, PETSc.Mat.Structure.SAME_NONZERO_PATTERN)(something like that, untested)
w3168
commented
3 weeks ago Thanks for looking at this @stephankramer I tried adding that in but I think it fails earlier actually in the init pressure solve... Also sorry I didn't make it clear in the first error message that if I run it in serial then there is no error.
It looks like we already have a similar bit of code in update()
if self.schur_plus is None:
self.schur_plus = self.schur.duplicate(copy=True)
self.schur_plus.aypx(-1.0, A11, PETSc.Mat.Structure.DIFFERENT_NONZERO_PATTERN)
else:
self.schur_plus = self.schur.copy(self.schur_plus, PETSc.Mat.Structure.DIFFERENT_NONZERO_PATTERN)
self.schur_plus.aypx(-1.0, A11, PETSc.Mat.Structure.SAME_NONZERO_PATTERN)```Here's the error message if I add the code you suggsted in at line 73 in assembled schur.
(firedrake) wscott@wscott-g41:~/thwaites/run_scripts_testing/ice_shelf_cavity$ mpiexec -np 2 python geostrophic_spinup_3d.py
Mesh dimension 3
/home/wscott/firedrake/src/firedrake/firedrake/function.py:325: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead
warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning)
/home/wscott/firedrake/src/firedrake/firedrake/function.py:325: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead
warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning)
/home/wscott/firedrake/src/firedrake/firedrake/functionspaceimpl.py:172: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead
warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning)
/home/wscott/firedrake/src/firedrake/firedrake/functionspaceimpl.py:172: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead
warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning)
/home/wscott/firedrake/src/firedrake/firedrake/function.py:325: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead
warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning)
firedrake:WARNING No comm specified for VectorSpaceBasis, COMM_WORLD assumed
0 SNES Function norm 7.676724482424e-11
Linear BalancePressureSolver_ solve converged due to CONVERGED_ATOL iterations 0
1 SNES Function norm 7.676724482424e-11
/home/wscott/firedrake/src/firedrake/firedrake/function.py:325: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead
warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning)
0 SNES Function norm 1.430679973179e-06
Linear predictor_momentum_initialise_pressure_ solve converged due to CONVERGED_RTOL iterations 3
1 SNES Function norm 7.327916201627e-14
0 SNES Function norm 1.430679973407e-06
Linear PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_fieldsplit_0_ solve converged due to CONVERGED_RTOL iterations 1
/home/wscott/firedrake/src/firedrake/firedrake/functionspaceimpl.py:172: FutureWarning: The .split() method is deprecated, please use the .subfunctions property instead
warnings.warn("The .split() method is deprecated, please use the .subfunctions property instead", category=FutureWarning)
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: Non conforming matrix add: global sizes X 4435 x 0, Y 4435 x 441
[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_fieldsplit_0_assembled_ksp_type value: preonly source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_fieldsplit_1_schur_ksp_converged_reason (no value) source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_ksp_rtol value: 1e-05 source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_ksp_type value: preonly source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_mat_type value: matfree source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_pc_factor_mat_mumps_icntl_14 value: 200 source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_pc_fieldsplit_schur_fact_type value: full source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_pc_fieldsplit_type value: schur source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_pc_type value: fieldsplit source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_snes_linesearch_type value: basic source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_snes_monitor (no value) source: code
[0]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_snes_type value: ksponly source: code
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.4.2-44465-g272f13d92c7 GIT Date: 2024-03-22 13:06:44 +0000
[0]PETSC ERROR: geostrophic_spinup_3d.py on a default named wscott-g41 by wscott Fri Jun 28 21:54:11 2024
[0]PETSC ERROR: Configure options PETSC_DIR=/home/wscott/firedrake/src/petsc PETSC_ARCH=default --download-netcdf --with-fortran-bindings=0 --with-zlib --download-mpich --download-cmake --download-suitesparse --download-hdf5 --download-pastix --with-debugging=0 --download-hypre --download-pnetcdf --download-hwloc --download-scalapack --download-chaco --download-superlu_dist --with-c2html=0 --download-mumps --download-metis --download-ptscotch --with-shared-libraries=1 --download-bison
[0]PETSC ERROR: #1 MatAXPY() at /home/wscott/firedrake/src/petsc/src/mat/utils/axpy.c:66
[0]PETSC ERROR: #2 MatAXPY_MPIAIJ() at /home/wscott/firedrake/src/petsc/src/mat/impls/aij/mpi/mpiaij.c:2091
[0]PETSC ERROR: #3 MatAXPY() at /home/wscott/firedrake/src/petsc/src/mat/utils/axpy.c:78
[0]PETSC ERROR: #4 MatAYPX() at /home/wscott/firedrake/src/petsc/src/mat/utils/axpy.c:374
[0]PETSC ERROR: #5 PCSetUp() at /home/wscott/firedrake/src/petsc/src/ksp/pc/interface/precon.c:1079
[0]PETSC ERROR: #6 KSPSetUp() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:415
[0]PETSC ERROR: #7 KSPSolve_Private() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:831
[0]PETSC ERROR: #8 KSPSolve() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:1078
[0]PETSC ERROR: #9 PCApply_FieldSplit_Schur() at /home/wscott/firedrake/src/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1196
[0]PETSC ERROR: #10 PCApply() at /home/wscott/firedrake/src/petsc/src/ksp/pc/interface/precon.c:497
[0]PETSC ERROR: #11 KSP_PCApply() at /home/wscott/firedrake/src/petsc/include/petsc/private/kspimpl.h:409
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Nonconforming object sizes
[1]PETSC ERROR: Non conforming matrix add: global sizes X 4826 x 0, Y 4826 x 882
[1]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_fieldsplit_0_assembled_ksp_type value: preonly source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_fieldsplit_1_schur_ksp_converged_reason (no value) source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_ksp_rtol value: 1e-05 source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_ksp_type value: preonly source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_mat_type value: matfree source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_pc_factor_mat_mumps_icntl_14 value: 200 source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_pc_fieldsplit_schur_fact_type value: full source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_pc_fieldsplit_type value: schur source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_pc_type value: fieldsplit source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_snes_linesearch_type value: basic source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_snes_monitor (no value) source: code
[1]PETSC ERROR: Option left: name:-PressureProjectionTimeIntegrator-MomentumEquation-ContinuityEquation_snes_type value: ksponly source: code
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.4.2-44465-g272f13d92c7 GIT Date: 2024-03-22 13:06:44 +0000
[1]PETSC ERROR: geostrophic_spinup_3d.py on a default named wscott-g41 by wscott Fri Jun 28 21:54:11 2024
[1]PETSC ERROR: Configure options PETSC_DIR=/home/wscott/firedrake/src/petsc PETSC_ARCH=default --download-netcdf --with-fortran-bindings=0 --with-zlib --download-mpich --download-cmake --download-suitesparse --download-hdf5 --download-pastix --with-debugging=0 --download-hypre --download-pnetcdf --download-hwloc --download-scalapack --download-chaco --download-superlu_dist --with-c2html=0 --download-mumps --download-metis --download-ptscotch --with-shared-libraries=1 --download-bison
[1]PETSC ERROR: #1 MatAXPY() at /home/wscott/firedrake/src/petsc/src/mat/utils/axpy.c:66
[1]PETSC ERROR: #2 MatAXPY_MPIAIJ() at /home/wscott/firedrake/src/petsc/src/mat/impls/aij/mpi/mpiaij.c:2091
[1]PETSC ERROR: #3 MatAXPY() at /home/wscott/firedrake/src/petsc/src/mat/utils/axpy.c:78
[1]PETSC ERROR: #4 MatAYPX() at /home/wscott/firedrake/src/petsc/src/mat/utils/axpy.c:374
[1]PETSC ERROR: #5 PCSetUp() at /home/wscott/firedrake/src/petsc/src/ksp/pc/interface/precon.c:1079
[1]PETSC ERROR: #6 KSPSetUp() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:415
[1]PETSC ERROR: #7 KSPSolve_Private() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:831
[1]PETSC ERROR: #8 KSPSolve() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:1078
[1]PETSC ERROR: #9 PCApply_FieldSplit_Schur() at /home/wscott/firedrake/src/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1196
[1]PETSC ERROR: #10 PCApply() at /home/wscott/firedrake/src/petsc/src/ksp/pc/interface/precon.c:497
[1]PETSC ERROR: #11 KSP_PCApply() at /home/wscott/firedrake/src/petsc/include/petsc/private/kspimpl.h:409
[1]PETSC ERROR: #12 KSPSolve_PREONLY() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/impls/preonly/preonly.c:23
[1]PETSC ERROR: #13 KSPSolve_Private() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:905
[1]PETSC ERROR: #14 KSPSolve() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:1078
[1]PETSC ERROR: #15 SNESSolve_KSPONLY() at /home/wscott/firedrake/src/petsc/src/snes/impls/ksponly/ksponly.c:49
[1]PETSC ERROR: #16 SNESSolve() at /home/wscott/firedrake/src/petsc/src/snes/interface/snes.c:4738
[0]PETSC ERROR: #12 KSPSolve_PREONLY() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/impls/preonly/preonly.c:23
[0]PETSC ERROR: #13 KSPSolve_Private() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:905
[0]PETSC ERROR: #14 KSPSolve() at /home/wscott/firedrake/src/petsc/src/ksp/ksp/interface/itfunc.c:1078
[0]PETSC ERROR: #15 SNESSolve_KSPONLY() at /home/wscott/firedrake/src/petsc/src/snes/impls/ksponly/ksponly.c:49
[0]PETSC ERROR: #16 SNESSolve() at /home/wscott/firedrake/src/petsc/src/snes/interface/snes.c:4738
velocity dofs: 72000
Pressure dofs: 9261
scalar dofs: 24000
hello pb
hello pb
option 1
hello A11.getsize
Traceback (most recent call last):
File "/home/wscott/thwaites/run_scripts_testing/ice_shelf_cavity/geostrophic_spinup_3d.py", line 203, in <module>
velocity dofs: 72000
Pressure dofs: 9261
scalar dofs: 24000
hello pb
hello pb
option 1
hello A11.getsize
Traceback (most recent call last):
File "/home/wscott/thwaites/run_scripts_testing/ice_shelf_cavity/geostrophic_spinup_3d.py", line 203, in <module>
vp_timestepper.initialize_pressure()
vp_timestepper.initialize_pressure()
File "/home/wscott/thwaites/thwaites/coupled_integrators.py", line 199, in initialize_pressure
File "/home/wscott/thwaites/thwaites/coupled_integrators.py", line 199, in initialize_pressure
self.solver.solve()
File "petsc4py/PETSc/Log.pyx", line 188, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
File "petsc4py/PETSc/Log.pyx", line 189, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
self.solver.solve()
File "petsc4py/PETSc/Log.pyx", line 188, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
File "petsc4py/PETSc/Log.pyx", line 189, in petsc4py.PETSc.Log.EventDecorator.decorator.wrapped_func
File "/home/wscott/firedrake/src/firedrake/firedrake/adjoint_utils/variational_solver.py", line 89, in wrapper
File "/home/wscott/firedrake/src/firedrake/firedrake/adjoint_utils/variational_solver.py", line 89, in wrapper
out = solve(self, **kwargs)
File "/home/wscott/firedrake/src/firedrake/firedrake/variational_solver.py", line 318, in solve
self.snes.solve(None, work)
File "petsc4py/PETSc/SNES.pyx", line 1555, in petsc4py.PETSc.SNES.solve
File "petsc4py/PETSc/libpetsc4py.pyx", line 1369, in petsc4py.PETSc.PCSetUp_Python
out = solve(self, **kwargs)
File "/home/wscott/firedrake/src/firedrake/firedrake/variational_solver.py", line 318, in solve
self.snes.solve(None, work)
File "petsc4py/PETSc/SNES.pyx", line 1555, in petsc4py.PETSc.SNES.solve
File "petsc4py/PETSc/libpetsc4py.pyx", line 1369, in petsc4py.PETSc.PCSetUp_Python
File "/home/wscott/firedrake/src/firedrake/firedrake/preconditioners/base.py", line 129, in setUp
File "/home/wscott/firedrake/src/firedrake/firedrake/preconditioners/base.py", line 129, in setUp
super().setUp(pc)
File "/home/wscott/firedrake/src/firedrake/firedrake/preconditioners/base.py", line 45, in setUp
self.initialize(pc)
File "/home/wscott/thwaites/thwaites/assembledschur.py", line 54, in initialize
self.update(pc)
File "/home/wscott/thwaites/thwaites/assembledschur.py", line 77, in update
self.schur_plus.aypx(-1.0, A11, PETSc.Mat.Structure.SAME_NONZERO_PATTERN)
File "petsc4py/PETSc/Mat.pyx", line 4094, in petsc4py.PETSc.Mat.aypx
petsc4py.PETSc.Error: error code 60
application called MPI_Abort(PYOP2_COMM_WORLD, 1) - process 0
super().setUp(pc)
File "/home/wscott/firedrake/src/firedrake/firedrake/preconditioners/base.py", line 45, in setUp
self.initialize(pc)
File "/home/wscott/thwaites/thwaites/assembledschur.py", line 54, in initialize
self.update(pc)
File "/home/wscott/thwaites/thwaites/assembledschur.py", line 77, in update
self.schur_plus.aypx(-1.0, A11, PETSc.Mat.Structure.SAME_NONZERO_PATTERN)
File "petsc4py/PETSc/Mat.pyx", line 4094, in petsc4py.PETSc.Mat.aypx
petsc4py.PETSc.Error: error code 60
application called MPI_Abort(PYOP2_COMM_WORLD, 1) - process 1
I think I hit this error on setonix on 2nd June 2024 when Angus updated Firedrake on setonix for me but now realised that it also seems to be affecting my installation on my anu comptuer. At a guess, It looks like something has gone wrong in assembled schur?
here's the full error message when runnning