search

gridapapps / GridapMHD.jl

A Gridap-based FEM solver for the MHD equations
MIT License
17 stars 2 forks source link

Obscure error in hunt driver with solid coupling #44

Open guillermogf opened 2 months ago

guillermogf commented 2 months ago

Hi,

I've detected an error when running the hunt driver with solid coupling (tw != 0).

The error cause has proved difficult to isolate as small variations in the model seem to suppress it or trigger it. Most notably, running the same problem setup with a different number of parallel processes can trigger it. The error has been observed at different mesh sizes, Hartmann numbers and tw values. It has been reproduced not only with the mpi backend and petsc solver but also with the julia solver and sequential backend (as reported by @pmartorell).

Here is a running script with which I can reproduce the issue:

using SparseMatricesCSR
using GridapPETSc

solver = Dict(
    :solver => :petsc,
    :matrix_type    => SparseMatrixCSR{0,PetscScalar,PetscInt},
    :vector_type    => Vector{PetscScalar},
    :petsc_options  => "-snes_monitor -ksp_error_if_not_converged true \
                         -ksp_converged_reason -ksp_type preonly -pc_type lu \
                         -pc_factor_mat_solver_type mumps -mat_mumps_icntl_28 1 \
                         -mat_mumps_icntl_29 2 -mat_mumps_icntl_4 3 \
                         -mat_mumps_cntl_1 0.001",
    :niter          => 100,
    :rtol           => 1e-5,
)

hunt(;
  title="mwe",
  backend=:mpi,
  np=(4,4),
  nc=(20,20),
  B=(0.0,50,0.0),
  L=1.0,
  σw1=0.0001,
  σw2=1000.0,
  tw=0.5,
  nsums=100,
  vtk=true,
  debug=false,
  BL_adapted=true,
  solver=solver,
  path="../data/"
  )

Notably either changing np=(4,4) to np=(2,2), or tw=0.5 to tw=0.1 suppresses the error. Toggling the BL_adapted flag also seems to affect whether the error is triggered.

Here follows the abriged output produced. Please find here the full output file. It's too long and a bit messy as it collects the error from several of the 16 mpi processes.

Abriged output ``` [...] [7.467e+00 s in MAIN] pre_process [3.753e+01 s in MAIN] fe_spaces [2.904e+00 s in MAIN] solver_setup [...] ┌ Error: │ exception = │ AssertionError: A check failed │ Stacktrace: │ [1] macro expansion │ @ ~/.julia/packages/Gridap/Ort08/src/Helpers/Macros.jl:61 [inlined] │ [2] evaluate!(cache::Gridap.Arrays.CachedArray{Float64, 3, Array{Float64, 3}}, f::Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, a::Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, b::Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}) │ @ Gridap.Fields ~/.julia/packages/Gridap/Ort08/src/Fields/FieldArrays.jl:539 │ [3] evaluate!(cache::Tuple{Gridap.Fields.MatrixBlock{Array{Float64, 3}}, Matrix{Gridap.Arrays.CachedArray{Float64, 3, Array{Float64, 3}}}}, k::Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, f::Gridap.Fields.VectorBlock{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}}, g::Gridap.Fields.MatrixBlock{Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}}) │ @ Gridap.Fields ~/.julia/packages/Gridap/Ort08/src/Fields/ArrayBlocks.jl:904 │ [4] evaluate │ @ ~/.julia/packages/Gridap/Ort08/src/Arrays/Maps.jl:87 [inlined] │ [5] return_value │ @ ~/.julia/packages/Gridap/Ort08/src/Fields/ArrayBlocks.jl:779 [inlined] │ [6] return_type │ @ ~/.julia/packages/Gridap/Ort08/src/Arrays/Maps.jl:62 [inlined] │ [7] lazy_map(::Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, ::FillArrays.Fill{Gridap.Fields.VectorBlock{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}}, 1, Tuple{Base.OneTo{Int64}}}, ::Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(+)}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.MatrixBlock{Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(GridapMHD.conv)}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.MatrixBlock{Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.PosNegReindex{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, Vector{Float64}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Float64}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Arrays.Table{Int32, Vector{Int32}, Vector{Int32}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.TransposeMap, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.MatrixBlock{Gridap.Fields.TransposeFieldIndices{Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BlockMap{1}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.VectorBlock{Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, 1, Tuple{Base.OneTo{Int64}}}, Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{typeof(Gridap.Fields.pinvJt)}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCoordinates{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCellNodes{3}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Base.OneTo{Int64}}}}}}}}}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(GridapMHD.conv)}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Fields.MatrixBlock{Array{Gridap.TensorValues.VectorValue{3, Float64}, 3}}, 1, Tuple{FillArrays.Fill{Gridap.Fields.MatrixBlock{Gridap.Fields.TransposeFieldIndices{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, Gridap.TensorValues.VectorValue{3, Float64}}}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.BroadcastingFieldOpMap{typeof(LinearAlgebra.dot)}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{typeof(Gridap.Fields.pinvJt)}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCoordinates{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCellNodes{3}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}}}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.PosNegReindex{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, Vector{Float64}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Float64}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Arrays.Table{Int32, Vector{Int32}, Vector{Int32}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}}}, FillArrays.Fill{Matrix{Gridap.TensorValues.TensorValue{3, 3, Float64, 9}}, 1, Tuple{Base.OneTo{Int64}}}}}}}}}}}) │ @ Gridap.Arrays ~/.julia/packages/Gridap/Ort08/src/Arrays/LazyArrays.jl:57 │ [8] evaluate!(cache::Nothing, f::Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, x::Gridap.CellData.CellPoint{Gridap.CellData.ReferenceDomain, FillArrays.Fill{Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCoordinates{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCellNodes{3}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}, FillArrays.Fill{Matrix{Float64}, 1, Tuple{Base.OneTo{Int64}}}}}, Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}) │ @ Gridap.CellData ~/.julia/packages/Gridap/Ort08/src/CellData/CellFields.jl:539 │ [9] evaluate │ @ ~/.julia/packages/Gridap/Ort08/src/Arrays/Maps.jl:87 [inlined] │ [10] (::Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain})(x::Gridap.CellData.CellPoint{Gridap.CellData.ReferenceDomain, FillArrays.Fill{Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Base.OneTo{Int64}}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Fields.LinearCombinationMap{Colon}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Broadcasting{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCoordinates{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Gridap.TensorValues.VectorValue{3, Float64}}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Geometry.CartesianCellNodes{3}}, 1, Tuple{Base.OneTo{Int64}}}, Vector{Int32}, 1, Tuple{Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}}}, FillArrays.Fill{Matrix{Float64}, 1, Tuple{Base.OneTo{Int64}}}}}, Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}}) │ @ Gridap.CellData ~/.julia/packages/Gridap/Ort08/src/CellData/CellFields.jl:388 │ [11] integrate(f::Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, quad::Gridap.CellData.CellQuadrature{Gridap.CellData.ReferenceDomain, Gridap.CellData.PhysicalDomain}) │ @ Gridap.CellData ~/.julia/packages/Gridap/Ort08/src/CellData/CellQuadratures.jl:154 │ [12] integrate(f::Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, b::Gridap.CellData.GenericMeasure) │ @ Gridap.CellData ~/.julia/packages/Gridap/Ort08/src/CellData/DomainContributions.jl:149 │ [13] #296 │ @ ~/.julia/packages/GridapDistributed/VR0ki/src/CellData.jl:234 [inlined] │ [14] map(::GridapDistributed.var"#296#297", ::PartitionedArrays.MPIArray{Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, 1}, ::PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}) │ @ PartitionedArrays ~/.julia/packages/PartitionedArrays/py6uo/src/mpi_array.jl:229 │ [15] integrate(f::GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.CellData.OperationCellField{Gridap.CellData.ReferenceDomain}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int32}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, Nothing}, b::GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}) │ @ GridapDistributed ~/.julia/packages/GridapDistributed/VR0ki/src/CellData.jl:233 │ [16] *(a::Gridap.CellData.Integrand, b::GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}) │ @ GridapDistributed ~/.julia/packages/GridapDistributed/VR0ki/src/CellData.jl:254 │ [17] dc_mhd_u_u(u::GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.FESpaces.SingleFieldFEFunction{Gridap.CellData.GenericCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int32}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, PartitionedArrays.MPIArray{SubArray{Float64, 1, Vector{Float64}, Tuple{UnitRange{Int64}}, true}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, du::GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TrialBasis, Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int32}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, Nothing}, v_u::GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TestBasis, Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int32}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, Nothing}, α::Float64, dΩ::GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}) │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/weakforms.jl:291 │ [18] dc_mhd(x::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, dx::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TrialBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}, dy::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TestBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}, α::Float64, dΩ::GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}) │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/weakforms.jl:289 │ [19] (::GridapMHD.var"#dc#38"{Float64, GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}})(x::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, dx::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TrialBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}, dy::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TestBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}) │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/weakforms.jl:61 │ [20] (::GridapMHD.var"#jac#40"{GridapMHD.var"#dc#38"{Float64, GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}}, GridapMHD.var"#a#35"{Vector{Any}, Vector{Any}, Vector{Any}, GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.CellData.GenericCellField{Gridap.CellData.ReferenceDomain}, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Gridap.Arrays.IdentityVector{Int64}}, 1, Tuple{Base.OneTo{Int64}}}, Int64, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}, Nothing}, GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}, Dict{Symbol, Any}, Float64, Gridap.TensorValues.VectorValue{3, Float64}, Float64, Float64, Float64, GridapDistributed.DistributedMeasure{PartitionedArrays.MPIArray{Gridap.CellData.GenericMeasure, 1}, GridapDistributed.DistributedTriangulation{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.BodyFittedTriangulation{3, 3, Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, Gridap.Geometry.GridView{3, 3, Gridap.Geometry.GridPortion{3, 3, Gridap.Geometry.CartesianGrid{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}, Vector{Int64}}, Gridap.Arrays.LazyArray{FillArrays.Fill{Gridap.Arrays.Reindex{Vector{Int32}}, 1, Tuple{Base.OneTo{Int64}}}, Int32, 1, Tuple{Vector{Int64}}}}, 1}, GridapDistributed.GenericDistributedDiscreteModel{3, 3, PartitionedArrays.MPIArray{Gridap.Geometry.CartesianDiscreteModel{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}, 1}, Vector{PartitionedArrays.PRange}, GridapDistributed.DistributedCartesianDescriptor{PartitionedArrays.MPIArray{Int64, 1}, Tuple{Int64, Int64, Int64}, Gridap.Geometry.CartesianDescriptor{3, Float64, GridapMHD.Meshers.var"#map1#19"{Float64, Float64, Float64}}}}}}}, Dict{Symbol, Any}})(x::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, dx::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TrialBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}, dy::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEBasisComponent{Gridap.FESpaces.SingleFieldFEBasis{Gridap.FESpaces.TestBasis, Gridap.CellData.ReferenceDomain}}, 1}, B, Nothing} where B}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}, 1}, Nothing}) │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/weakforms.jl:75 │ [21] residual_and_jacobian(op::Gridap.FESpaces.FEOperatorFromWeakForm, uh::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}) │ @ Gridap.FESpaces ~/.julia/packages/Gridap/Ort08/src/FESpaces/FEOperatorsFromWeakForm.jl:100 │ [22] residual_and_jacobian(op::Gridap.FESpaces.AlgebraicOpFromFEOp, x::PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}) │ @ Gridap.FESpaces ~/.julia/packages/Gridap/Ort08/src/FESpaces/FEOperators.jl:194 │ [23] _setup_cache(x::PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}, nls::PETScNonlinearSolver{typeof(GridapPETSc.snes_from_options)}, op::Gridap.FESpaces.AlgebraicOpFromFEOp) │ @ GridapPETSc ~/.julia/packages/GridapPETSc/MNuuG/src/PETScNonlinearSolvers.jl:129 │ [24] solve!(x::PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}, nls::PETScNonlinearSolver{typeof(GridapPETSc.snes_from_options)}, op::Gridap.FESpaces.AlgebraicOpFromFEOp, #unused#::Nothing) │ @ GridapPETSc ~/.julia/packages/GridapPETSc/MNuuG/src/PETScNonlinearSolvers.jl:168 │ [25] solve!(x::PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}, nls::PETScNonlinearSolver{typeof(GridapPETSc.snes_from_options)}, op::Gridap.FESpaces.AlgebraicOpFromFEOp) │ @ Gridap.Algebra ~/.julia/packages/Gridap/Ort08/src/Algebra/NonlinearSolvers.jl:22 │ [26] solve!(u::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, solver::Gridap.FESpaces.NonlinearFESolver, feop::Gridap.FESpaces.FEOperatorFromWeakForm, cache::Nothing) │ @ Gridap.FESpaces ~/.julia/packages/Gridap/Ort08/src/FESpaces/FESolvers.jl:163 │ [27] solve! │ @ ~/.julia/packages/Gridap/Ort08/src/FESpaces/FESolvers.jl:14 [inlined] │ [28] solve! │ @ ~/.julia/packages/Gridap/Ort08/src/FESpaces/FESolvers.jl:18 [inlined] │ [29] _solve │ @ ~/software/GridapMHD/GridapMHD.jl/src/Main.jl:329 [inlined] │ [30] _solve(xh::GridapDistributed.DistributedMultiFieldCellField{Vector{GridapDistributed.DistributedCellField{A, B, GridapDistributed.DistributedFEFunctionData{T}} where {A, B, T}}, PartitionedArrays.MPIArray{Gridap.MultiField.MultiFieldFEFunction{Gridap.MultiField.MultiFieldCellField{Gridap.CellData.ReferenceDomain}}, 1}, GridapDistributed.DistributedFEFunctionData{PartitionedArrays.PVector{Vector{Float64}, PartitionedArrays.MPIArray{Vector{Float64}, 1}, PartitionedArrays.MPIArray{PartitionedArrays.LocalIndices, 1}, PartitionedArrays.MPIArray{PartitionedArrays.VectorAssemblyCache{Float64}, 1}, Float64}}}, solver::PETScNonlinearSolver{typeof(GridapPETSc.snes_from_options)}, op::Gridap.FESpaces.FEOperatorFromWeakForm, params::Dict{Symbol, Any}) │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Main.jl:327 │ [31] main(_params::Dict{Symbol, Any}; output::Dict{Symbol, Any}) │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Main.jl:145 │ [32] (::GridapMHD.var"#65#66"{Dict{Symbol, Any}})() │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:161 │ [33] with(f::GridapMHD.var"#65#66"{Dict{Symbol, Any}}; kwargs::Base.Pairs{Symbol, Vector{SubString{String}}, Tuple{Symbol}, NamedTuple{(:args,), Tuple{Vector{SubString{String}}}}}) │ @ GridapPETSc ~/.julia/packages/GridapPETSc/MNuuG/src/Environment.jl:38 │ [34] _hunt(; distribute::PartitionedArrays.var"#88#89"{MPI.Comm, Bool}, rank_partition::Tuple{Int64, Int64}, nc::Tuple{Int64, Int64}, ν::Float64, ρ::Float64, σ::Float64, B::Tuple{Float64, Int64, Float64}, f::Tuple{Float64, Float64, Float64}, ζ::Float64, L::Float64, u0::Float64, B0::Float64, σw1::Float64, σw2::Float64, tw::Float64, nsums::Int64, vtk::Bool, title::String, path::String, debug::Bool, res_assemble::Bool, jac_assemble::Bool, solve::Bool, solver::Dict{Symbol, Any}, verbose::Bool, BL_adapted::Bool, kmap_x::Int64, kmap_y::Int64, ranks_per_level::Nothing) │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:160 │ [35] _hunt │ @ ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:41 [inlined] │ [36] #59 │ @ ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:23 [inlined] │ [37] with_mpi(f::GridapMHD.var"#59#62"{Tuple{Int64, Int64}, String, Base.Pairs{Symbol, Any, NTuple{11, Symbol}, NamedTuple{(:nc, :B, :L, :σw1, :σw2, :tw, :nsums, :vtk, :debug, :BL_adapted, :solver), Tuple{Tuple{Int64, Int64}, Tuple{Float64, Int64, Float64}, Float64, Float64, Float64, Float64, Int64, Bool, Bool, Bool, Dict{Symbol, Any}}}}, String}; comm::MPI.Comm, duplicate_comm::Bool) │ @ PartitionedArrays ~/.julia/packages/PartitionedArrays/py6uo/src/mpi_array.jl:73 │ [38] with_mpi │ @ ~/.julia/packages/PartitionedArrays/py6uo/src/mpi_array.jl:64 [inlined] │ [39] hunt(; backend::Symbol, np::Tuple{Int64, Int64}, title::String, nruns::Int64, path::String, kwargs::Base.Pairs{Symbol, Any, NTuple{11, Symbol}, NamedTuple{(:nc, :B, :L, :σw1, :σw2, :tw, :nsums, :vtk, :debug, :BL_adapted, :solver), Tuple{Tuple{Int64, Int64}, Tuple{Float64, Int64, Float64}, Float64, Float64, Float64, Float64, Int64, Bool, Bool, Bool, Dict{Symbol, Any}}}}) │ @ GridapMHD ~/software/GridapMHD/GridapMHD.jl/src/Applications/hunt.jl:22 │ [40] top-level scope │ @ none:19 │ [41] eval │ @ ./boot.jl:370 [inlined] │ [42] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:280 │ [43] _start() │ @ Base ./client.jl:522 └ @ PartitionedArrays ~/.julia/packages/PartitionedArrays/py6uo/src/mpi_array.jl:75 -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. [...] ```

Environment used:

I will report back should I find simpler or further examples to reproduce the issue.

pmartorell commented 2 months ago

Thanks for reporting @guillermogf ,

you forgot to mention that the error did not reproduce in 9ea6729 for the following configuration. However, this example is too large for debugging.

using GridapMHD: hunt
using SparseMatricesCSR
using GridapPETSc

solver = Dict(
    :solver => :petsc,
    :matrix_type    => SparseMatrixCSR{0,PetscScalar,PetscInt},
    :vector_type    => Vector{PetscScalar},
    :petsc_options  => "-snes_monitor -ksp_error_if_not_converged true \
                         -ksp_converged_reason -ksp_type preonly -pc_type lu \
                         -pc_factor_mat_solver_type mumps -mat_mumps_icntl_28 1 \
                         -mat_mumps_icntl_29 2 -mat_mumps_icntl_4 3 \
                         -mat_mumps_cntl_1 0.001",
    :niter          => 100,
    :rtol           => 1e-5,
)

hunt(;
  title="hunt_solidcoupling_Ha1000_120x120",
  backend=:mpi,
  np=(4,4),
  nc=(120,120),
  B=(0.0,1000,0.0),
  L=1.0,
  σw1=0.0001,
  σw2=1000.0,
  tw=0.1,
  nsums=100,
  vtk=true,
  debug=false,
  BL_adapted=true,
  solver=solver,
  path="../data/"
  )
pmartorell commented 2 months ago

FYI @principejavier