Closed bennn closed 1 year ago
ashton314
commented
1 year ago Reproduced all results. See message on ac05aec
Short of it: on that commit you should be able to reproduce all the results with:
for tst in $(ls advection_recordings); do echo "-- Testing $tst at $(date)"; julia --project=. -- examples/finch/injection_investigator.jl --replay advection_recordings/$tst; doneI like having command line args we can pass to test files—that might be a little bit of polish to consider adding: an optional module that lets you easily pass recording files into your test scripts.
bennn
commented
1 year ago For 28, refel_nodes.jl:
refel_nodes.jl in abs here: qf3(x) = abs(x[1] - 1) < tol
grid.jl because fpi is too big in tmpf2glb[fpi] = loc2glb[refeli.face2local[gfi][fpi], ei];refeli.Nfp[x] = length(refeli.face2local[gfi])qf3 is refel.r, defined up aboverefel.r starts as a vector with zeros, but gets updated with values from refel.r1drefel.r1d = [-1; r; 1](r,w) = jacobi_gauss_quad(1,1,refel.N-2) , from Finch's jacobi_gauss_quad.jlSummary: i don't see a bug. More likely, injection should not target the function refel_nodes! at all.
bennn
commented
1 year ago @ashton314 what's the incantation to make injection ignore the function refel_nodes! from Finch's refel_nodes.jl?
In other news, the 2nd NaN appears at l.248 of grid.jl. Looks like the only way for a real NaN to appear there is for the mesh object referenced on l.238 to return a NaN. Can it? I'll check.
EDIT: a bad mesh can put a NaN on this line. Finch should check mesh files --- need to open an issue or PR. EDIT: Finch issue here
ashton314
commented
1 year ago It's a list of FunctionRef structs. (This should be documented better.) See Injector.jl for the source, and example-poisson1d.jl in this repo for an example of usage.
I think you'd do
fns = [FunctionRef(:refel_nodes!, Symbol("refel_nodes.jl"))]
set_inject_nan(..., fns ...)Oh no, that injects only in refel_nodes!. There is no way to ignore yet.
bennn
commented
1 year ago Closing b/c the 4 tasks are done, but it's worth trying this again outside the mesh-creation code.
I ran Finch's 2d advection a few times with injection & recording turned on. Some runs gave errors. Some were ok (not shown below). One run was 30x slower than normal.
TODO: get a sense of what's happening in each
should injection be turned off for some libs or functions ?
are there any Finch bugs lurking?
The program injects 1 NaN at 1/1000 odds.
``` #= 2D advection using higher order FV or structured or unstructured mesh =# ### If the Finch package has already been added, use this line ######### using Finch # Note: to add the package, first do: ]add "https://github.com/paralab/Finch.git" using Dates using FloatTracker: TrackedFloat64, write_log_to_file, set_inject_nan, set_logger, set_exclude_stacktrace set_exclude_stacktrace([:prop]) set_logger(filename="inj-adv2d", buffersize=20, cstg=true, cstgArgs=false, cstgLineNum=true) fns = [] ##[FunctionRef(:run_simulation, Symbol("nbody_simulation_result.jl"))] libs = [] ##["NBodySimulator", "OrdinaryDiffEq"] now_str = Dates.format(now(), "yyyymmddHHMMss") recording_file = "rand-adv2d-recording_$now_str" println("Recording to $recording_file...") ### If not, use these four lines (working from the examples directory) ### # if !@isdefined(Finch) # include("../Finch.jl"); # using .Finch # end ########################################################################## initFinch("inj-adv2d", TrackedFloat64); # Configuration setup domain(2) solverType(FV) timeStepper(EULER_EXPLICIT,cfl=20000) # NaN-making timestepper ######## set_inject_nan(true, 1000, 1, fns, libs, record=recording_file) ######## println("begin MESH") # a uniform grid of quads on a 0.1 x 0.3 rectangle domain mesh(QUADMESH, elsperdim=[15, 45], bids=4, interval=[0, 0.1, 0, 0.3]) println("end MESH") # Variables and BCs u = variable("u", location=CELL) boundary(u, 1, FLUX, "(abs(y-0.06) < 0.033 && sin(3*pi*t)>0) ? 1 : 0") # x=0 boundary(u, 2, NO_BC) # x=0.1 boundary(u, 3, NO_BC) # y=0 boundary(u, 4, NO_BC) # y=0.3 # Time interval and initial condition T = 1.3; # T=10 timeInterval(T) initial(u, "0") # Coefficients coefficient("a", ["0.1*cos(pi*x/2/0.1)","0.3*sin(pi*x/2/0.1)"], type=VECTOR) # advection velocity coefficient("s", ["0.1 * sin(pi*x)^4 * sin(pi*y)^4"]) # source # The "upwind" function applies upwinding to the term (a.n)*u with flow velocity a. # The optional third parameter is for tuning. Default upwind = 0, central = 1. Choose something between these. conservationForm(u, "s + surface(upwind(a,u))"); println("begin SOLVE") solve(u) println("end SOLVE") finalizeFinch() write_log_to_file() ```Recordings & Results
28, refel_nodes.jlLoadError: BoundsError: attempt to access 1-element Vector{Int64} at index [2]
``` ERROR: LoadError: BoundsError: attempt to access 1-element Vector{Int64} at index [2] Stacktrace: [1] getindex @ ./array.jl:924 [inlined] [2] grid_from_mesh(mesh::MeshData; grid_type::String, order::Int64, mixed::Bool) @ Finch ~/.julia/packages/Finch/qXoRn/src/grid.jl:366 [3] macro expansion @ ~/.julia/packages/Finch/qXoRn/src/Finch.jl:252 [inlined] [4] macro expansion @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined] [5] add_mesh(state::FinchState{TrackedFloat64}, mesh::MeshData; partitions::Int64) @ Finch ~/.julia/packages/Finch/qXoRn/src/Finch.jl:145 [6] macro expansion @ ~/.julia/packages/Finch/qXoRn/src/finch_interface.jl:414 [inlined] [7] macro expansion @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined] [8] mesh(msh::String; elsperdim::Vector{Int64}, bids::Int64, interval::Vector{Float64}, partitions::Int64) @ Finch ~/.julia/packages/Finch/qXoRn/src/finch_interface.jl:359 [9] top-level scope @ ~/code/uu/fpx/examples/examples/finch/inj-adv2d-fv.jl:41 in expression starting at /home/ben/code/uu/fpx/examples/examples/finch/inj-adv2d-fv.jl:41 ```20681, grid.jlLoadError: BoundsError: attempt to access 2×736 Matrix{TrackedFloat64} at index [1, 0]
``` ERROR: LoadError: BoundsError: attempt to access 2×736 Matrix{TrackedFloat64} at index [1, 0] Stacktrace: [1] getindex @ ./array.jl:925 [inlined] [2] build_geometric_factors(refel::Refel{TrackedFloat64}, grid::Grid{TrackedFloat64}; do_face_detj::Bool, do_vol_area::Bool, constant_jacobian::Bool) @ Finch ~/.julia/packages/Finch/qXoRn/src/geometric_factors.jl:96 [3] macro expansion @ ~/.julia/packages/Finch/qXoRn/src/Finch.jl:283 [inlined] [4] macro expansion @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined] [5] add_mesh(state::FinchState{TrackedFloat64}, mesh::MeshData; partitions::Int64) @ Finch ~/.julia/packages/Finch/qXoRn/src/Finch.jl:282 [6] macro expansion @ ~/.julia/packages/Finch/qXoRn/src/finch_interface.jl:414 [inlined] [7] macro expansion @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined] [8] mesh(msh::String; elsperdim::Vector{Int64}, bids::Int64, interval::Vector{Float64}, partitions::Int64) @ Finch ~/.julia/packages/Finch/qXoRn/src/finch_interface.jl:359 [9] top-level scope @ ~/code/uu/fpx/examples/examples/finch/inj-adv2d-fv.jl:41 in expression starting at /home/ben/code/uu/fpx/examples/examples/finch/inj-adv2d-fv.jl:41 ```46086, grid.jl(maybe yes: https://github.com/paralab/Finch/issues/17 .... old notes: same cause as#2above, NaN appears at l.248 ofgrid.jl, but ends up crashinggeometric_factors.jl... well hang ongeometric_factorsmight need defense)LoadError: BoundsError: attempt to access 2×736 Matrix{TrackedFloat64} at index [1, 0]
``` ERROR: LoadError: BoundsError: attempt to access 2×736 Matrix{TrackedFloat64} at index [1, 0] Stacktrace: [1] getindex @ ./array.jl:925 [inlined] [2] build_geometric_factors(refel::Refel{TrackedFloat64}, grid::Grid{TrackedFloat64}; do_face_detj::Bool, do_vol_area::Bool, constant_jacobian::Bool) @ Finch ~/.julia/packages/Finch/qXoRn/src/geometric_factors.jl:96 [3] macro expansion @ ~/.julia/packages/Finch/qXoRn/src/Finch.jl:283 [inlined] [4] macro expansion @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined] [5] add_mesh(state::FinchState{TrackedFloat64}, mesh::MeshData; partitions::Int64) @ Finch ~/.julia/packages/Finch/qXoRn/src/Finch.jl:282 [6] macro expansion @ ~/.julia/packages/Finch/qXoRn/src/finch_interface.jl:414 [inlined] [7] macro expansion @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:237 [inlined] [8] mesh(msh::String; elsperdim::Vector{Int64}, bids::Int64, interval::Vector{Float64}, partitions::Int64) @ Finch ~/.julia/packages/Finch/qXoRn/src/finch_interface.jl:359 [9] top-level scope @ ~/code/uu/fpx/examples/examples/finch/inj-adv2d-fv.jl:41 in expression starting at /home/ben/code/uu/fpx/examples/examples/finch/inj-adv2d-fv.jl:41 ```80861, grid.jlThis one looks bogus. Injects at l.1523 ofgrid.jl:xlim[1] = min(xlim[1], x);. Butxlimstarts off initialized andxcomes from thenodesmatrix ... very unlikely to have a NaN.Very slow
Output: ``` Recording to rand-adv2d-recording_202304251302329... begin MESH end MESH begin SOLVE Time step progress(%) 01020end SOLVE ──────────────────────────────────────────────────────────────────────────────── Time Allocations ─────────────────────── ──────────────────────── Tot / % measured: 2210s / 98.4% 75.6GiB / 91.1% Section ncalls time %tot avg alloc %tot avg ──────────────────────────────────────────────────────────────────────────────── Mesh 1 2165s 99.5% 2165s 68.1GiB 98.9% 68.1GiB mesh2grid 1 2157s 99.2% 2157s 67.3GiB 97.8% 67.3GiB bids 1 1.45s 0.1% 1.45s 171MiB 0.2% 171MiB geo factors 1 766ms 0.0% 766ms 87.4MiB 0.1% 87.4MiB fv_info 1 128ms 0.0% 128ms 12.0MiB 0.0% 12.0MiB gen/read 1 346ms 0.0% 346ms 12.9MiB 0.0% 12.9MiB CodeGen 1 9.27s 0.4% 9.27s 693MiB 1.0% 693MiB Solve 1 1.20s 0.1% 1.20s 110MiB 0.2% 110MiB FV_solve 1 925ms 0.0% 925ms 82.5MiB 0.1% 82.5MiB time_steps 1 96.4ms 0.0% 96.4ms 20.7MiB 0.0% 20.7MiB step_assembly 2 96.1ms 0.0% 48.1ms 20.5MiB 0.0% 10.2MiB update_sol 2 121μs 0.0% 60.5μs 211KiB 0.0% 105KiB allocate 1 9.71μs 0.0% 9.71μs 29.2KiB 0.0% 29.2KiB ──────────────────────────────────────────────────────────────────────────────── Finch has completed. ``` Typical Output: ``` Recording to rand-adv2d-recording_202304251300023... begin MESH end MESH begin SOLVE Time step progress(%) 01020end SOLVE ──────────────────────────────────────────────────────────────────────────────── Time Allocations ─────────────────────── ──────────────────────── Tot / % measured: 64.0s / 46.8% 9.20GiB / 26.5% Section ncalls time %tot avg alloc %tot avg ──────────────────────────────────────────────────────────────────────────────── Mesh 1 19.4s 64.6% 19.4s 1.66GiB 67.9% 1.66GiB mesh2grid 1 11.5s 38.5% 11.5s 918MiB 36.7% 918MiB bids 1 1.43s 4.8% 1.43s 171MiB 6.8% 171MiB geo factors 1 722ms 2.4% 722ms 87.4MiB 3.5% 87.4MiB fv_info 1 85.9ms 0.3% 85.9ms 12.0MiB 0.5% 12.0MiB gen/read 1 348ms 1.2% 348ms 12.9MiB 0.5% 12.9MiB CodeGen 1 9.47s 31.6% 9.47s 693MiB 27.7% 693MiB Solve 1 1.14s 3.8% 1.14s 110MiB 4.4% 110MiB FV_solve 1 888ms 3.0% 888ms 82.5MiB 3.3% 82.5MiB time_steps 1 74.2ms 0.2% 74.2ms 20.7MiB 0.8% 20.7MiB step_assembly 2 74.0ms 0.2% 37.0ms 20.5MiB 0.8% 10.2MiB update_sol 2 122μs 0.0% 61.2μs 211KiB 0.0% 105KiB allocate 1 10.5μs 0.0% 10.5μs 29.2KiB 0.0% 29.2KiB ──────────────────────────────────────────────────────────────────────────────── Finch has completed. ```