JuliaCon review: errors

[ranocha]

This is part of the review https://github.com/JuliaCon/proceedings-review/issues/165

I installed the package on Julia 1.10 and started the tests. They errored with several messages like

set_progress_bars!: Log Test Failed at ~/.julia/packages/Peridynamics/SePS6/test/auxiliary/test_logs.jl:15
  Expression: Peridynamics.set_progress_bars!()
  Log Pattern: (:warn,)
  Captured Logs: 

and

┌ Error: Multibody simulations with MPI are not yet implemented!
└ @ Peridynamics ~/.julia/packages/Peridynamics/SePS6/src/discretization/multibody_setup.jl:122
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=1 message=application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
:
system msg for write_line failure : Bad file descriptor
ERROR: Package Peridynamics errored during testing

Please fix the tests so that they can be run locally.

[kaipartmann]

Hello @ranocha, Thank you very much for your review and for finding these errors! I could not reproduce all the errors, but I think the changes made in https://github.com/kaipartmann/Peridynamics.jl/pull/169 should fix them. Please test the code in the branch juliacon_review_errors and check if my changes make the tests work on your machine. Please also see #170 for the bug that should be the root cause of the errors.

[ranocha]

I get

julia> Pkg.add(url="git@github.com:kaipartmann/Peridynamics.jl.git", rev="juliacon_review_errors")
[...]

(jl_vbKyJL) pkg> test Peridynamics
[...]
Precompiling project...
  11 dependencies successfully precompiled in 18 seconds. 69 already precompiled.
     Testing Running tests...
┌ Warning: parameters for CKIMaterial specified manually!
│ Be careful when adjusting these parameters to avoid unexpected outcomes!
└ @ Peridynamics ~/.julia/packages/Peridynamics/cinip/src/physics/continuum_kinematics_inspired.jl:110
┌ Warning: parameters for CKIMaterial specified manually!
│ Be careful when adjusting these parameters to avoid unexpected outcomes!
└ @ Peridynamics ~/.julia/packages/Peridynamics/cinip/src/physics/continuum_kinematics_inspired.jl:110
┌ Warning: parameters for CKIMaterial specified manually!
│ Be careful when adjusting these parameters to avoid unexpected outcomes!
└ @ Peridynamics ~/.julia/packages/Peridynamics/cinip/src/physics/continuum_kinematics_inspired.jl:110
                                                               _
           _____         _     _                             _(_)_
          | ___ \       (_)   | |                           (_) (_)
          | |_/ /__ _ __ _  __| |_   _ _ __   __ _ _ __ ___  _  ___ ___
          |  __/ _ \ '__| |/ _` | | | | '_ \ / _` | '_ ` _ \| |/ __/ __|
          | | |  __/ |  | | (_| | |_| | | | | (_| | | | | | | | (__\__ \
          \_|  \___|_|  |_|\__,_|\__, |_| |_|\__,_|_| |_| |_|_|\___|___/
                                  __/ |
                                 |___/   Copyright (c) 2024 Kai Partmann

MULTITHREADING SIMULATION WITH 1 THREADS
BODY
  POINT CLOUD
    number of points ........................................................ 2700
    min, max values x-direction ............................ -0.4833333, 0.4833333
    min, max values y-direction ............................ -0.4833333, 0.4833333
    min, max values z-direction .......................... -0.03333333, 0.03333333
  POINT SETS
    number of points in set `all_points` .................................... 2700
    number of points in set `set_top` ......................................... 90
    number of points in set `set_bottom` ...................................... 90
    number of points in set `set_a` .......................................... 270
    number of points in set `set_b` .......................................... 270
  BOUNDARY CONDITIONS
    velocity condition ............................. set `set_bottom`, dimension 2
    velocity condition ................................ set `set_top`, dimension 2
  MATERIAL
    material type ............. Peridynamics.BBMaterial{Peridynamics.NoCorrection}
    horizon ............................................................... 0.1005
    density ................................................................ 8e-06
    Young's modulus ....................................................... 210000
    Poisson's ratio ......................................................... 0.25
    shear modulus .......................................................... 84000
    bulk modulus .......................................................... 140000
DATA HANDLER CREATION COMPLETED ✔
BOND SYSTEM
  number of bonds ......................................................... 187372
VELOCITY VERLET TIME SOLVER
  number of time steps ....................................................... 100
  time step size .................................................... 1.493972e-07
  time step safety factor .................................................... 0.7
  simulation time ................................................... 1.493972e-05
SIMULATION COMPLETED AFTER 0.209326 SECONDS ✔
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399)........: MPI_Init_thread(argc=0x0, argv=0x0, required=2, provided=0x16d99da20) failed
MPII_Init_thread(242)..............: 
MPID_Init(67)......................: 
init_world(171)....................: channel initialization failed
MPIDI_CH3_Init(84).................: 
MPID_nem_init(314).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(397): 
GetSockInterfaceAddr(370)..........: gethostbyname failed, mac1783 (errno 0)
ERROR: LoadError: failed process: Process(setenv(`~/.julia/artifacts/6cc2bbe71648ff80b2aa4fe2795b862d148fd829/bin/mpiexec -n 2 julia -C native --depwarn=yes --check-bounds=yes -g1 --color=yes --startup-file=no --project -e 'using Peridynamics
function sim_bb(N::Int, path::String)
    l, Δx, δ, a = 1.0, 1/N, 3.015/N, 0.5
    pos, vol = uniform_box(l, l, 0.1l, Δx)
    ids = sortperm(pos[2,:])
    b = Body(BBMaterial(), pos[:, ids], vol[ids])
    material!(b; horizon=3.015Δx, E=2.1e5, rho=8e-6, Gc=2.7)
    point_set!(p -> p[1] ≤ -l/2+a && 0 ≤ p[2] ≤ 2δ, b, :set_a)
    point_set!(p -> p[1] ≤ -l/2+a && -2δ ≤ p[2] < 0, b, :set_b)
    precrack!(b, :set_a, :set_b)
    point_set!(p -> p[2] > l/2-Δx, b, :set_top)
    point_set!(p -> p[2] < -l/2+Δx, b, :set_bottom)
    velocity_bc!(t -> -30, b, :set_bottom, :y)
    velocity_bc!(t -> 30, b, :set_top, :y)
    vv = VelocityVerlet(steps=100)
    job = Job(b, vv; path=path, freq=50)
    submit(job)
    return nothing
end
sim_bb(30, "~/.julia/packages/Peridynamics/cinip/test/integration/temp_mpi_threads_comparison/results_mpi")
[...]
Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run(::Cmd)
    @ Base ./process.jl:477
  [4] top-level scope
    @ ~/.julia/packages/Peridynamics/cinip/test/integration/mpi_threads_comparison.jl:50
  [5] eval
    @ ./boot.jl:385 [inlined]
  [6] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:2076
  [7] include_string(m::Module, txt::String, fname::String)
    @ Base ./loading.jl:2086
  [8] #invokelatest#2
    @ ./essentials.jl:892 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] #8
    @ ~/.julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl:118 [inlined]
 [11] withpath(f::TestItemRunner.var"#8#12"{String, String, Module}, path::String)
    @ TestItemRunner ~/.julia/packages/TestItemRunner/Qioo0/src/vendored_code.jl:7
 [12] #7
    @ ~/.julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl:117 [inlined]
 [13] cd(f::TestItemRunner.var"#7#11"{String, String, Module}, dir::String)
    @ Base.Filesystem ./file.jl:112
 [14] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any})
    @ TestItemRunner ~/.julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl:116
 [15] run_tests(path::String; filter::Nothing, verbose::Bool)
    @ TestItemRunner ~/.julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl:221
 [16] top-level scope
    @ ~/.julia/packages/Peridynamics/cinip/test/runtests.jl:3
 [17] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [18] top-level scope
    @ none:6
in expression starting at ~/.julia/packages/Peridynamics/cinip/test/integration/mpi_threads_comparison.jl:50
in expression starting at ~/.julia/packages/Peridynamics/cinip/test/runtests.jl:3
ERROR: Package Peridynamics errored during testing

[kaipartmann]

Hello @ranocha, thank you very much for your help! So the remaining errors are an issue with these tests that run MPI like suggested in the MPI.jl docs: https://github.com/kaipartmann/Peridynamics.jl/blob/main/test/integration/mpi_threads_comparison.jl

Can you run this code in the same environment?

julia> mpi_cmd = """
       using Peridynamics
       using Peridynamics: mpi_rank, mpi_run
       @mpiroot sleep(0.5)
       @show mpi_rank(), mpi_run()
       """
"using Peridynamics\nusing Peridynamics: mpi_rank, mpi_run\n@mpiroot sleep(0.5)\n@show mpi_rank(), mpi_run()\n"

julia> run(`$(Peridynamics.MPI.mpiexec()) -n 2 $(Base.julia_cmd()) --project -e $(mpi_cmd)`);
(mpi_rank(), mpi_run()) = (1, true)
(mpi_rank(), mpi_run()) = (0, true)

If not, maybe it could be an issue with the MPI installation.

[ranocha]

julia> run(`$(Peridynamics.MPI.mpiexec()) -n 2 $(Base.julia_cmd()) --project -e $(mpi_cmd)`);
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399)........: MPI_Init_thread(argc=0x0, argv=0x0, required=2, provided=0x16b195d80) failed
MPII_Init_thread(242)..............: 
MPID_Init(67)......................: 
init_world(171)....................: channel initialization failed
MPIDI_CH3_Init(84).................: 
MPID_nem_init(314).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(397): 
GetSockInterfaceAddr(370)..........: gethostbyname failed, mac (errno 0)
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399)........: MPI_Init_thread(argc=0x0, argv=0x0, required=2, provided=0x16dd59d80) failed
MPII_Init_thread(242)..............: 
MPID_Init(67)......................: 
init_world(171)....................: channel initialization failed
MPIDI_CH3_Init(84).................: 
MPID_nem_init(314).................: 
MPID_nem_tcp_init(175).............: 
MPID_nem_tcp_get_business_card(397): 
GetSockInterfaceAddr(370)..........: gethostbyname failed, mac1783 (errno 0)
ERROR: failed process: Process(setenv(`~/.julia/artifacts/6cc2bbe71648ff80b2aa4fe2795b862d148fd829/bin/mpiexec -n 2 .../julia -C native -g1 --project -e 'using Peridynamics
using Peridynamics: mpi_rank, mpi_run
@mpiroot sleep(0.5)
@show mpi_rank(), mpi_run()
...

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:480
 [3] run(::Cmd)
   @ Base ./process.jl:477
 [4] top-level scope
   @ REPL[6]:1

[kaipartmann]

@ranocha Can you run this code? This is just the hello world example of the MPI.jl package:

julia> mpi_cmd = raw"""
       using Peridynamics
       using Peridynamics: MPI
       MPI.Init()
       comm = MPI.COMM_WORLD
       print("Hello world, I am rank $(MPI.Comm_rank(comm)) of $(MPI.Comm_size(comm))\n")
       MPI.Barrier(comm)
       """;

julia> run(`$(Peridynamics.MPI.mpiexec()) -n 2 $(Base.julia_cmd()) --project -e $(mpi_cmd)`);
Hello world, I am rank 0 of 2
Hello world, I am rank 1 of 2

If not, then please add MPI

(jl_vbKyJL) pkg> add MPI

and try again. Thank you very much for your ongoing help and support!

[ranocha]

Looks like for some reason I got the problem described at https://juliaparallel.org/MPI.jl/latest/knownissues/#MPICH This fixes the MWE above.

However, running tests with Peridynamics v0.3.2 locally, I get

┌ Error: Multibody simulations with MPI are not yet implemented!
└ @ Peridynamics ~/.julia/packages/Peridynamics/SePS6/src/discretization/multibody_setup.jl:122
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=1 message=application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
:
system msg for write_line failure : Bad file descriptor
ERROR: Package Peridynamics errored during testing

as last error output

[ranocha]

I also get some additional errors earlier, e.g.,

log_create_data_handler: Test Failed at ~/.julia/packages/Peridynamics/SePS6/test/auxiliary/test_logs.jl:46
  Expression: Peridynamics.progress_bars() == false
   Evaluated: true == false

[kaipartmann]

running tests with Peridynamics v0.3.2 locally

So did you try to run the tests using the version in the juliacon_review_errors branch?

However, this seems to be related to #170. Do you have an environment parameter MPI_LOCALRANKID defined at your system? If yes, then mpi_run_initial_check would always decide to use the MPI backend, which could result in the errors.

Thank you again for your patience and ongoing support regarding these issues! It really helps to improve the package!

[ranocha]

If you think your PR fixes the issue, could you please make it available in the main release? It's kind of annoying for me to test a PR branch.

I don't have MPI_LOCALRANKID defined

[kaipartmann]

@ranocha The PR has now been merged with the main branch. I'm really sorry for any inconvenience it may have caused.