search

jipolanco / PencilArrays.jl

Distributed Julia arrays using the MPI protocol
https://jipolanco.github.io/PencilArrays.jl/dev/
MIT License
59 stars 8 forks source link

Writing grid obtained via `localgrid` to file #90

Open erny123 opened 4 months ago

erny123 commented 4 months ago

Looking to see how I could save other types from the PencilArrays package.

for example: 

using MPI
using PencilArrays
using HDF5

MPI.Init()
comm = MPI.COMM_WORLD

Nx, Ny, Nz = 65, 17, 21
xs = range(0, 1; length = Nx)
ys = range(-1, 1; length = Ny)
zs = range(0, 2; length = Nz)

pen = Pencil((Nx, Ny, Nz), comm)

grid = localgrid(pen, (xs, ys, zs))

fid = open(PHDF5Driver(),"test.hdf5", comm; write=true);

fid["grid"] = grid

This obviously doesn't work. The following error is from my own code but should be similar:

jfptr_task_done_hook_75297.1 at /data/homezvol0/ernestob/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_finish_task at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/task.c:320
start_task at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/task.c:1249
unknown function (ip: (nil))
btl_openib_component_progress at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/openmpi/mca_btl_openib.so (unknown line)
opal_progress at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libopen-pal.so.40 (unknown line)
ompi_sync_wait_mt at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libopen-pal.so.40 (unknown line)
ompi_request_default_wait at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libmpi.so (unknown line)
ompi_coll_base_sendrecv_actual at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libmpi.so (unknown line)
ompi_coll_base_allreduce_intra_recursivedoubling at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libmpi.so (unknown line)
ompi_coll_base_allreduce_intra_ring at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libmpi.so (unknown line)
ompi_coll_base_allreduce_intra_ring_segmented at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libmpi.so (unknown line)
ompi_coll_tuned_allreduce_intra_dec_fixed at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/openmpi/mca_coll_tuned.so (unknown line)
PMPI_Allreduce at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libmpi.so (unknown line)
H5D__mpio_opt_possible at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.souned.so (unknown line)
PMPI_Allreduce at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libmpi.so (unknown line)
H5D__mpio_opt_possible at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5D__ioinfo_adjust at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5D__write at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5VL__native_dataset_write at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5VL_dataset_write_direct at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5D__write_api_common.constprop.0 at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5Dwrite at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
h5d_write at /data/homezvol0/ernestob/.julia/packages/HDF5/Ws1wH/src/api/functions.jl:908
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
setindex! at /data/homezvol0/ernestob/.julia/packages/HDF5/Ws1wH/src/datasets.jl:381
setindex! at /data/homezvol0/ernestob/.julia/packages/HDF5/Ws1wH/src/datasets.jl:439 [inlined]
to_hdf5 at /data/homezvol0/ernestob/.julia/packages/PencilArrays/D1kJx/ext/PencilArraysHDF5Ext.jl:188 [inlined]
macro expansion at /data/homezvol0/ernestob/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:253 [inlined]
macro expansion at /data/homezvol0/ernestob/.julia/packages/PencilArrays/D1kJx/ext/PencilArraysHDF5Ext.jl:117 [inlined]
macro expansion at /data/homezvol0/ernestob/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:253 [inlined]
#setindex!#4 at /data/homezvol0/ernestob/.julia/packages/PencilArrays/D1kJx/ext/PencilArraysHDF5Ext.jl:96
setindex! at /data/homezvol0/ernestob/.julia/packages/PencilArrays/D1kJx/ext/PencilArraysHDF5Ext.jl:89
unknown function (ip: 0x7fdbf01a94c7)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
include_string at ./loading.jl:2076
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
_include at ./loading.jl:2136
include at ./Base.jl:495
jfptr_include_46403.1 at /data/homezvol0/ernestob/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
exec_options at ./client.jl:318
_start at ./client.jl:552
jfptr__start_82738.1 at /data/homezvol0/ernestob/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/buned.so (unknown line)
PMPI_Allreduce at /opt/apps/openmpi/4.1.2/gcc/11.2.0/lib/libmpi.so (unknown line)
H5D__mpio_opt_possible at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5D__ioinfo_adjust at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5D__write at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5VL__native_dataset_write at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5VL_dataset_write_direct at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5D__write_api_common.constprop.0 at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
H5Dwrite at /opt/apps/hdf5/1.14.1/gcc/11.2.0/openmpi/4.1.2/lib/libhdf5.so (unknown line)
h5d_write at /data/homezvol0/ernestob/.julia/packages/HDF5/Ws1wH/src/api/functions.jl:908
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
setindex! at /data/homezvol0/ernestob/.julia/packages/HDF5/Ws1wH/src/datasets.jl:381
setindex! at /data/homezvol0/ernestob/.julia/packages/HDF5/Ws1wH/src/datasets.jl:439 [inlined]
to_hdf5 at /data/homezvol0/ernestob/.julia/packages/PencilArrays/D1kJx/ext/PencilArraysHDF5Ext.jl:188 [inlined]
macro expansion at /data/homezvol0/ernestob/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:253 [inlined]
macro expansion at /data/homezvol0/ernestob/.julia/packages/PencilArrays/D1kJx/ext/PencilArraysHDF5Ext.jl:117 [inlined]
macro expansion at /data/homezvol0/ernestob/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:253 [inlined]
#setindex!#4 at /data/homezvol0/ernestob/.julia/packages/PencilArrays/D1kJx/ext/PencilArraysHDF5Ext.jl:96
setindex! at /data/homezvol0/ernestob/.julia/packages/PencilArrays/D1kJx/ext/PencilArraysHDF5Ext.jl:89
jipolanco commented 3 months ago

The PHDF5Driver (and the MPIIODriver for that matter) does not currently support writing grid objects returned by localgrid. This is definitely something which would be interesting to have.