Closed n01r closed 6 years ago
The data set that we read is a 1D array written by several process groups. At the offset of concern, a process group wrote zero entries. This is an issue we encountered (& fixed) before, e.g. with zlib transforms.
The numpy wrapper version is:
import adios as ad
ad.__version__
'1.13.0'
The blockinfo
from the file shows
In[9]: px.blockinfo
Out[9]:
[[AdiosBlockinfo (process_id=0, time_index=1, start=(0,), count=(19,)),
AdiosBlockinfo (process_id=1, time_index=1, start=(19,), count=(45,)),
AdiosBlockinfo (process_id=2, time_index=1, start=(64,), count=(1477,)),
AdiosBlockinfo (process_id=3, time_index=1, start=(1541,), count=(61,)),
AdiosBlockinfo (process_id=4, time_index=1, start=(1602,), count=(82,)),
AdiosBlockinfo (process_id=5, time_index=1, start=(1684,), count=(1154,)),
AdiosBlockinfo (process_id=6, time_index=1, start=(2838,), count=(22,)),
AdiosBlockinfo (process_id=7, time_index=1, start=(2860,), count=(46,)),
AdiosBlockinfo (process_id=8, time_index=1, start=(2906,), count=(570,)),
AdiosBlockinfo (process_id=9, time_index=1, start=(3476,), count=(18,)),
AdiosBlockinfo (process_id=10, time_index=1, start=(3494,), count=(18,)),
AdiosBlockinfo (process_id=11, time_index=1, start=(3512,), count=(198,)),
AdiosBlockinfo (process_id=12, time_index=1, start=(3710,), count=(16,)),
AdiosBlockinfo (process_id=13, time_index=1, start=(3726,), count=(4,)),
AdiosBlockinfo (process_id=14, time_index=1, start=(3730,), count=(2,)),
AdiosBlockinfo (process_id=15, time_index=1, start=(3732,), count=(4,)),
AdiosBlockinfo (process_id=16, time_index=1, start=(3736,), count=(5,)),
AdiosBlockinfo (process_id=17, time_index=1, start=(3741,), count=(0,)),
AdiosBlockinfo (process_id=18, time_index=1, start=(3741,), count=(2,)),
AdiosBlockinfo (process_id=19, time_index=1, start=(3743,), count=(1,)),
AdiosBlockinfo (process_id=20, time_index=1, start=(3744,), count=(1,)),
AdiosBlockinfo (process_id=21, time_index=1, start=(3745,), count=(0,)),
AdiosBlockinfo (process_id=22, time_index=1, start=(3745,), count=(2,)),
AdiosBlockinfo (process_id=23, time_index=1, start=(3747,), count=(2,)),
AdiosBlockinfo (process_id=24, time_index=1, start=(3749,), count=(1,)),
AdiosBlockinfo (process_id=25, time_index=1, start=(3750,), count=(1,)),
AdiosBlockinfo (process_id=26, time_index=1, start=(3751,), count=(2,)),
AdiosBlockinfo (process_id=27, time_index=1, start=(3753,), count=(0,)),
AdiosBlockinfo (process_id=28, time_index=1, start=(3753,), count=(0,)),
AdiosBlockinfo (process_id=29, time_index=1, start=(3753,), count=(2,)),
AdiosBlockinfo (process_id=30, time_index=1, start=(3755,), count=(0,)),
AdiosBlockinfo (process_id=31, time_index=1, start=(3755,), count=(1,)),
AdiosBlockinfo (process_id=32, time_index=1, start=(3756,), count=(1,)),
AdiosBlockinfo (process_id=33, time_index=1, start=(3757,), count=(0,)),
AdiosBlockinfo (process_id=34, time_index=1, start=(3757,), count=(2,)),
AdiosBlockinfo (process_id=35, time_index=1, start=(3759,), count=(1,)),
AdiosBlockinfo (process_id=36, time_index=1, start=(3760,), count=(2,)),
AdiosBlockinfo (process_id=37, time_index=1, start=(3762,), count=(6,)),
AdiosBlockinfo (process_id=38, time_index=1, start=(3768,), count=(3,)),
AdiosBlockinfo (process_id=39, time_index=1, start=(3771,), count=(3,)),
AdiosBlockinfo (process_id=40, time_index=1, start=(3774,), count=(3,)),
AdiosBlockinfo (process_id=41, time_index=1, start=(3777,), count=(0,)),
AdiosBlockinfo (process_id=42, time_index=1, start=(3777,), count=(7,)),
AdiosBlockinfo (process_id=43, time_index=1, start=(3784,), count=(3,)),
AdiosBlockinfo (process_id=44, time_index=1, start=(3787,), count=(0,)),
AdiosBlockinfo (process_id=45, time_index=1, start=(3787,), count=(10,)),
AdiosBlockinfo (process_id=46, time_index=1, start=(3797,), count=(15,)),
AdiosBlockinfo (process_id=47, time_index=1, start=(3812,), count=(0,)),
AdiosBlockinfo (process_id=48, time_index=1, start=(3812,), count=(7,)),
AdiosBlockinfo (process_id=49, time_index=1, start=(3819,), count=(21,)),
AdiosBlockinfo (process_id=50, time_index=1, start=(3840,), count=(171,)),
AdiosBlockinfo (process_id=51, time_index=1, start=(4011,), count=(7,)),
AdiosBlockinfo (process_id=52, time_index=1, start=(4018,), count=(50,)),
AdiosBlockinfo (process_id=53, time_index=1, start=(4068,), count=(593,)),
AdiosBlockinfo (process_id=54, time_index=1, start=(4661,), count=(34,)),
AdiosBlockinfo (process_id=55, time_index=1, start=(4695,), count=(48,)),
AdiosBlockinfo (process_id=56, time_index=1, start=(4743,), count=(1264,)),
AdiosBlockinfo (process_id=57, time_index=1, start=(6007,), count=(16,)),
AdiosBlockinfo (process_id=58, time_index=1, start=(6023,), count=(28,)),
AdiosBlockinfo (process_id=59, time_index=1, start=(6051,), count=(1537,))]]
I am looking into this.
On Wed, Feb 14, 2018 at 5:04 AM, Marco Garten notifications@github.com wrote:
The numpy wrapper version is:
import adios as ad ad.version'1.13.0'
The blockinfo from the file shows
In[9]: px.blockinfo Out[9]: [[AdiosBlockinfo (process_id=0, time_index=1, start=(0,), count=(19,)), AdiosBlockinfo (process_id=1, time_index=1, start=(19,), count=(45,)), AdiosBlockinfo (process_id=2, time_index=1, start=(64,), count=(1477,)), AdiosBlockinfo (process_id=3, time_index=1, start=(1541,), count=(61,)), AdiosBlockinfo (process_id=4, time_index=1, start=(1602,), count=(82,)), AdiosBlockinfo (process_id=5, time_index=1, start=(1684,), count=(1154,)), AdiosBlockinfo (process_id=6, time_index=1, start=(2838,), count=(22,)), AdiosBlockinfo (process_id=7, time_index=1, start=(2860,), count=(46,)), AdiosBlockinfo (process_id=8, time_index=1, start=(2906,), count=(570,)), AdiosBlockinfo (process_id=9, time_index=1, start=(3476,), count=(18,)), AdiosBlockinfo (process_id=10, time_index=1, start=(3494,), count=(18,)), AdiosBlockinfo (process_id=11, time_index=1, start=(3512,), count=(198,)), AdiosBlockinfo (process_id=12, time_index=1, start=(3710,), count=(16,)), AdiosBlockinfo (process_id=13, time_index=1, start=(3726,), count=(4,)), AdiosBlockinfo (process_id=14, time_index=1, start=(3730,), count=(2,)), AdiosBlockinfo (process_id=15, time_index=1, start=(3732,), count=(4,)), AdiosBlockinfo (process_id=16, time_index=1, start=(3736,), count=(5,)), AdiosBlockinfo (process_id=17, time_index=1, start=(3741,), count=(0,)), AdiosBlockinfo (process_id=18, time_index=1, start=(3741,), count=(2,)), AdiosBlockinfo (process_id=19, time_index=1, start=(3743,), count=(1,)), AdiosBlockinfo (process_id=20, time_index=1, start=(3744,), count=(1,)), AdiosBlockinfo (process_id=21, time_index=1, start=(3745,), count=(0,)), AdiosBlockinfo (process_id=22, time_index=1, start=(3745,), count=(2,)), AdiosBlockinfo (process_id=23, time_index=1, start=(3747,), count=(2,)), AdiosBlockinfo (process_id=24, time_index=1, start=(3749,), count=(1,)), AdiosBlockinfo (process_id=25, time_index=1, start=(3750,), count=(1,)), AdiosBlockinfo (process_id=26, time_index=1, start=(3751,), count=(2,)), AdiosBlockinfo (process_id=27, time_index=1, start=(3753,), count=(0,)), AdiosBlockinfo (process_id=28, time_index=1, start=(3753,), count=(0,)), AdiosBlockinfo (process_id=29, time_index=1, start=(3753,), count=(2,)), AdiosBlockinfo (process_id=30, time_index=1, start=(3755,), count=(0,)), AdiosBlockinfo (process_id=31, time_index=1, start=(3755,), count=(1,)), AdiosBlockinfo (process_id=32, time_index=1, start=(3756,), count=(1,)), AdiosBlockinfo (process_id=33, time_index=1, start=(3757,), count=(0,)), AdiosBlockinfo (process_id=34, time_index=1, start=(3757,), count=(2,)), AdiosBlockinfo (process_id=35, time_index=1, start=(3759,), count=(1,)), AdiosBlockinfo (process_id=36, time_index=1, start=(3760,), count=(2,)), AdiosBlockinfo (process_id=37, time_index=1, start=(3762,), count=(6,)), AdiosBlockinfo (process_id=38, time_index=1, start=(3768,), count=(3,)), AdiosBlockinfo (process_id=39, time_index=1, start=(3771,), count=(3,)), AdiosBlockinfo (process_id=40, time_index=1, start=(3774,), count=(3,)), AdiosBlockinfo (process_id=41, time_index=1, start=(3777,), count=(0,)), AdiosBlockinfo (process_id=42, time_index=1, start=(3777,), count=(7,)), AdiosBlockinfo (process_id=43, time_index=1, start=(3784,), count=(3,)), AdiosBlockinfo (process_id=44, time_index=1, start=(3787,), count=(0,)), AdiosBlockinfo (process_id=45, time_index=1, start=(3787,), count=(10,)), AdiosBlockinfo (process_id=46, time_index=1, start=(3797,), count=(15,)), AdiosBlockinfo (process_id=47, time_index=1, start=(3812,), count=(0,)), AdiosBlockinfo (process_id=48, time_index=1, start=(3812,), count=(7,)), AdiosBlockinfo (process_id=49, time_index=1, start=(3819,), count=(21,)), AdiosBlockinfo (process_id=50, time_index=1, start=(3840,), count=(171,)), AdiosBlockinfo (process_id=51, time_index=1, start=(4011,), count=(7,)), AdiosBlockinfo (process_id=52, time_index=1, start=(4018,), count=(50,)), AdiosBlockinfo (process_id=53, time_index=1, start=(4068,), count=(593,)), AdiosBlockinfo (process_id=54, time_index=1, start=(4661,), count=(34,)), AdiosBlockinfo (process_id=55, time_index=1, start=(4695,), count=(48,)), AdiosBlockinfo (process_id=56, time_index=1, start=(4743,), count=(1264,)), AdiosBlockinfo (process_id=57, time_index=1, start=(6007,), count=(16,)), AdiosBlockinfo (process_id=58, time_index=1, start=(6023,), count=(28,)), AdiosBlockinfo (process_id=59, time_index=1, start=(6051,), count=(1537,))]]
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ornladios/ADIOS/issues/161#issuecomment-365554717, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGMLfVLbt3mvIaawJqHRf7RjwvA9AWuks5tUq-0gaJpZM4SFCKT .
First question: I cannot even write a zero-length block with zlib transformation into the output because the write segfaults. How do you produce the file? Do you turn off zlib for the zero blocks?
Hi @pnorbert,
@psychocoderHPC just found the root of the issue and will provide a fix in a few minutes. Affects about half of the transforms: blosc, zlib, bzip2, lz4.
Writing zero-length blocks with transformations is possible since a long time (I think we fixed that together in 1.10 or so) and is an important use case for unstructured, domain-decomposed data. We were writing with blosc (not zlib) where we skip compression on zero-size input in the write transform. Maybe the zlib transform has a bug if it does not do the same - but I seem to remember it worked in the past for us.
Or maybe it's just a misunderstanding of what we do: our overall variable is not zero-sized, it's just individual process groups that contribute zero in parallel writes.
It's just a missing meta-data check on read that is causing the crash right now: see #162
@pnorbert The zero length write with zlib is fixed in #165
Hey there, I encountered a segmentation fault that occurs when reading a slice from a linear
AdiosVar
array when the slice contains a certain index. The index can change (checked different files). The values can be read on their own but not when accessing more than one element.The ADIOS 1.13.0 I'm using for post-processing has been built with the same flags as the one I used for the creation of the data - just without
mpi
. I built the numpy wrapper from source usingpython setup.py
. I am usingblosc
withzstd
for compression, full parameters:threshold=2048,shuffle=bit,lvl=1,threads=10,compressor=zstd
.This is what I observe:
A run with
gdb
showsWhat could I specifically look into?