search

ornladios / ADIOS

The old ADIOS 1.x code repository. Look for ADIOS2 for new repo
https://csmd.ornl.gov/adios
Other
54 stars 40 forks source link

calling adios_set_max_buffer_size with different values leads to different final file sizes ? #136

Closed fouriaux closed 7 years ago

fouriaux commented 7 years ago

Using no-XML API, setting adios_set_max_buffer_size with different values leads to different sizes of final file. This behavior appears when using adios_write_byid on a variable declared several times to access several offsets by same rank. When adios_write_byid is called only one time per variable name declared, we have identical final file size whatever adios_set_max_buffer_size we set.

This was tested on x86 machine with Intel Compiler, using Master branch from June 5, 49b720f76ab9097392fdadfeda0f13067dc9e800

max_buffer_size.zip

pnorbert commented 7 years ago

I cannot recreate the issue on my VM with gnu compiler, with either the master HEAD or 49b720f, with either single process or 4 process runs. How did you run the tests?

fouriaux commented 7 years ago

using srun -n 4 on each executable I got the following output files:

du -h *.dat 64K 10_write_max_buffer_1M.dat 1.0M 10_write_max_buffer_4M.dat 1.0M 1_write_max_buffer_1M.dat 1.0M 1_write_max_buffer_4M.dat

[fouriaux@bbpviz1 build]$ $ADIOS_ROOT/bin/bpls 10_write_max_buffer_1M.dat integer global_size scalar integer batch_size scalar integer offset scalar byte data {40960} [fouriaux@bbpviz1 build]$ $ADIOS_ROOT/bin/bpls 10_write_max_buffer_4M.dat integer global_size scalar integer batch_size scalar integer offset scalar byte data {40960}

pnorbert commented 7 years ago

Can you please attach the 'bpdump' output for each file. I found no difference between the two in my runs.

On Tue, Jul 4, 2017 at 3:23 PM, Jeremy FOURIAUX notifications@github.com wrote:

using srun -n 4 on each executable I got the following output files:

du -h *.dat 64K 10_write_max_buffer_1M.dat 1.0M 10_write_max_buffer_4M.dat 1.0M 1_write_max_buffer_1M.dat 1.0M 1_write_max_buffer_4M.dat

[fouriaux@bbpviz1 build]$ $ADIOS_ROOT/bin/bpls 10_write_max_buffer_1M.dat integer global_size scalar integer batch_size scalar integer offset scalar byte data {40960} [fouriaux@bbpviz1 build]$ $ADIOS_ROOT/bin/bpls 10_write_max_buffer_4M.dat integer global_size scalar integer batch_size scalar integer offset scalar byte data {40960}

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ornladios/ADIOS/issues/136#issuecomment-312937894, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGMLXP6ZZT7hdPSiJd-YcOO4yHi3yW4ks5sKpFPgaJpZM4ONe_I .

fouriaux commented 7 years ago

actually I have recompiled everything, and I cannot reproduce this problem anymore. I will check what difference could have happened between those runs at our system side.

Thanks !