search

bendudson / BOUT

NOTE: BOUT++ now at https://github.com/boutproject/BOUT-dev
GNU Lesser General Public License v3.0
24 stars 34 forks source link

Parallel input / output #33

Open bendudson opened 12 years ago

bendudson commented 12 years ago

To scale to more processors and make handling large numbers of output files less painful, we need a parallel I./O system. The ADIOS system from ORNL seems like a good option

http://www.olcf.ornl.gov/center-projects/adios/

xxu commented 12 years ago

Dear Ben,

Yes, it is. I have already written it into our SciDac proposal. If we will get it funded, we will do it.

Xu

Ben Dudson wrote:

To scale to more processors and make handling large numbers of output files less painful, we need a parallel I./O system. The ADIOS system from ORNL seems like a good option

http://www.olcf.ornl.gov/center-projects/adios/

Reply to this email directly or view it on GitHub: https://github.com/bendudson/BOUT/issues/33

seanfarley commented 12 years ago

I must recommend against ADIOS. (You know why they call it ADIOS, right? Because once you use it, you can say "adios" to your files).

ADIOS is just a gloried wrapper of MPI-IO with a bunch of xml files and obfuscation in-between. First of all, look at the requirements:

If you're going to require python, why not ditch xml and go with yaml? Also, have you seen the manual?

http://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.3.1.pdf

It doesn't seem easier to me using this than writing MPI-IO. At least with your own MPI you could output VTK or something else easier that allows viewing (XDMF could work).

Why not just skip this layer of locking ourselves into a whacky framework with ADIOS, and use something much more lightweight (and also scalable): binary-appended data (with perhaps VTK) over MPI-IO? This would require no extra libraries since the format is dead simple.

Plus, it'd be easy to convert the data later if we decide to change the format.

xxu commented 12 years ago

Sean,

Thank you for your comments. If our SciDac project get funded, I will have computer scientists to work those issues out.

Xu

-----Original Message----- From: Sean Farley [mailto:reply@reply.github.com] Sent: Friday, December 16, 2011 3:13 PM To: Xu, Xueqiao Subject: Re: [BOUT] Parallel input / output (#33)

I must recommend against ADIOS. (You know why they call it ADIOS, right? Because once you use it, you can say "adios" to your files).

ADIOS is just a gloried wrapper of MPI-IO with a bunch of xml files and obfuscation in-between. First of all, look at the requirements:

If you're going to require python, why not ditch xml and go with yaml? Also, have you seen the manual?

http://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.3.1.pdf

It doesn't seem easier to me using this than writing MPI-IO. At least with your own MPI you could output VTK or something else easier that allows viewing (XDMF could work).

Why not just skip this layer of locking ourselves into a whacky framework with ADIOS, and use something much more lightweight (and also scalable): binary-appended data (with perhaps VTK) over MPI-IO? This would require no extra libraries since the format is dead simple.

Plus, it'd be easy to convert the data later if we decide to change the format.

Reply to this email directly or view it on GitHub: https://github.com/bendudson/BOUT/issues/33#issuecomment-3184727