Reading AMReX data with custom DistributionMapping

[tesari]

Hi,

Following the I/O manual distribution mapping is based on the box arrays:

// create a distribution mapping DistributionMapping dm { ba, ParallelDescriptor::NProcs() };

Can I define a distribution mapping based not on BoxArray but on a group of ranks, e.g., excluding rank 0 so that only ranks 1 to NProc-1 read the FABs?. Is there a test or example I could use as a template?

Thanks, Ted

[WeiqunZhang]

There are many possibilities. For example,

Vector<int> pmap;
for (int i = 0; i < ba.size(); ++i) {
    pmap.push_back(i%(ParallelDescriptor::NProcs()-1) + 1);
}
DistributionMapping dm(std::move(pmap));

or

Vector<int> pmap(ba.size(), 1); // Put everything on process 1. Likely a bad idea. But just want to show you can do that.
DistributionMapping dm(std::move(pmap);

or

// Space filling curve chopped into nprocs-1 pieces. The size of the outer vector is nprocs-1.
std::vector<std::vector<int>> sfc = DistributionMapping::makeSFC(ba, true, ParallelDescriptor::NProcs()-1);
Vector<int> pmap(ba.size());
for (int iproc = 0; iproc < sfc.size(); ++iproc) {
    for (auto ibox : sfc[iproc]) {
        pmap[ibox] = iproc+1;
    }
}
DistributionMapping dm{std::move(pmap)};

[tesari]

The space filling curve snippet is what I was looking for. Thanks a lot,

Ted

From: Weiqun Zhang @.> Date: Monday, May 8, 2023 at 10:06 AM To: AMReX-Codes/amrex @.> Cc: Sariyski, Tzvetan Emilov @.>, Author @.> Subject: [EXTERNAL] Re: [AMReX-Codes/amrex] Reading AMReX data with custom DistributionMapping (Issue #3302)

There are many possibilities. For example,

Vector pmap;

for (int i = 0; i < ba.size(); ++i) {

pmap.push_back(i%(ParallelDescriptor::NProcs()-1) + 1);

}

DistributionMapping dm(std::move(pmap));

or

Vector pmap(ba.size(), 1); // Put everything on process 1. Likely a bad idea. But just want to show you can do that.

DistributionMapping dm(std::move(pmap);

or

// Space filling curve chopped into nprocs-1 pieces. The size of the outer vector is nprocs-1.

std::vector<std::vector> sfc = DistributionMapping::makeSFC(ba, true, ParallelDescriptor::NProcs()-1);

Vector pmap(ba.size());

for (int iproc = 0; iproc < sfc.size(); ++iproc) {

for (auto ibox : sfc[iproc]) {

    pmap[ibox] = iproc+1;

}

}

DistributionMapping dm{std::move(pmap)};

— Reply to this email directly, view it on GitHubhttps://github.com/AMReX-Codes/amrex/issues/3302#issuecomment-1538657843, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUR7V3PXIY5FB4JRLJIODDDXFEKYNANCNFSM6AAAAAAX2DNQAY. You are receiving this because you authored the thread.Message ID: @.***>

[tesari]

Hi,

Before I proceed with coding I’d like to make sure that I am not missing something and that there are not better ways or shortcuts. I am reading AMReX data from plot files, and there is no BoxArray data in the Header file, so I will construct the box array from data headers. Here is my plan of work:

1) call amrex::VisMF::ReadFAHeader. 2) Parse the box array data. 3) Construct a vector of box arrays for all leves, ba. 4) Use the SFC snippet as you suggested.

std::vector<std::vector> sfc = DistributionMapping::makeSFC(ba, true, ParallelDescriptor::NProcs()-1); Vector pmap(ba.size()); for (int iproc = 0; iproc < sfc.size(); ++iproc) { for (auto ibox : sfc[iproc]) { pmap[ibox] = iproc+1; } } DistributionMapping dm{std::move(pmap)};

5) Call amrex::VisMF::Read(levels[i].fab, leves[i].path, nullptr).

I wonder how Read knows to use the newly generated distribution mapping and not the on-disk distribution mapping?

Do I miss something? Is there a better way to read a plt data with distribution mapping different from the one used when data was written?

Thanks, Ted

[drummerdoc]

By default, AMReX will generate a BoxArray upon reading the info, based on the current set of procs. The distribution mapping in place when the data was written is not documented and has no bearing on the read.

[WeiqunZhang]

Re: VisMF::Read

    /**                                                                                                                                                                      
    * \brief Read a FabArray<FArrayBox> from disk written using                                                                                                              
    * VisMF::Write().  If the FabArray<FArrayBox> fafab has been                                                                                                             
    * fully defined, the BoxArray on the disk must match the BoxArray                                                                                                        
    * in fafab.  If it is constructed with the default constructor,                                                                                                          
    * the BoxArray on the disk will be used and a new                                                                                                                        
    * DistributionMapping will be made.  A pre-read FabArray header                                                                                                          
    * can be passed in to avoid a read and broadcast.                                                                                                                        
    */
    static void Read (FabArray<FArrayBox> &mf,
                      const std::string &name,
                      const char *faHeader = nullptr,
                      int coordinatorProc = ParallelDescriptor::IOProcessorNumber(),
                      int allow_empty_mf = 0);

DistributionMappings are not stored in plotfiles. If Read gets a FabArray that has been defined with BoxArray and DistributionMapping, it will use it. If the FabArray has not been defined, it will make a new for you. Does that answer your question?

You might want to use PlotFileDataImpl in AMReX_PlotFileDataImpl.cpp as a template.

[WeiqunZhang]

By default, AMReX will generate a BoxArray upon reading the info, based on the current set of procs. The distribution mapping in place when the data was written is not documented and has no bearing on the read.

A typo in Marc's comment. BoxArray should read DistributionMapping.

[drummerdoc]

By default, AMReX will generate a BoxArray upon reading the info, based on the current set of procs. The distribution mapping in place when the data was written is not documented and has no bearing on the read.

A typo in Marc's comment. BoxArray should read DistributionMapping.

oops! Duh. Yeah, what he said.

[tesari]

Thank you so much for pointing me to PlotFileDataImpl, that's what I need. One related question. When data on the disk contains ghosts, is it possible to strip them when reading? Thanks again, Ted

[WeiqunZhang]

No, you have to strip them after reading. However, plotfiles do not have ghost cells unless the users have their own functions to do it.