Ask Non-Local Grid From Other MPI Rank

[cindytsai]

Ask Non-Local Grid From Other MPI Rank

• Possible Solution:
  • libyt get non local grids from other rank, and pass it back to yt, just like how derived fields did.
  • Reference
  • Code template

[cindytsai]

Solution

Because window creation in MPI RMA operation is a collective operation, every rank must be in the same state. Which means we have to enter this state at the very beginning of the IO. ( I didn't consider load balancing. But if we really need to, we only need to make sure every rank goes to the same function that will enter the same state.) To do this,

Distinguish local/non-local grids.

See cindytsai/yt branch libyt-NonLocal.

At libyt yt frontend, gather all the non-local grids at each rank's perspective.

See cindytsai/yt branch libyt-NonLocal.

RMA operation in C extended python method.

• [x] Implement all the method in class.
• [x] Some data member may be redundant, remove them.
• [x] Modelize this get remote grid method in libyt frontend io.
• [x] Tuning RMA, set info and assert in RMA.
  • [x] MPI_Win_fence
  • [x] MPI_Win_create_dynamic
• [x] Modelize MPI_Gatherv and MPI_Bcast deal with big send count in yt_commit_grids.cpp.
  • [x] Apply to yt_rma class gather_all_prepare_data.
• [x] Rename yt_rma class to yt_rma_field, and construct yt_rma_particle.
  • [x] Test run on particle data.
  • [x] What if particle count is 0.
  • [x] Test run on field data.
• [x] Log messages.
• [ ] Sync field to get and particle to get before calling C extension(?)
  • [ ] It doesn't seem to need this, they are symmetry.
  • [x] Use sorted when getting fname_list and ptf_c dictionary, so that each fields to-get are unique.
• [x] Do not call back to c extension when there are no remote grids needed to get.
• [x] Filter out those who really has particles in their grid.
• [x] Set log msg in libyt yt frontend if using yt_rma_field or yt_rma_particle.
• [x] Extended C++ method should be able to respond failure return from yt_rma if anything bad really happens.
• [x] Support MPI_Get for big sendcount and receive count.
  • Haven't encounter so not tested yet.

Collect the field and return.

Reference

• BTW, we should avoid creating lots of windows. (https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/node278.htm)
• Tuning RMA. (https://www.mcs.anl.gov/research/projects/mpi/tutorial/advmpi/sc2005-advmpi.pdf)
• Tuning create window. (https://cvw.cac.cornell.edu/mpionesided/lul)

Test Run

• [ ] yt functionality (These functions did not support parallelism before we support getting non-local grids)
  • [x] Off Axis Projection Plot (:tada:)
  • [x] Slice Plot (:tada:)
  • [x] Off Axis Slice Plot
  • [x] Gradient (:tada:)
  • [x] LinePlot (:tada:)
  • [x] Volume Rendering, MPI size even (:tada:)
  • Comments in below.
  • [ ] Particle Plot (:x:)

  • 32

  • [ ] Particle Projection Plot (:x:)

  • 32

• [x] Field Data
  • [x] cell-centered
  • [x] face-centered
  • [x] derived_func
• [x] Particle Data

TODO

• [x] Check lifetime of data members in field_list and particle_list, they should exist till the very end.
  • User input field_list and particle_list should exist till the end of the inline-analysis.
• [x] Reference counting in Python, remember to free the unused reference.
• [x] Check if I freed every allocated data.
• [x] Will there be ranks that have no grids to read.
• [ ] What if particle attributes eat up all the memory. Have not encounter yet.

[hyschive]

@cindytsai I like this approach. We can easily distinguish local and non-local grids when collecting the complete AMR structure.

[cindytsai]

Test Run

Volume Rendering

• Test Problem: 3D Blast Wave.

• Inline script:

  import yt
  yt.enable_parallelism()
  def yt_inline():
      ds = yt.frontends.libyt.libytDataset()
      sc = yt.create_scene(ds, lens_type="perspective")
      source = sc[0]
      source.tfh.set_log(True)
      source.tfh.grey_opacity = False
      source.tfh.plot("transfer_function.png", profile_field=("gas", "density"))
  
      sc.save("rendering.png", sigma_clip=4.0)

  • End with same error as in post-processing.
  • Inline-Analysis
  • Post-Processing
  • yt-project issue A work around method will be set the log level to info. Issue Closed.
  • Run the script while turning debug message off. (They have fixed the issue.)
  • Post-Processing
    • MPI=1,2: Give the correct result.
  • Inline
    • MPI=2: (Ignore the zoom in and the color transfer function.)
    • MPI=3, 5 (odd number): Stuck and failed when saving rendering image.
    • MPI=4: Success.
• Inline script (Failed):

  import yt
  yt.enable_parallelism()
  def yt_inline():
      ds = yt.frontends.libyt.libytDataset()
      sc = yt.create_scene(ds, lens_type="perspective")
      source = sc[0]
      source.tfh.set_log(True)
      source.tfh.grey_opacity = False
      source.tfh.plot("transfer_function.png", profile_field=("gas", "density"))
      if yt.is_root():
          sc.save("rendering.png", sigma_clip=4.0)

  • Not all ranks call for the _read_fluid_selection, because there is a if clause yt.is_root() when saving figure.

[cindytsai]

Test Run

OffAxisProjectionPlot

• Test Problem: 3D Blast Wave.
• Inline script: (Again, this should turn the log level to info.)

  import yt
  yt.enable_parallelism()
  yt.set_log_level("info")
  def yt_inline():
    ds = yt.frontends.libyt.libytDataset()
    L = [1, 1, 0]
    north_vector = [-1, 1, 0]
    prj = yt.OffAxisProjectionPlot(ds, L, ("gas", "density"), north_vector=north_vector)
    if yt.is_root():
        prj.save()

  

[cindytsai]

Test Run

OffAxisSlicePlot

• Test Problem: 3D Blast Wave
• Inline script:

  import yt
  yt.enable_parallelism()  
  def yt_inline():
    ds = yt.frontends.libyt.libytDataset()
    L = [1, 1, 0]
    north_vector = [-1, 1, 0]
    cut = yt.SlicePlot(ds, L, ("gas", "density"), north_vector=north_vector, center=[0.5, 0.5, 0.5])
    if yt.is_root():
        cut.save()

  

[cindytsai]

Test Run

• 32

ParticlePlot

• Test Problem: gamer Plummer
• Inline Script: Both of them failed.

  import yt
  yt.enable_parallelism()
  def yt_inline():
    ds = yt.frontends.libyt.libytDataset()
    par = yt.ParticlePlot( ds, 'particle_position_x', 'particle_position_y', ("io", "particle_mass"), center='c' )
    if yt.is_root():
        par.save()

  import yt
  yt.enable_parallelism()
  def yt_inline():
    ds = yt.frontends.libyt.libytDataset()
    par = yt.ParticlePlot( ds, 'particle_position_x', 'particle_position_y', ("io", "particle_mass"), center='c' )
    par.save()

Output

Expected Output

[cindytsai]

Test Run

• 32

ParticleProjectionPlot

• Test Problem: gamer Plummer
• Inline Script: Both of them failed.

  import yt
  yt.enable_parallelism()
  def yt_inline():
    ds = yt.frontends.libyt.libytDataset()
    par_prj = yt.ParticleProjectionPlot( ds, "z" )
    if yt.is_root():
        par.save()

  import yt
  yt.enable_parallelism()
  def yt_inline():
    ds = yt.frontends.libyt.libytDataset()
    par_prj = yt.ParticleProjectionPlot( ds, "z" )
    par.save()

MPI = 2

MPI = 1 (Expected)