yt-project / libyt

In-situ analysis with yt

https://libyt.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

9 stars 3 forks source link

Support Derived Function and Particle Get Attribute Function Prepare Multiple Data Chunks At A Time #54

Open cindytsai opened 2 years ago

cindytsai commented 2 years ago

Support Derived Function and Get Attribute Function Prepare Multiple Data Chunks At A Time

Problem

Currently, derived_func and derived_func_with_name prepare one data grid at a time.
This may cause poor performance for code heavily use hybrid OpenMP/MPI (ex: gamer) to generate derived field.
This limits a derived function style. One must write derived function in a way libyt likes.
Though this problem isn't actually affecting libyt's overall performance for now.

Solution

libyt

Data Structure

56
[ ] Update data members derived_func_chunk and derived_func_with_name_chunk in yt_field struct.
- void (*derived_func_chunk) ( int list_length, long *list_gid, yt_array *data )
- void (*derived_func_with_name_chunk) ( int list_length, long *list_gid, char *field, yt_array *data )
[ ] Update data member get_attr in yt_particle struct.
- void (*get_attr) ( int list_length, long *list_gid, char *attr_name, yt_array *data_array)

Derived Function/Get Attribute Function C++ Extended Python Method

[ ] Python dictionary structure for wrapping them and storing them inside NumPy Array.

RMA

Since one will only need remote memory access when MPI size > 1. Do we really need to prepare many grid at a time if one already have parallelized in MPI?
Update yt_rma_field.cpp when preparing grid data for remote ranks.
Update init_libyt_module.cpp function libyt_field_get_field_remote so that it asks many grids at a time.

yt

Update io.py, find places to store grids.
Redesign _get_field_from_libyt, _read_chunk_data, and _read_fluid_selection.