yt-project / yt

Main yt repository
http://yt-project.org
Other
468 stars 280 forks source link

Simplify grid frontend io.py #3308

Open matthewturk opened 3 years ago

matthewturk commented 3 years ago

Based on the format of the chombo io.py, it should be straightforward to make all the grid frontends utilize a standard fluid selection routine if a _read_data function is implemented by the frontend. For instance, this is the Chombo frontend's _read_fluid_selection:

    def _read_fluid_selection(self, chunks, selector, fields, size):
        rv = {}
        chunks = list(chunks)
        fields.sort(key=lambda a: self.field_dict[a[1]])
        if isinstance(selector, GridSelector):
            if not (len(chunks) == len(chunks[0].objs) == 1):
                raise RuntimeError
            grid = chunks[0].objs[0]
            for ftype, fname in fields:
                rv[ftype, fname] = self._read_data(grid, fname)
            return rv
        if size is None:
            size = sum(g.count(selector) for chunk in chunks for g in chunk.objs)
        for field in fields:
            ftype, fname = field
            fsize = size
            rv[field] = np.empty(fsize, dtype="float64")
        ng = sum(len(c.objs) for c in chunks)
        mylog.debug(
            "Reading %s cells of %s fields in %s grids",
            size,
            [f2 for f1, f2 in fields],
            ng,
        )

        ind = 0
        for chunk in chunks:
            for g in chunk.objs:
                nd = 0
                for field in fields:
                    ftype, fname = field
                    data = self._read_data(g, fname)
                    nd = g.select(selector, data, rv[field], ind)  # caches
                ind += nd
        return rv

Note that if we get rid of the debug call, etc, it actually is quite terse. It calls out to a _read_data function. This could be the basis for refactoring the grid frontends into a smaller, less-complex footprint.

chummels commented 3 years ago

This would be awesome! Thanks, Matt.

neutrinoceros commented 3 years ago

This looks like a doable first step towards simplifying grid based frontends indeed, I'll add it to the 4.1.0 milestone.

neutrinoceros commented 2 years ago

actually this sounds like a refactor with no repercussions on the user side (which is good), so now I think it doesn't need to be targeted at a specific feature release.