Closed eric-czech closed 3 years ago
A consideration for how these should be integrated is how extensible any plugin framework for them is. Here are a few python examples that span a spectrum of possibilities:
read_parquet
(code), read_html
, and read_excel
all take "engine" or "flavor" arguments that result in use of a different backend packageio.parquet.engine
make it possible to set defaultsimportlib
to load the module and work with a reference to it makes it similar to oneimread
, imsave
, imshow
, etc behavior with a plugin
argument on each of those..ini
file.ini
file describes what the plugin supports so this much can be known without needing to import the module.ini
descriptor files DO NOT actually maintain what dependencies the plugin has (so it kind of defeats the purpose of a more formal plugin framework IMO)flask.commands
as a way for someone to build a PyPI package that is compatible with Flask's dependencies and then when both are installed, the plugin functionality is automatically available in FlaskMy take on this is that the lack of making plugin dependencies a managed part of the framework in #2 (i.e. what scikit-image does) makes the extra abstraction mostly unnecessary. Realistically, I think the uses of some IO plugin will result from either:
I highly doubt anyone is going to ever publish nice PyPI packages that operate as IO plugins to our project (presumably skallel eventually), so I'm favoring the Pandas model at the moment. This seems like the best way to support a few different backends for the same things and it could easily be customized by a user in the second situation above by us allowing them to pass engine
arguments equal to base class instances, rather than a string name, that conform to our protocol/interface in an abstract class (or some other runtime-only hook).
@hammer I'm sure you've seen this play out a ton of times -- got any words of wisdom/caution?
Note: Jeff mentioned that the new Spark IO extension framework may make for good inspiration:
On reading file collections:
ds = reduce(xr.merge, [load_vcf_dataset(f) for f in glob('dir/*.vcf')])
merge
command looks for conflicting values in arrays where the indexes are the same compat='override'
but then only a single data var is usedxr.concat
is better, specifically via open_mfdataset
combine_by_coords
or combine_nested
, which uses concat
Note: Jeff mentioned that the new Spark IO extension framework may make for good inspiration
The reason I think this framework may make for good inspiration is because the first version of the extension framework did not allow extensions to pass enough metadata to Spark to allow for query plan optimization.
One other point we discussed in Slack is that not installing IO extensions is a good idea because many of them will make use of compression and/or encryption native libraries that create complexity for installation. However, it's critical to design a pleasant UX for identifying and downloading extensions so that the user is not confused when they are not able to work with a file format they perceive to be standard.
Other sources of inspiration
Here is a notebook that sketches a few ideas for this: I/O Notebook.
Some thoughts here:
Backend
be responsible for surfacing row/column statistics to inform I/O operations on collections of files da.from_array
, or the BedArray
class I wrote that from_array
is calling with index slices to read, to carry out some kind of predicate pushdown. That would then have to pass down to the underlying storage engines (e.g. zarr or pysnptools in this case) to be useful, but if Dask had hypothetical features like this than presumably it would be things like my BedArray
class that actually have to worry about them rather than something at the skallel API level. In other words, I think optimizations like those in DataSourceV2 will be a concern for custom implementations provided to the array backend used by I/O backends (so they shouldn't be a part of our I/O API).To test our IO libraries, it would be useful to gather a library of small, ugly files to parse, for example https://discuss.hail.is/t/import-vcf-from-an-ugly-file-format-issues/1404.
This issue will track details related to selection and integration of IO libraries for genetic file formats.
For a discussion on libraries in the Python ecosystem for this, see Python IO libraries.
Any solutions here should also consider what it means to support reads of file collections rather than individual files. This is likely not always as simple as concatenating results from a bunch of individual reads (e.g. what do we do if separate VCF files have different numbers of samples/columns?), so our framework should reflect this. This is analogous to how
imread_collection
is part of the scikit-image plugin interface.Rechunking may be an additional concern if files are, for example, separated by chromosome and then stacked into a single dask array.