PyData prototype IO

[eric-czech]

This issue will track details related to selection and integration of IO libraries for genetic file formats.

For a discussion on libraries in the Python ecosystem for this, see Python IO libraries.

Any solutions here should also consider what it means to support reads of file collections rather than individual files. This is likely not always as simple as concatenating results from a bunch of individual reads (e.g. what do we do if separate VCF files have different numbers of samples/columns?), so our framework should reflect this. This is analogous to how imread_collection is part of the scikit-image plugin interface.

Rechunking may be an additional concern if files are, for example, separated by chromosome and then stacked into a single dask array.

[eric-czech]

A consideration for how these should be integrated is how extensible any plugin framework for them is. Here are a few python examples that span a spectrum of possibilities:

1. Pandas (simplest plugin system)
   • IO functions like read_parquet (code), read_html, and read_excel all take "engine" or "flavor" arguments that result in use of a different backend package
   • Each type of file read has an abstract class from which base classes are maintained by Pandas committers for each backend
   • Note: see here for a great example on how to import modules specific to a backend only when needed
   • Global options like io.parquet.engine make it possible to set defaults
   • This isn't really a "plugin" framework, but using importlib to load the module and work with a reference to it makes it similar to one
2. Scikit-image (somewhat extensible plugins)
   • Plugin docs
   • Plugins allow for overriding of imread, imsave, imshow, etc behavior with a plugin argument on each of those.
   • They are represented as a single module file and an accompanying .ini file
   • The .ini file describes what the plugin supports so this much can be known without needing to import the module
   • There isn't a framework used for the plugins, it's simply a convention for skimage committers to follow when adding new ones
   • The .ini descriptor files DO NOT actually maintain what dependencies the plugin has (so it kind of defeats the purpose of a more formal plugin framework IMO)
   • We may be able to use something like yapsy to avoid re-inventing this wheel
3. Flask (very extensible plugins)
   • Flask allows for integration of CLI extensions through the setuptools entry_point system
   • For example, Flask specifically uses the entry point flask.commands as a way for someone to build a PyPI package that is compatible with Flask's dependencies and then when both are installed, the plugin functionality is automatically available in Flask

My take on this is that the lack of making plugin dependencies a managed part of the framework in #2 (i.e. what scikit-image does) makes the extra abstraction mostly unnecessary. Realistically, I think the uses of some IO plugin will result from either:

• We as maintainers find a new library and want to integrate it
• A user has some old VCF file or uncommon format they know they have a python script for reading, and they simply want to hack in support by our library for themselves

I highly doubt anyone is going to ever publish nice PyPI packages that operate as IO plugins to our project (presumably skallel eventually), so I'm favoring the Pandas model at the moment. This seems like the best way to support a few different backends for the same things and it could easily be customized by a user in the second situation above by us allowing them to pass engine arguments equal to base class instances, rather than a string name, that conform to our protocol/interface in an abstract class (or some other runtime-only hook).

@hammer I'm sure you've seen this play out a ton of times -- got any words of wisdom/caution?

[eric-czech]

Note: Jeff mentioned that the new Spark IO extension framework may make for good inspiration:

• https://www.slideshare.net/databricks/improving-apache-sparks-reliability-with-datasourcev2
• https://jira.apache.org/jira/browse/SPARK-22386

[eric-czech]

On reading file collections:

• A bad approach is: ds = reduce(xr.merge, [load_vcf_dataset(f) for f in glob('dir/*.vcf')])
  • This is not efficient because the merge command looks for conflicting values in arrays where the indexes are the same
  • This can be ignored with compat='override' but then only a single data var is used
  • This is very generalizable though since it would cover reading files that potentially contain information for the same variants/samples
• Using xr.concat is better, specifically via open_mfdataset
  • This will require the user to say whether files are split along variant or sample dimensions
  • It allows arbitrary preprocessing callable
  • It calls combine_by_coords or combine_nested, which uses concat
  • To combine split files along both variant and sample dimensions, it looks like we would have to call combine_nested ourselves which supports ordered concatenation along one axis first and then across a different axis using all the results of the initial "nested" concatenation

[hammer]

Note: Jeff mentioned that the new Spark IO extension framework may make for good inspiration

The reason I think this framework may make for good inspiration is because the first version of the extension framework did not allow extensions to pass enough metadata to Spark to allow for query plan optimization.

One other point we discussed in Slack is that not installing IO extensions is a good idea because many of them will make use of compression and/or encryption native libraries that create complexity for installation. However, it's critical to design a pleasant UX for identifying and downloading extensions so that the user is not confused when they are not able to work with a file format they perceive to be standard.

[hammer]

Other sources of inspiration

• Astropy's I/O Registry
• tabulator-py's Custom file sources and formats
• PostgreSQL foreign data wrappers. Cf. Chapter 56. Writing a Foreign Data Wrapper.
• https://github.com/tidyverse/readr and https://github.com/tidyverse/haven.

[eric-czech]

Here is a notebook that sketches a few ideas for this: I/O Notebook.

Some thoughts here:

• I think it will be useful to expose contexts like tabulator, pandas, and uarray do for scoping operations that need some kind of pluggable backend
• I think it is more important to scope the configuration for those operations though rather than the implementations used. Tabulator and pandas do that, but uarray does not and the reason I think this is useful is that it still allows our framework to take care of initialization and validation. Things like this include setting 'auto' options that would let us scan packages installed first and set a reasonable default, or to possibly analyze custom extensions by forcing them to return dependent package names so we can check that minimal versions are installed.
• The registration mechanism there would work more or less like it does in Astropy (i.e. everything happens at runtime and nothing is read from disk like w/ scikit-image)
• Some potential extensions like those in spark DataSourceV2 to keep in mind would be:
  • Having a Backend be responsible for surfacing row/column statistics to inform I/O operations on collections of files
  • Allowing backends to pass arguments to functions like da.from_array, or the BedArray class I wrote that from_array is calling with index slices to read, to carry out some kind of predicate pushdown. That would then have to pass down to the underlying storage engines (e.g. zarr or pysnptools in this case) to be useful, but if Dask had hypothetical features like this than presumably it would be things like my BedArray class that actually have to worry about them rather than something at the skallel API level. In other words, I think optimizations like those in DataSourceV2 will be a concern for custom implementations provided to the array backend used by I/O backends (so they shouldn't be a part of our I/O API).

[hammer]

To test our IO libraries, it would be useful to gather a library of small, ugly files to parse, for example https://discuss.hail.is/t/import-vcf-from-an-ugly-file-format-issues/1404.