Design ZIP and/or streaming data servers and clients

[tomLandry]

Following AWG discussion here, now that several files are expected for large subsetted dowloads, the group needs to determine the way to serve and consume metalinks: https://ccdpwiki.atlassian.net/wiki/spaces/CCDP/pages/64520247/Architecture?focusedCommentId=64618644#comment-64618644

This issue is linked to https://github.com/CanadianClimateDataPortal/Canadian-Climate-Data-Portal/issues/56 as we need the UI/UX perspective along the way.

[tomLandry]

We report advancement of metalink feature here, and collaboration of DKRZ: https://github.com/geopython/pywps/issues/298#issuecomment-472751682 Note that priority of this task is still medium, but will become blocker on April 1st. So expect to go back furiously to drawing boards.

[tomLandry]

Hello @huard ! Could you take 5 min to describe recent progress with Metalinks and ZIP support, and how it relates to this issue? Thanks.

[huard]

We have a couple of branches and PR that strive to improve the user experience with respect to WPS outputs storing multiple files under the same identifier. For example, if you are computing an indicator on an ensemble of climate simulations, the output is going to be a list of files. At the moment, the only way to send that back to the user is to either zip these files and provide the reference to the zip file, or to create a txt file storing the list of references.

On the PyWPS side:

• a PR was accepted to add the ZIP format to the list of recognized formats. This is mostly for developer's convenience, it does not change the server's behavior or the user experience.
• a branch is in development that adds MetaFile and MetaLink classes. A process developer would declare an output as a FORMATS.METALINK, instantiate a MetaLink object and fill it with all the output files and their metadata (name, description). The server returns an XML file matching the metalink schema. This works well for metalink version 3, but I'm having issues validating the schema for metalink version 4. Help from XML gurus would be appreciated, because I'm stuck.

On the birdy side

• zip files are now recognized and individual files inside the archive are uncompressed and converted into objects.
• metalink files are now recognized and individual files downloaded and converted into objects. This uses the pymetalink library, which does seem to be actively developed.
• all this is is made possible by a refactoring using both the mime type and the file extension to match a given file with an adequate converter, allowing "nested" conversion.

The metalink format opens up a couple of nice possibilities, including distributing files as torrents, and including a checksum. I suggest looking into the torrent option as a way to distribute files to users.