Data managers and data tables "best-practice" suggestions

[wm75]

With DMs as a major data source, some degree of standardization between them may be desirable.

[wm75]

A few things we may want/need to decide on:

• standard and non-standard columns data tables are supposed to/allowed to include
• handling of data/index versions; when a new version of a tool requires a new index format, should we move to a new data table everytime or should we use a version column to store different index formats in one and the same data table?
• since it's typically not clear whether future versions of a tool will introduce new index formats, should a version column be mandatory for new DMs/datatables? examples where I recently introduced a version column in a data table for tools are gemini and rnastar; the tools themselves define a minimal index version they are compatible with and filter the data table records using this version
• how much work should a DM do in its wrapper vs in its code, i.e. .py, file? currently some DMs use the wrapper only to gather user input, then do all the actual work of downloading/building reference data in Python. Others include the download/build step in the wrapper and use the code file only to write the json output. At best, that makes it unnecessary hard to read and understand data managers

[dfornika]

I've been thinking about some of these issues while working on the kraken2 data manager and also a data manager that I'm currently working on for mob_suite

A few ideas:

• The current minimal set of 'standard' columns in a data table seem to be value, name, and path.
• Since the contents of the value field must be unique but there are not many other guidelines for how those values should be formatted, it might be a good place to use a UUID. If a UUID(1) were used then it would incorporate timestamp information as well.
• I'm experimenting with using bagit on the contents of the data/index (inspired by the CWLProv project). If that were used routinely, would it provide useful provenance information that could be captured by workflows? (see also https://github.com/galaxyproject/galaxy/issues/7949 and https://github.com/galaxyproject/idc/issues/5)