Composite data type for ISA-Tab + plus data in the context of metabolomics (mass spec and NMR)

[pcm32]

At the PhenoMeNal project we are working towards making large scale computation for metabolomics possible on cloud providers. We are working to make use of Galaxy at the user interface layer, on top of our stack. In this context, we are in need of a proper way of accessing ISA-Tab metadata files and accompanying data from Galaxy. We are hoping to contribute an ISA-Tab composite data type in the context of metabolomics (so, the composite would manage files like mzML, nmrML, etc.). Our current approach is just to hide all of these from Galaxy by putting files inside a tar/zip, this if course far from being the proper solution.

ISA lends itself very well for composite data types, as there is a single initial file (i_investigation file) from which the whole directory structure (s, a, maf and data files) can be explained (including assignment of defined experimental factor values to data files).

Initially our concerns for the use case are as follows:

• [ ] Scalability: Metabolomics studies, specially mass spec ones, can be very large in terms of data size (many mzML files, each a few hundred MBs easily). As such we want a solution that scales well, paritcularly that avoids file duplication/copy. One should be able to download a MetaboLights study into Galaxy (and hopefully analyse it as well) without any file duplication. This is due to some studies reaching the hundreds of GBs in size.
• [ ] Jobs distribution: Ability to distribute jobs that process the main data files (ie. mzMLs) contained in a study. We realise that job distribution is something done over collections rather than inside composite data types, but we wonder if with some guidance we could enable such a feature with some changes in the core galaxy code. Maybe you can have collections within composites?
• [ ] Recognized by core metabolomics tools: we would aim to provide integration with some of the metabolomics tools already ported to Galaxy that need to process the main data files within this new composite.

Formats that we should support initially:

• [ ] mzML
• [ ] nmrML
• [ ] ISA files
• [ ] MAF file (maybe this could be done in an inherited version of the composite)

We have found so far some interesting examples, like this in proteomics, besides the commonly cited one for genetics. Any other good example that we should be aware of?

At this point we are after feedback from the Galaxy community and core developers on how to build this in the best possible way (most compliant with what Galaxy expects). I'm tagging people as requested: @dannon. More collaborators will add more comments as I share this link (I don't seem to be able to tag them here, I guess you can only tag project members).

[pkrog]

Hi @bgruening , I don't know if you had time to look at my hack. Anyway I have another concern. It's about the uploader. In case the uploaded file is not a valid ISA archive I send an exception. However my exception message is not displayed. I've tried also to send messages on stdout and stderr and on the logger on the different channel, but nothing is shown. I've looked at other composite datatypes, but I found none that fails on wrong file submission, all dataset obtained are in green state. Is this the default behaviour to implement? This seems strange to me, but having the dataset in a red state with no clear message to the user is a bit strange too. What would you suggest?