FASTQ datatype enhancements

[bgruening]

As discussed with @frederikcoppens it would be nice to upload compressed FASTQ-files and to handle them properly in tools.

One way to do this is to enhance the already existing FASTQ datatype with a new metadata element that indicates if it compressed or not. Tools should then be able to recognize this metadata element. GSNAP is one of the tools which would benefit from it.

You can learn about:

• datatypes
• metadata
• tools & metadata

Skills:

• python

[bwlang]

great idea - could be a big speed improvement for slow IO environments

[timothom]

So are you talking about uploading and storeing/manageing the new uploaded data as compressed data? Or just uploading a compressed fastq file and then extacting it once the upload is complete server-side?

Or do you want the data to remain compressed in storage on the server?

[apetkau]

This is a good idea. @timothom I think Galaxy already decompresses fastq files that are uploaded, so I assume this would involve storing as compressed data? Would this involve disabling the decompression of uploaded fastq files?

It would be nice for us in our lab to be able to operate directly on compressed fastq files though.

[dpryan79]

I'd love if even linking in compressed files worked seemlessly. That wouldn't require mucking with the upload tool to not autodecompress stuff.

[abretaud]

This would make very happy several french galaxy admins! We have a kind of patch/hack that works on our instances, but I'm sure there would be a more elegant way to do it: https://www.e-biogenouest.org/wiki/ManArchiveGalaxy

[mvdbeek]

There has also been some work by @yhoogstrate in https://github.com/galaxyproject/galaxy/pull/2535 with a different approach.

[pvanheus]

@mvdbeek: @ashvark and I reviewed @yhoogstrate's work and it requires this squashfs thing to be installed all over the cluster, no?

[pvanheus]

@ashvark has started some work on a compressed Fastq type, see https://github.com/bxlab/galaxy_hackathon/issues/38. @bgruening: how would this work with tools that do not support compressed fastq? And how would compressing existing datasets work - would set_meta() compress / decompress if that key changed?

Finally, see the issue @frederikcoppens mentioned on https://github.com/bxlab/galaxy_hackathon/issues/38 - something to look out for.

[frederikcoppens]

@pvanheus With a new compressed fastq datatype, this would require updating the wrappers to also allow this datatype I assume? Then tools that do not support it require a conversion to use it as input. Would adding a "convert" tool to uncompress (and compress) be an option?

[ashvark]

Yes. I am planning to trying to add converters but i am afraid that would not be good idea for larger fastq files

[bgruening]

Why a new format, just annotate the old format and convert tools that do not support compressed fastq to react on the metadata. This should be compatible and doable without much effort. I'm assuming here that most of the tools already have native support for gzipped fastq.

[pvanheus]

@bgruening because metadata is per-user not per-dataset. However, how about we make a new type: uncompressed fastq. So Fastq is compressed fastq. I'm just thinking of a way to convert existing datasets... @natefoo also pointed out to me that the correct way to handle tools that depend on .gz extension is that at job run time the dataset is linked in with the extension as per datatypes_conf.xml.

[ashvark]

@bgruening and @pvanheus . I have created a separate branch (https://github.com/ashvark/galaxy/tree/fastq_enhancements) in my repository for the enhancement of fastq datatype to handle gzipped fastq files as such. I have tested this only with simpe testcases. Below is the explanation of the changes

• added metadata element 'is_gzipped' for the Fastq datatype in the file datatypes/Sequence.py
• modified get_headers() method in datatypes/sniff.py to handle zipped file.
• added a condition in upload.py to avoid the decompression of gzipped fastq files during upload

TO DO

• test with various scenarios so that it does not disturb any other functionalities

I would like to know your suggestions and improvements.

[yhoogstrate]

+ref: https://github.com/galaxyproject/tools-iuc/pull/354

[zipho]

@yhoogstrate that pull request remains open and seems no further development has been done against it.

Another discussing is here: https://github.com/bxlab/galaxy_hackathon/issues/38

Perhaps we should a combined efforts around this.

@ashvark I briefly tested your changes locally and worked ok.

The other issue is file/dataset extension that sometimes tools use to determine the format of the file, is there any reasons why Galaxy forces the .dat extension. I know it will be a big change, but can files be stored and tracked in their original extension in Galaxy?