BAM files for large genomes and their .csi index

FredericBGA commented 3 years ago

Maybe we've lost the csi index of BAM with Galaxy 20_05 I don't understand, maybe this is a small issue because all the code seems to be here (galaxyproject/galaxy#9570)

When I create a BAM (with BWA for example, on wheat genome, so the BAM needs a .csi) I don't have the link to download the .csi index anymore (it was the case previously). And we use a lot the "display with IGV" feature. And it looks like that the .csi index is missing (IGV is complaining...).

two remarks:

I noticed that pysam has been update from 0.15.4 to 0.16.0.1 and now the code alignment_file.header.lengths (in lib/galaxy/datatypes/binary.py) will display a message [E::idx_find_and_load] Could not retrieve index file for ....
We are working with Libraries. And when I try to send a wheat BAM from the library to my history it fails (It works with a maize BAM). I can not see any error in error log (neither in developer mode of the browser). But the job is in error state An error occurred with this dataset:. I don't see any other messages.

I'm really keen to help, but I need maybe some help to start because right now I don't really understand.

mvdbeek commented 3 years ago

Can you upload a problematic bam file with just a few reads to usegalaxy.org and then publish that history?

FredericBGA commented 3 years ago

I've created this history: BAM

The .bai is the index that I've downloaded from my Galaxy history. The .csi is the index that I've created with samtools 1.9 or 1.11

I observe something I can't understand: the .csi has been created on my server, when I upload it to Galaxy and then launch a diffbetween the two files (the .csi and the .dat that should hold the .csi) it says: Binary files x and Y differ

galaxyproject / galaxy

BAM files for large genomes and their .csi index #10909