Define use cases & metrics

[voges]

Use cases:

• Storage (long-term)
• Storage (short-term)
• Streaming
• Random access
  • By class
  • By genomic range

Metrics:

• Compression ratio
• Compression speed
• Decompression speed
• Compression memory usage
• Decompression memory usage
• Parallelization scaling

[voges]

For every use case, define a number of test cases. Each test case is coupled to specific data. Check whether that data is available in the MPEG-G Genomic Information Database, or whether it needs to be generated.

[voges]

Preprocessing for aligned data

For each SAM file:

1. Clean SAM file: remove data that is in scope of ISO/IEC 23092-3, i.e., data that is out of scope of ISO/IEC 23092-2. (A script needs to be written for this task.)
2. Categorize into i) files with multiple alignments and ii) files without multiple alignments
3. Sort by (reference sequence and) mapping position

[PaoloRibeca]

Use cases

• Short-term storage, streaming: Applies to unaligned (FASTQ). The same as Illumina's compression to move FASTQ data from the sequencer to the platform performing alignment/downstream analysis (usually done with gzip). Important: trade-offs between compression speed and compression ratio (might be limited by I/O). Test items: any sequencing file should do
• Long-term storage: Applies to both unaligned (FASTQ) or aligned (SAM) data. Archival, with or without searching capabilities (typically sequence, genomic range, MPEG-G classes). Important: trade-offs between compression ratio and searchability. Covid-19 data would be a good example of classes (both high-throughput sequencing and consensus data are aligned against the Wuhan strain). Anchors: either 1 BAM with each sequenced virus as a different read ID, or 1 BAM file for each sequenced virus (that would probably improve searchability). Tests: search by sequence and by genomic range (spike, furin cleavage site, other hypervariable positions), reads containing variants (PAOLO to update link). Perhaps something in human genetics (exome sequencing)
• When to use unaligned reads? For instance: Want to assemble viral or bacterial contaminants. First map to host (human) and then one would be interested in the reads that do not map. Could be done with Covid data too, by mapping to human - or metagenomics (environmental samples)
• When to use classes? Variant calling - one would only consider reads that map with errors. Random access by genomic range would also be required (calling of small-scale variants is usually done by genomic regions). The number of reads mapping exactly might still be needed. Qualities at the location of the putative variant will typically be needed. Procedure: map to e.g. human, split genome into chunks. For each chunk: retrieve reads which do not map exactly, and their associated qualities.