File's Metadata

[drtamermansour]

Before we start, let us talk about one experimental design as an example to highlight what metadata we might need to annotate the files of this experiment . We have a sequencing experiment to test RNA expression and DNA mutations in skin cancer.

• The study includes 15 participants. To identify each participant, we need participant ID (case ID). Each participant will be associated with several participant's specific features (e.g. Gender, age at diagnosis, Race, diagnosis of participant, etc)
• Each participant provides 3 samples: One from the tumor, another one from nearby healthy skin and a third one from the same skin locus after 2 years of complete cure. To identify each samples, we need sample ID (Biospecimen ID). Each sample will be associated with several sample's specific features (e.g. site of sampling, age of participant at sampling, sampling technique, etc)
• Each sample will be used for RNA and DNA extractions for RNA expression and DNA mutations analyses respectively. Two technical replicates will be generated for each analyte (Now we have 4 tubes for each sample: two RNA replicates and two DNA replicates). Each replicate will be prepared for two different sequencing technologies: short read and long read sequencing. Now we have 8 tubes in total where each tube represents independent library preparation. To identify each tube, we need tube ID (library ID). Each library will be associated with several library's specific features (e.g. experimental strategy either RNA expression or WGS, replicate ID, read length, stand specification, etc).
• Each library will run on two machines (Pacbio and Illumina). For Illumina, each library will run on two lanes. This way each library will end up producing 3 FASTQ files. Each file will be associated with several file's specific features (platform, platform unit, facility ID, etc).

[drtamermansour]

This experiment will be associated with several types of files:

• Raw data files (FASTQ files): Each file should be representing every unique platform or platform unit (unless the users decided to make wrong decision by pooling them)
• Intermediate files e.g. BAM files. These files may allow proper pooling of several Raw data files while keeping enough metadata e.g. one bam file can pool reads from multiple lanes, multiple sequencing machines, multiple library preps, multiple replicates, or even multiple samples)
• Final output files e.g. VCF files or RNA count tables might have some sort of irreversible aggregation. It is irreversible because it does not have metadata to reverse this aggregation process

[drtamermansour]

Now, When we look at metadata of a file, we need to consider The type of the file. In a perfect world, each raw data file should contain data from one platform or even one platform unit if the platform has multiple units (e.g. Illumina machine has multiple lanes). Intermediate and final output files should annotate every data aggregate. For example, in a BAM file, we should have annotation for every read group (or the Platform unit if available). With VCF or count tables, we should have annotation for every column