search

nf-core / sarek

Analysis pipeline to detect germline or somatic variants (pre-processing, variant calling and annotation) from WGS / targeted sequencing
https://nf-co.re/sarek
MIT License
409 stars 417 forks source link

Non-unique of `sample` values across several patients causes name clashes #1451

Open hsk6328 opened 8 months ago

hsk6328 commented 8 months ago

Description of the bug

Hello, I am very grateful for your development of the Sarek pipeline. This pipeline has been very helpful to me in handling WGS analysis. However, I encountered an error when testing the pipeline with the test dataset. I would like to ask what might have caused this error.

When I provide a pair of normal and tumor data, an error occurs when calling BAM_VARIANT_CALLING_SOMATIC_ALL in the variant_calling step. The error message is as follows:

ERROR ~ Error executing process > 'NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:MPILEUP_NORMAL:CAT_MPILEUP (1)'

Caused by:
  Process `NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:MPILEUP_NORMAL:CAT_MPILEUP` input file name collision -- There are multiple input files for each of the following file names: HCC1395T_vs_HCC1395N.mpileup.gz

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details

And this is the sample.stomatic.csv:

patient,sex,status,sample,lane,fastq_1,fastq_2
HCC1395,XX,0,HCC1395N,1,./SRR7890919.10M_1.fastq.gz,./SRR7890919.10M_2.fastq.gz
HCC1395,XX,1,HCC1395T,1,./SRR7890918.10M_1.fastq.gz,./SRR7890918.10M_2.fastq.gz

This is the configuration file that I set up, with other parameters kept at default values:

params {
    config_profile_name        = 'WES Demo'
    max_cpus   = 8

    input = '/mnt/disk0/01.nf-core-pipelines/demo/sarek_3.4.0/wes.demo/sample.stomatic.csv'

    // Other params
    tools       = 'controlfreec,vep'
    split_fastq = 20000000
    intervals   = '/mnt/disk0/01.nf-core-pipelines/demo/sarek_3.4.0/wes.demo/S07604624_Padded_Agilent_SureSelectXT_allexons_V6_UTR.bed'
    wes         = true
}

Could you please provide valuable suggestions for this runtime error? Thank you very much!

Command used and terminal output

nextflow run ${nfcorePath}/nf-core-sarek_3.4.0/3_4_0 -profile singularity -c wes.conf --outdir ./outdir --genome GATK.GRCh38

Relevant files

nextflow.log

System information

brandon-hastings commented 3 months ago

I am also getting a similar input file name collision error when NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES is called:

Aug-08 16:16:14.925 [Actor Thread 456] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES (2); work-dir=null error [nextflow.exception.ProcessUnrecoverableException]: Process NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES input file name collision -- There are multiple input files for each of the following file names: Tumor1-L3.0010.bam, Tumor1-L3.0001.bam, Tumor1-L3.0007.bam, Tumor1-L3.0008.bam, Tumor1-L3.0004.bam, Tumor1-L3.0012.bam, Tumor1-L3.0003.bam, Tumor1-L3.0009.bam, Tumor1-L3.0005.bam, Tumor1-L3.0011.bam, Tumor1-L3.0002.bam, Tumor1-L3.0006.bam

nextflow.log

hsk6328 commented 3 months ago

谢谢您~我已收到邮件，稍后将会回复您~

asp8200 commented 3 months ago

I am also getting a similar input file name collision error when NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES is called:

Aug-08 16:16:14.925 [Actor Thread 456] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES (2); work-dir=null error [nextflow.exception.ProcessUnrecoverableException]: Process NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES input file name collision -- There are multiple input files for each of the following file names: Tumor1-L3.0010.bam, Tumor1-L3.0001.bam, Tumor1-L3.0007.bam, Tumor1-L3.0008.bam, Tumor1-L3.0004.bam, Tumor1-L3.0012.bam, Tumor1-L3.0003.bam, Tumor1-L3.0009.bam, Tumor1-L3.0005.bam, Tumor1-L3.0011.bam, Tumor1-L3.0002.bam, Tumor1-L3.0006.bam

nextflow.log

@brandon-hastings : Which version of Sarek are you using? Can you reproduce the error with the latest v3.4.3?

brandon-hastings commented 3 months ago

I was using v3.4.2, but I ran the pipeline again with v3.4.3 and received the same error.

hsk6328 commented 3 months ago

谢谢您~我已收到邮件，稍后将会回复您~

brandon-hastings commented 2 months ago

I was able to look into this more for my specific case. The offending files have the same name within different folders in the work directory. An example of the file structure I am seeing would be:

|-work
   |--3a
      |--82845aa9f7c8a2e485c011c0bb4a5d
         |--bwa
         |--0004.Tumor1-L3_1.fastp.fastq.gz
         |--0004.Tumor1-L3_2.fastp.fastq.gz
         |--Tumor1-L3.0004.bam
   |--5e
      |--f98724d17e629dbde43af318021266
         |--bwa
         |--0004.Tumor1-L3_1.fastp.fastq.gz
         |--0004.Tumor1-L3_2.fastp.fastq.gz
         |--Tumor1-L3.0004.bam

Comparing these bam files via cmp or md5 reveals that they are different files, even after converting them to sam format first.

It might be worth noting that the pipeline failed multiple times due to errors pulling singularity images or errors with processes exceeding running time limits and was run again using the -resume flag. This cycle continued about 10 times, so I’m not sure if the duplicate folders are due to an error in caching the pipeline progress and resuming or if this occurred during the original fastq splitting process in FASTP.

FriederikeHanssen commented 2 months ago

thanks for investigating @brandon-hastings . Wuold you be able to try and reproduce this with a completely clean work directory so we can see whether or not the work directory structure comes from cached steps?

brandon-hastings commented 2 months ago

Yes I am running the workflow again now from the beginning with a clean working directory and I can check for the duplicate directory structure if it crashes.

I have previously replicated the behavior where multiple crashes and resumes resulted in the work directory structure I presented above, one in version 3.4.2 and the other in version 3.4.3, both of which were started from the beginning of the Sarek workflow with a clean work directory.

brandon-hastings commented 2 months ago

I found that the error was caused by the naming in my sample sheet, which I have included a minimum example of as a txt file. I had unique patient IDs, but was reusing naming for sample IDs across patients which I believe led to the file naming error I saw during the FASTP split because the bam file is named using only the sample ID and lane.

samplesheet: samplesheet_example.txt

I managed to solve it by manually adding the patient name to the beginning of each sample ID and restarting the pipeline.

hsk6328 commented 2 months ago

谢谢您~我已收到邮件，稍后将会回复您~

FriederikeHanssen commented 2 months ago

Ah interesting. thanks for investigating @brandon-hastings . I will mark this issue with the label input validation. We should add an additional validation step that makes sure sampleIDs themselves are unique for different patients. They still need to be the same within a single patient to account for multiple lanes

brandon-hastings commented 2 months ago

I just forked it to take a look for myself and I should be able to submit a pull request to include input validation regarding this issue with an updated subworkflow test sometime over the next few days.

Dorothynyamai commented 1 week ago

Hi @brandon-hastings and @FriederikeHanssen Thank you for your insights. I am having the same error even though I have the unique tumor IDs as my patient ID. The difference is that in my case I have the several lanes for the normal samples and this may be the issue. Do you have any suggestion on how I can make changes to my sample sheet to solve this error. Thank you so much

FriederikeHanssen commented 1 week ago

Could you share your samplesheet please?

Dorothynyamai commented 1 week ago

@FriederikeHanssen sure here is a snippet of my samplesheet. testsample_sheet.csv Thank you

FriederikeHanssen commented 1 week ago

the sample columns need to be unique per patient. So you'd want to maybe do something like this: instead of "normal" use "SD1590_normal". I agree that we should try handeling this internally in the future.

Dorothynyamai commented 1 week ago

Hi @FriederikeHanssen should I change this only for the normal samples or I should also do it for the tumor samples? Thank you so much for the quick response.

Dorothynyamai commented 1 week ago

@FriederikeHanssen Thank you so much for the recommendation to revise my samplesheet. I have submitted the job and hopefully it will run fine. I have another sample which has several files with the same lane name for one tumor. However, the flowcell Id is different. Kindly suggest how I can modify the samplesheet to avoid file name collisions. Here is a snippet of the sample sheet. Thank you in advance.SLX-14388_sampleshet_uniqueids.txt

FriederikeHanssen commented 1 week ago

the lane number for the same sample-patient combo must be unique. i.e. you have SD0329,1,SD0329_Tumor,lane_5, this combination several times. If they all belong to the same sample, you need to make the lane column unique, for example:

SD0329,1,SD0329_Tumor,lane_5_1,
SD0329,1,SD0329_Tumor,lane_5_2,

and so on

Dorothynyamai commented 1 week ago

@FriederikeHanssen Thank you so much for the response. I will make these changes. Thanks

Dorothynyamai commented 1 day ago

@FriederikeHanssen Thank you for your suggestions the pipeline is now working Thanks, Dorothy