Add case name to output

[yashpatel6]

Description

Closes #34

Adding case name to output directory to avoid over-writing

Checklist

• [X] This PR does NOT contain PHI or germline genetic data. A repo may need to be deleted if such data is uploaded. Disclosing PHI is a major problem.
• [X] This PR does NOT contain molecular files, compressed files, output files such as images (e.g. .png, .jpeg), .pdf, .RData, .xlsx, .doc, .ppt, or other non-plain-text files. To automatically exclude such files using a .gitignore file, see here for example.
• [X] I have read the code review guidelines and the code review best practice on GitHub check-list.
• [X] I have set up or verified the main branch protection rule following the github standards before opening this pull request.
• [X] The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
• [X] I have added the major changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.

Pipeline Run Results

• Case 1
  • Tested with pipeline-demo-pipeline: /hot/software/pipeline/pipeline-demo-pipeline/Nextflow/development/unreleased/yashpatel-test-case-name

[sorelfitzgibbon]

Currently nftest run for call-sSNV fails with the below error message (ERROR-1). But, if I first manually make the output dir sfitz-nftest-setup, it fails after the first test case finishes with the second error message below. It does make the first case-specific output dir, e.g. sfitz-nftest-setup/a_mini_n2-all-tools-std-input

This may be because I haven't made the correct adjustments to call-sSNV. I tried this change in methods.config, but it didn't help:

164  params.output_dir = "${params.output_dir}/${manifest.name}-${manifest.version}/${params.sample_id}"
165  params.log_output_dir = "${params.output_dir}/log-${manifest.name}-${manifest.version}-${date}"

to

    params.output_dir_base = "${params.output_dir}/${manifest.name}-${manifest.version}/${params.sample_id}"
    params.log_output_dir = "${params.output_dir_base}/log-${manifest.name}-${manifest.version}-${date}"

I maybe was supposed to change e.g. these lines in main.nf, but I've failed to figure out what change to make. Lines 58-64

include { mutect2 } from './module/mutect2' addParams(
    workflow_output_dir: "${params.output_dir}/Mutect2-${params.GATK_version}",
    workflow_log_output_dir: "${params.log_output_dir}/process-log/Mutect2-${params.GATK_version}",
    output_filename: generate_standard_filename("Mutect2_${params.strelka2_version}",
        params.dataset_id,
        params.sample_id,
        [:]))

ERROR-1:

LOG: Loaded .env from /hot/user/sfitzgibbon/git/pipeline-call-sSNV
Traceback (most recent call last):
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/common.py", line 77, in generate_logger
    file_handler = logging.FileHandler(_env.NFT_LOG)
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/logging/__init__.py", line 1169, in __init__
    StreamHandler.__init__(self, self._open())
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/logging/__init__.py", line 1201, in _open
    return open_func(self.baseFilename, self.mode,
FileNotFoundError: [Errno 2] No such file or directory: '/hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-nftest-setup/log-nftest-20230214T231447Z.log'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hot/user/sfitzgibbon/miniconda3/bin/nftest", line 8, in <module>
    sys.exit(main())
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/__main__.py", line 119, in main
    args.func(args)
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/__main__.py", line 68, in run
    setup_loggers()
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/common.py", line 58, in setup_loggers
    _ = generate_logger('NFTest')
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/common.py", line 79, in generate_logger
    raise Exception(f'Unable to create log file: {_env.NFT_LOG}') from file_error
Exception: Unable to create log file: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-nftest-setup/log-nftest-20230214T231447Z.log

ERROR-2:

Traceback (most recent call last):
  File "/hot/user/sfitzgibbon/miniconda3/bin/nftest", line 8, in <module>
    sys.exit(main())
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/__main__.py", line 119, in main
    args.func(args)
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/__main__.py", line 71, in run
    runner.main()
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/NFTestRunner.py", line 51, in main
    case.test()
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/NFTestCase.py", line 61, in wrapper
    func(self)
  File "/hot/user/sfitzgibbon/miniconda3/lib/python3.10/site-packages/nftest/NFTestCase.py", line 78, in test
    for ass in self.asserts:
TypeError: 'NoneType' object is not iterable

[yashpatel6]

I fixed the issue from NFTest's side (the default value when no assertions are made was None when it should've been []).

For the output_dir changes, basically wherever in the pipeline you see params.output_dir being used, you'll instead want to use params.output_dir_base

[sorelfitzgibbon]

Hmm, I'm getting the same error. This is what I did before testing again:

rm -r tool-NF-test
git clone -b yashpatel-add-case-name-to-output git@github.com:uclahs-cds/tool-NF-test.git
cd tool-NF-test/
pip install .

It said it successfully uninstalled and reinstalled despite it being the same version name...

Successfully built nftest
Installing collected packages: nftest
  Attempting uninstall: nftest
    Found existing installation: nftest 1.0.0rc2
    Uninstalling nftest-1.0.0rc2:
      Successfully uninstalled nftest-1.0.0rc2
Successfully installed nftest-1.0.0rc2

[yashpatel6]

@sorelfitzgibbon Anything to add here?

[sorelfitzgibbon]

@sorelfitzgibbon Anything to add here?

Nothing to add, all good. My earlier problem was user error: failing to create the output dir.

[yashpatel6]

Great! Then could you go ahead and review/approve?