Open lindakohn opened 1 year ago
Hi Linda @lindakohn !
Thanks for raising the issue! I think the issue stems from the major upgrade in gatk and picard which both switched to java17:
While other packages in java env in bcbio are not compatible with newer java: https://github.com/chapmanb/cloudbiolinux/blob/master/contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml#L292
I am testing a solution. I will likely be a separate java17 environment for gatk/picard and java8 for other packages.
SN
Hi Linda @lindakohn !
I've made a fix and ran a test installation - picard and gatk work! For now we are just pinning pre-java17 gatk4 and picard version, since there is no gatk4.4.0/java17 in conda yet. https://github.com/chapmanb/cloudbiolinux/blob/master/contrib/flavor/ngs_pipeline_minimal/packages-conda.yaml#L277
Once it is published, we will switch to java17 and unpin picard and gatk4, placing gatk and other legacy tools in java8 env.
SN
Thank you!
I ran
bcbio_nextgen.py upgrade -u development --tools
and saw that gatk ang picard were downgraded to 4.2.5.0 and 2.27.5, respectively. However, I now encountered another error:
Traceback (most recent call last):
File "/home/bcbio/bcb/tools/bin/bcbio_nextgen.py", line 245, in <module>
main(**kwargs)
File "/home/bcbio/bcb/tools/bin/bcbio_nextgen.py", line 46, in main
run_main(**kwargs)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/pipeline/main.py", line 50, in run_main
fc_dir, run_info_yaml)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/pipeline/main.py", line 91, in _run_toplevel
for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/pipeline/main.py", line 194, in variant2pipeline
samples = qcsummary.generate_parallel(samples, run_parallel)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/pipeline/qcsummary.py", line 43, in generate_parallel
qced = run_parallel("pipeline_summary", to_analyze)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
return run_multicore(fn, items, config, parallel=parallel)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(*x) for x in items):
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/joblib/parallel.py", line 1051, in __call__
while self.dispatch_one_batch(iterator):
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/joblib/parallel.py", line 866, in dispatch_one_batch
self._dispatch(tasks)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/joblib/parallel.py", line 784, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
result = ImmediateResult(func)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 572, in __init__
self.results = batch()
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/joblib/parallel.py", line 263, in __call__
for func, args, kwargs in self.items]
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/joblib/parallel.py", line 263, in <listcomp>
for func, args, kwargs in self.items]
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/utils.py", line 59, in wrapper
return f(*args, **kwargs)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/distributed/multitasks.py", line 243, in pipeline_summary
return qcsummary.pipeline_summary(*args)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/pipeline/qcsummary.py", line 79, in pipeline_summary
data["summary"] = _run_qc_tools(work_bam, work_data)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/pipeline/qcsummary.py", line 177, in _run_qc_tools
out = qc_fn(bam_file, data, cur_qc_dir)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/qc/fastqc.py", line 52, in run
do.run(cl, "FastQC: %s" % dd.get_sample_name(data))
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/provenance/do.py", line 26, in run
_do_run(cmd, checks, log_stdout, env=env)
File "/home/bcbio/bcb/anaconda/lib/python3.7/site-packages/bcbio/provenance/do.py", line 106, in _do_run
raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; export LD_PRELOAD=/home/bcbio/bcb/anaconda/lib/libfreetype.so && export PATH=/home/bcbio/bcb/anaconda/bin:"$PATH" && /home/bcbio/bcb/anaconda/bin/fastqc -d /home/bcbio/data/test_230301/calling/work/qc/15-5935D/bcbiotx/tmpkkh6gva8 -t 16 --extract -o /home/bcbio/data/test_230301/calling/work/qc/15-5935D/bcbiotx/tmpkkh6gva8 -f bam /home/bcbio/data/test_230301/calling/work/align/15-5935D/15-5935D-sort.bam
Option d is ambiguous (delete, dir, dup_length)
Failed to process /home/bcbio/data/test_230301/calling/work/qc/15-5935D/bcbiotx/tmpkkh6gva8
java.io.FileNotFoundException: /home/bcbio/data/test_230301/calling/work/qc/15-5935D/bcbiotx/tmpkkh6gva8 (Is a directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at uk.ac.babraham.FastQC.Sequence.BAMFile.<init>(BAMFile.java:62)
at uk.ac.babraham.FastQC.Sequence.SequenceFactory.getSequenceFile(SequenceFactory.java:81)
at uk.ac.babraham.FastQC.Sequence.SequenceFactory.getSequenceFile(SequenceFactory.java:62)
at uk.ac.babraham.FastQC.Analysis.OfflineRunner.processFile(OfflineRunner.java:163)
at uk.ac.babraham.FastQC.Analysis.OfflineRunner.<init>(OfflineRunner.java:125)
at uk.ac.babraham.FastQC.FastQCApplication.main(FastQCApplication.java:316)
Started analysis of 15-5935D-sort.bam
Approx 5% complete for 15-5935D-sort.bam
Approx 10% complete for 15-5935D-sort.bam
Approx 15% complete for 15-5935D-sort.bam
Approx 20% complete for 15-5935D-sort.bam
Approx 25% complete for 15-5935D-sort.bam
Approx 30% complete for 15-5935D-sort.bam
Approx 35% complete for 15-5935D-sort.bam
Approx 40% complete for 15-5935D-sort.bam
Approx 45% complete for 15-5935D-sort.bam
Approx 50% complete for 15-5935D-sort.bam
Approx 55% complete for 15-5935D-sort.bam
Approx 60% complete for 15-5935D-sort.bam
Approx 65% complete for 15-5935D-sort.bam
Approx 70% complete for 15-5935D-sort.bam
Approx 75% complete for 15-5935D-sort.bam
Approx 80% complete for 15-5935D-sort.bam
Approx 85% complete for 15-5935D-sort.bam
Approx 90% complete for 15-5935D-sort.bam
Approx 95% complete for 15-5935D-sort.bam
Approx 100% complete for 15-5935D-sort.bam
Analysis complete for 15-5935D-sort.bam
' returned non-zero exit status 1.
Something seems to have changed for fastqc?
Yes, it looks like we received a major update of fastqc and we should adapt to it: https://github.com/s-andrews/FastQC/releases/tag/v0.12.0
Pinned to the previous version in the meanwhile - no error, please upgrade tools and it should downgrade for you as well.
Perfect, thank you, works now!
Hi!
Thank you for maintaining this great workflow!
I've run into java problems after a fresh install of v1.2.9, see traceback below
Checking the java versions yields:
So I tried setting an environment variable to get bcbio to use java version 17 since it's asking for Java Runtime (class file version 61.0):
Bcbio then continues to run but further along another java error is thrown:
Unsetting the environment variable at this point solves the issue
and bcbio finishes successfully.
Please suggest on how to solve this
Kind regards Linda