DKFZ-ODCF / AlignmentAndQCWorkflows

The DKFZ alignment workflow plugin originally developed at the eilslabs
https://github.com/DKFZ-ODCF/AlignmentAndQCWorkflows/wiki
Other
7 stars 5 forks source link

Roddy cannot zip folder #91

Open AndreasKienzle opened 5 months ago

AndreasKienzle commented 5 months ago

Hello,

I want to run the AlignmentAndQCWorkflow for my tumor/normal FastQ-files (WES), but I get an error during the testrun. Here a part of my code:

FASTQ_LIST_TUMOR="tumor/tumor_R1.fastq.gz;tumor/tumor_R2.fastq.gz" FASTQ_LIST_NORMAL="normal/normal_R1.fastq.gz;normal/normal_R2.fastq.gz" echo "Running Roddy command for tumor dataset..." bash roddy.sh testrun WES_Vienna@WES tumor \ --useconfig=$main_directory/RoddyProject/Roddy/applicationProperties.ini \ --useRoddyVersion=3.3.1 \ --cvalues="fastq_list=$FASTQ_LIST_TUMOR" \ --verbositylevel=5

These are the XML and applicationProperties.ini files (for uploading converted into txt files): applicationProperties.txt WES_Vienna.txt

In the following is the log-output after running the code:

(DKFZ_env) onko153-115:Roddy gerwin$ cat /Users/gerwin/.roddy/logs/default_240622_203128515.tsv Roddy INFO Roddy version 3.3.1 NoExceptionThrown Roddy INFO Timing initial checks: 89.995715 ms NoExceptionThrown Roddy INFO Timing ftoggleini: 63.123059 ms NoExceptionThrown Roddy INFO Timing initialize mode, print arguments: 63.941647 ms NoExceptionThrown Roddy INFO [0] - testrun NoExceptionThrown Roddy INFO [1] - WES_Vienna@WES NoExceptionThrown Roddy INFO [2] - tumor NoExceptionThrown Roddy INFO [3] - --useconfig=/Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/applicationProperties.ini NoExceptionThrown Roddy INFO [4] - --useRoddyVersion=3.3.1 NoExceptionThrown Roddy INFO [5] - --cvalues=fastq_list=tumor/tumor_R1.fastq.gz;tumor/tumor_R2.fastq.gz NoExceptionThrown Roddy INFO [6] - --verbositylevel=5 NoExceptionThrown Roddy INFO Timing initial: 12.575537 ms NoExceptionThrown Roddy INFO Timing parseopt: 1.645531 ms NoExceptionThrown Roddy INFO Loading properties file /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/applicationProperties.ini. NoExceptionThrown Roddy INFO Timing loadprop: 11.39948 ms NoExceptionThrown Roddy INFO Timing init proxy: 1.949407 ms NoExceptionThrown de.dkfz.roddy.execution.io.fs.FileSystemAccessProvider INFO public static void initializeProvider(boolean fullSetup) NoExceptionThrown Roddy INFO Timing init fsap: 52.11291 ms NoExceptionThrown de.dkfz.roddy.execution.io.ExecutionService INFO public static void initializeService(boolean fullSetup) NoExceptionThrown de.dkfz.roddy.execution.io.ExecutionService INFO Timing initialize exec service: 0.003398 ms NoExceptionThrown Roddy INFO Timing init execserv: 22.0559 ms NoExceptionThrown Roddy INFO public static void initializeFactory(boolean fullSetup) NoExceptionThrown Roddy INFO Roddy will try to use the DirectSynchronousExecutionJobManager job manager class to manage jobs. NoExceptionThrown Roddy INFO Timing init cmd fac: 81.779829 ms NoExceptionThrown Roddy INFO Timing initserv: 26.759641 ms NoExceptionThrown ConfigurationFactory CONFIG Searching for configuration files in: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/configurationfiles NoExceptionThrown ConfigurationFactory CONFIG Searching for configuration files in: /Users/gerwin/.roddy/configurationFreeModeFiles NoExceptionThrown ConfigurationFactory CONFIG Searching for configuration files in: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/configurationfiles NoExceptionThrown ProjectLoader INFO Loading information configuration context for WES_Vienna from /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/configurationfiles/WES_Vienna.xml NoExceptionThrown XSDValidator INFO Will validate configuration WES_Vienna. NoExceptionThrown XSDValidator INFO Skipped configuration with id default, not available. NoExceptionThrown LibrariesFactory INFO Parsing plugins folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/.DS_Store NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/COWorkflowsBasePlugin_1.2.0 NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/DefaultPlugin_1.2.2 NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/NativeWorkflowTestPlugin NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/PluginBase_1.2.0 NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/PluginBase_1.2.1 NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/Template NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/TestPluginWithJarFile NoExceptionThrown LibrariesFactory INFO Parsing plugin folder: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/TestPluginWithoutJarFile NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/COWorkflowsBasePlugin_1.2.0 NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/DefaultPlugin_1.2.2 NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/NativeWorkflowTestPlugin NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/PluginBase_1.2.0 NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/PluginBase_1.2.1 NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/Template NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/TestPluginWithJarFile NoExceptionThrown LibrariesFactory INFO Processing plugin entry: /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/TestPluginWithoutJarFile NoExceptionThrown LibrariesFactory INFO Found tool folders: bisulfiteWorkflow : /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows/resources/analysisTools/bisulfiteWorkflow exomePipeline : /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows/resources/analysisTools/exomePipeline qcPipeline : /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows/resources/analysisTools/qcPipeline qcPipelineTools : /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows/resources/analysisTools/qcPipelineTools roddyNativeTools : /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/DefaultPlugin_1.2.2/resources/analysisTools/roddyNativeTools roddyTools : /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/DefaultPlugin_1.2.2/resources/analysisTools/roddyTools tools : /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/COWorkflowsBasePlugin_1.2.0/resources/analysisTools/tools NoExceptionThrown LibrariesFactory INFO Loaded plugin AlignmentAndQCWorkflows:develop (/Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows) Loaded plugin COWorkflowsBasePlugin:1.2.0-0 (/Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/COWorkflowsBasePlugin_1.2.0) Loaded plugin PluginBase:1.2.1-0 (/Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/PluginBase_1.2.1) Loaded plugin DefaultPlugin:1.2.2-0 (/Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/DefaultPlugin_1.2.2) NoExceptionThrown ConfigurationFactory INFO Load configuration file /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/configurationfiles/WES_Vienna.xml NoExceptionThrown ConfigurationFactory INFO Load configuration file /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/DefaultPlugin_1.2.2/resources/configurationFiles/default.xml NoExceptionThrown ConfigurationFactory INFO Processing tool compressionDetection NoExceptionThrown ConfigurationFactory INFO Processing tool defaultPluginLib NoExceptionThrown ConfigurationFactory INFO Processing tool wrapinScript NoExceptionThrown ConfigurationFactory INFO Load configuration file /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows/resources/configurationFiles/analysisExome.xml NoExceptionThrown ConfigurationFactory INFO Processing tool onTargetCoverage NoExceptionThrown ConfigurationFactory INFO Processing tool onTargetCoveragePlotter NoExceptionThrown ConfigurationFactory INFO Processing tool onTargetCoveragePlotterBinary NoExceptionThrown ConfigurationFactory INFO Processing tool compressionDetection NoExceptionThrown ConfigurationFactory INFO Processing tool targetCoveragePerlScript NoExceptionThrown ConfigurationFactory INFO Processing tool targetExtractCoverageSlim NoExceptionThrown ConfigurationFactory INFO Processing tool annotateCovWindows NoExceptionThrown ConfigurationFactory INFO Processing tool mergeAndFilterCovWindows NoExceptionThrown ConfigurationFactory INFO Processing tool correctGc NoExceptionThrown ConfigurationFactory INFO Load configuration file /Users/gerwin/Andreas/Pipeline3DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows/resources/configurationFiles/analysisQC.xml NoExceptionThrown ConfigurationFactory SEVERE null: <filename pattern='${outputAnalysisBaseDirectory}/${coverageOutputDirectory}/${sourcefileAtomicPrefix,delimiter=""}.DepthOfCoverageGenome.txt' onMethod='BamFile.calcCoverage' class='CoverageTextFile'> NoExceptionThrown ConfigurationFactory SEVERE null: <filename pattern='${outputAnalysisBaseDirectory}/${coverageOutputDirectory}/${sourcefileAtomicPrefix,delimiter=""}.DepthOfCoverage_RawBamFile.txt' onMethod='BamFile.rawBamCoverage' class='CoverageTextFile'> NoExceptionThrown ConfigurationFactory SEVERE null: NoExceptionThrown ConfigurationFactory INFO Processing tool workflowEnvironment_tbiLsf NoExceptionThrown ConfigurationFactory INFO Processing tool workflowEnvironment_tbiPbs NoExceptionThrown ConfigurationFactory INFO Processing tool workflowEnvironment_conda NoExceptionThrown ConfigurationFactory INFO Processing tool bashLib NoExceptionThrown ConfigurationFactory INFO Processing tool workflowLib NoExceptionThrown ConfigurationFactory INFO Processing tool groupedGenomeCoverages NoExceptionThrown ConfigurationFactory INFO Processing tool cleanupScript NoExceptionThrown ConfigurationFactory INFO Processing tool fastqcClassify NoExceptionThrown ConfigurationFactory INFO Processing tool fastqc NoExceptionThrown ConfigurationFactory INFO Processing tool alignAndPairSlim NoExceptionThrown ConfigurationFactory INFO Processing tool accelerated_alignAndPairSlim NoExceptionThrown ConfigurationFactory INFO Processing tool commonAlignmentSettingsScript NoExceptionThrown ConfigurationFactory INFO Processing tool coveragePlotScript NoExceptionThrown ConfigurationFactory INFO Processing tool insertSizePlotScript NoExceptionThrown ConfigurationFactory INFO Processing tool writeQcSummary NoExceptionThrown ConfigurationFactory INFO Processing tool qcJson NoExceptionThrown ConfigurationFactory INFO Processing tool sequencerDetection NoExceptionThrown ConfigurationFactory INFO Processing tool convertIlluminaScores NoExceptionThrown ConfigurationFactory INFO Processing tool genomeCoverage NoExceptionThrown ConfigurationFactory INFO Processing tool plotDiffchrom NoExceptionThrown ConfigurationFactory INFO Processing tool filterReadBins NoExceptionThrown ConfigurationFactory INFO Processing tool coverageQcDImpl NoExceptionThrown ConfigurationFactory INFO Processing tool genomeCoverageDImpl NoExceptionThrown ConfigurationFactory INFO Processing tool collectBamMetrics NoExceptionThrown ConfigurationFactory INFO Processing tool coveragePlot NoExceptionThrown ConfigurationFactory INFO Processing tool coveragePlotSingle NoExceptionThrown ConfigurationFactory INFO Processing tool fingerprint NoExceptionThrown ConfigurationFactory INFO Processing tool mergeAndRemoveDuplicatesSlimPicard NoExceptionThrown ConfigurationFactory INFO Processing tool mergeAndRemoveDuplicatesSlimBioBambam NoExceptionThrown ConfigurationFactory INFO Processing tool mergeAndRemoveDuplicatesSlimSambamba NoExceptionThrown ConfigurationFactory INFO Processing tool fakeDupmarkMetrics NoExceptionThrown ConfigurationFactory INFO Processing tool printMissingReadGroups NoExceptionThrown ConfigurationFactory INFO Processing tool combinedBamAnalysis NoExceptionThrown ConfigurationFactory INFO Processing tool annotateCnvVcf NoExceptionThrown ConfigurationFactory INFO Processing tool addMappability NoExceptionThrown ConfigurationFactory INFO Processing tool estimateSex NoExceptionThrown ConfigurationFactory INFO Processing tool mergeFilterCnv NoExceptionThrown ConfigurationFactory INFO Processing tool correctGcBiasR NoExceptionThrown ConfigurationFactory INFO Processing tool convertToJson NoExceptionThrown ConfigurationFactory INFO Processing tool annotateCovWindows NoExceptionThrown ConfigurationFactory INFO Processing tool mergeAndFilterCovWindows NoExceptionThrown ConfigurationFactory INFO Processing tool correctGc NoExceptionThrown ConfigurationFactory INFO Load configuration file /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/COWorkflowsBasePlugin_1.2.0/resources/configurationFiles/commonCOWorkflowsSettings.xml NoExceptionThrown ConfigurationFactory INFO Processing tool analyzeBamHeader NoExceptionThrown ConfigurationFactory INFO Processing tool checkBwaAndIndexVersions NoExceptionThrown ConfigurationFactory INFO Processing tool annotateVcfFile NoExceptionThrown ConfigurationFactory INFO Processing tool convertNewcolsToVcf NoExceptionThrown ConfigurationFactory INFO Processing tool createpipes NoExceptionThrown ConfigurationFactory INFO Processing tool headeredFileConcatenator NoExceptionThrown ConfigurationFactory INFO Processing tool newColsToVcf NoExceptionThrown ConfigurationFactory INFO Processing tool processAnnovar NoExceptionThrown ConfigurationFactory INFO Processing tool seqContextAnnotator NoExceptionThrown ConfigurationFactory INFO Processing tool vcfToAnnovar NoExceptionThrown ConfigurationFactory INFO Processing tool bamIsComplete NoExceptionThrown XSDValidator INFO Will validate configuration exomeAnalysis. NoExceptionThrown XSDValidator INFO Will validate configuration qcAnalysis. NoExceptionThrown ProjectLoader INFO Found project class: de.dkfz.roddy.core.Project NoExceptionThrown RuntimeService WARNING Reading in jobs is not fully enabled! See RuntimeService readInExecutionContext(). The method does not reconstruct parent files and dependencies. NoExceptionThrown ProjectLoader INFO Created an analysis object of class de.dkfz.roddy.core.Analysis. NoExceptionThrown de.dkfz.roddy.execution.io.LocalExecutionService INFO de.dkfz.roddy.execution.io.MetadataTableFactory.getTable: Building metadata table from filesystem input is not implemented. NoExceptionThrown RuntimeService INFO Looking for datasets in /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/Data_folder/data. NoExceptionThrown RuntimeService INFO Found 2 datasets in /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/Data_folder/data. NoExceptionThrown RuntimeService INFO Looking for datasets in /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/Data_folder/results. NoExceptionThrown RuntimeService INFO Found 2 datasets in /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/Data_folder/results. NoExceptionThrown RuntimeService INFO Found 2 datasets in the in- and output directories. NoExceptionThrown RuntimeService INFO Selected dataset tumor for processing. NoExceptionThrown Analysis INFO QUERY_STATUS NoExceptionThrown Analysis INFO QUERY_STATUS NoExceptionThrown de.dkfz.roddy.execution.io.fs.FileSystemAccessProvider INFO There was an error while trying to create file /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/Data_folder/results/tumor/roddyExecutionStore/exec_240622_203131504_gerwin_WES/jobStateLogfile.txt NoExceptionThrown de.dkfz.roddy.execution.io.fs.FileSystemAccessProvider INFO There was an error while trying to load file /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/Data_folder/results/.roddyExecutionStore/zippedAnalysesMD5.txt NoExceptionThrown Analysis INFO An unknown / unhandled exception occurred: 'Could not zip folder /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows/resources/analysisTools/exomePipeline to zip archive /Users/gerwin/.roddy/compressedAnalysisTools/cTools_AlignmentAndQCWorkflows:develop_exomePipeline.zip' NoExceptionThrown Analysis INFO Could not zip folder /Users/gerwin/Andreas/Pipeline3_DKFZ/RoddyProject/Roddy/dist/plugins/AlignmentAndQCWorkflows/resources/analysisTools/exomePipeline to zip archive /Users/gerwin/.roddy/compressedAnalysisTools/cTools_AlignmentAndQCWorkflows:develop_exomePipeline.zip de.dkfz.roddy.tools.RoddyIOHelperMethods$NativeLinuxZipCompressor.compressDirectory(RoddyIOHelperMethods.groovy:86) de.dkfz.roddy.tools.RoddyIOHelperMethods.compressDirectory(RoddyIOHelperMethods.groovy:222) de.dkfz.roddy.tools.RoddyIOHelperMethods.compressDirectory(RoddyIOHelperMethods.groovy) de.dkfz.roddy.execution.io.ExecutionService$_compressToolFolders_closure11.doCall(ExecutionService.groovy:710) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027) groovy.lang.Closure.call(Closure.java:414) groovy.lang.Closure.call(Closure.java:430) org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2030) org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1890) de.dkfz.roddy.execution.io.ExecutionService.compressToolFolders(ExecutionService.groovy:691) de.dkfz.roddy.execution.io.ExecutionService.copyAnalysisToolsForContext(ExecutionService.groovy:629) de.dkfz.roddy.execution.io.ExecutionService.writeFilesForExecution(ExecutionService.groovy:493) de.dkfz.roddy.core.Analysis.prepareExecution(Analysis.java:364) de.dkfz.roddy.core.Analysis.executeRun(Analysis.java:401) de.dkfz.roddy.core.Analysis.run(Analysis.java:202) de.dkfz.roddy.client.cliclient.RoddyCLIClient.testrun(RoddyCLIClient.groovy:524) de.dkfz.roddy.client.cliclient.RoddyCLIClient.startMode(RoddyCLIClient.groovy:119) de.dkfz.roddy.Roddy.parseRoddyStartupModeAndRun(Roddy.java:735) de.dkfz.roddy.Roddy.startup(Roddy.java:287) de.dkfz.roddy.Roddy.main(Roddy.java:214) NoExceptionThrown Analysis INFO There were errors for the execution context for dataset tumor