broadinstitute / warp

WDL Analysis Research Pipelines
https://broadinstitute.github.io/warp
BSD 3-Clause "New" or "Revised" License
200 stars 93 forks source link

Make ReblockGVCFs more robust to large inputs #1330

Closed meganshand closed 1 month ago

meganshand commented 1 month ago

When running ReblockGVCFs with large input occasionally the VM was running out of disk space in the process of converting the input GVCF into an interval list for validation. This fixes that by including the interval list in the disk_size calculation.

Additionally a separate very large sample was running out of memory on the interval list generation step in the validation task. This allows the user to increase the memory of the machine and actually use it both in Validating and converting the GVCF to interval list.

In the next point release of GATK there will be a fix that's already been merged into htsjdk that will fix the requirement for a large amount of memory for the conversion, but for now giving the command more memory will do the trick.

github-actions[bot] commented 1 month ago

Remember to squash merge!

github-actions[bot] commented 1 month ago

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
WholeGenomeGermlineSingleSample.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
ExomeGermlineSingleSample.wdl has not been changed and needs updating
VariantCalling.wdl has not been changed and needs updating
ReblockGVCF.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeCramOnly.wdl has not been changed and needs updating
IlluminaGenotypingArray.wdl has not been changed and needs updating
BroadInternalUltimaGenomics.wdl has not been changed and needs updating
BroadInternalRNAWithUMIs.wdl has not been changed and needs updating
BroadInternalArrays.wdl has not been changed and needs updating
ExternalWholeGenomeReprocessing.wdl has not been changed and needs updating
ExternalExomeReprocessing.wdl has not been changed and needs updating
WholeGenomeReprocessing.wdl has not been changed and needs updating
ExomeReprocessing.wdl has not been changed and needs updating
CheckFingerprint.wdl has not been changed and needs updating
Arrays.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed
github-actions[bot] commented 1 month ago

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
WholeGenomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
ExomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
VariantCalling.changelog.md has not been changed and needs to be updated
ReblockGVCF.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeCramOnly.changelog.md has not been changed and needs to be updated
IlluminaGenotypingArray.changelog.md has not been changed and needs to be updated
BroadInternalUltimaGenomics.changelog.md has not been changed and needs to be updated
BroadInternalRNAWithUMIs.changelog.md has not been changed and needs to be updated
BroadInternalArrays.changelog.md has not been changed and needs to be updated
ExternalWholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExternalExomeReprocessing.changelog.md has not been changed and needs to be updated
WholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExomeReprocessing.changelog.md has not been changed and needs to be updated
CheckFingerprint.changelog.md has not been changed and needs to be updated
Arrays.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed
github-actions[bot] commented 1 month ago

Remember to squash merge!

github-actions[bot] commented 1 month ago

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
WholeGenomeGermlineSingleSample.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
ExomeGermlineSingleSample.wdl has not been changed and needs updating
VariantCalling.wdl has not been changed and needs updating
ReblockGVCF.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeCramOnly.wdl has not been changed and needs updating
IlluminaGenotypingArray.wdl has not been changed and needs updating
BroadInternalUltimaGenomics.wdl has not been changed and needs updating
BroadInternalRNAWithUMIs.wdl has not been changed and needs updating
BroadInternalArrays.wdl has not been changed and needs updating
ExternalWholeGenomeReprocessing.wdl has not been changed and needs updating
ExternalExomeReprocessing.wdl has not been changed and needs updating
WholeGenomeReprocessing.wdl has not been changed and needs updating
ExomeReprocessing.wdl has not been changed and needs updating
CheckFingerprint.wdl has not been changed and needs updating
Arrays.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed
github-actions[bot] commented 1 month ago

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
WholeGenomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
ExomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
VariantCalling.changelog.md has not been changed and needs to be updated
ReblockGVCF.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeCramOnly.changelog.md has not been changed and needs to be updated
IlluminaGenotypingArray.changelog.md has not been changed and needs to be updated
BroadInternalUltimaGenomics.changelog.md has not been changed and needs to be updated
BroadInternalRNAWithUMIs.changelog.md has not been changed and needs to be updated
BroadInternalArrays.changelog.md has not been changed and needs to be updated
ExternalWholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExternalExomeReprocessing.changelog.md has not been changed and needs to be updated
WholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExomeReprocessing.changelog.md has not been changed and needs to be updated
CheckFingerprint.changelog.md has not been changed and needs to be updated
Arrays.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed
github-actions[bot] commented 1 month ago

Remember to squash merge!

github-actions[bot] commented 1 month ago

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed
github-actions[bot] commented 1 month ago

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed
github-actions[bot] commented 1 month ago

Remember to squash merge!

github-actions[bot] commented 1 month ago

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.
github-actions[bot] commented 1 month ago

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.
github-actions[bot] commented 1 month ago

Remember to squash merge!

github-actions[bot] commented 1 month ago

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.
github-actions[bot] commented 1 month ago

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
UltimaGenomicsWholeGenomeGermline.wdl and UltimaGenomicsWholeGenomeGermline.changelog.md do not have matching versions
ExomeGermlineSingleSample.wdl and ExomeGermlineSingleSample.changelog.md do not have matching versions
UltimaGenomicsWholeGenomeCramOnly.wdl and UltimaGenomicsWholeGenomeCramOnly.changelog.md do not have matching versions
IlluminaGenotypingArray.wdl and IlluminaGenotypingArray.changelog.md do not have matching versions
BroadInternalUltimaGenomics.wdl and BroadInternalUltimaGenomics.changelog.md do not have matching versions
BroadInternalRNAWithUMIs.wdl and BroadInternalRNAWithUMIs.changelog.md do not have matching versions
BroadInternalArrays.wdl and BroadInternalArrays.changelog.md do not have matching versions
CheckFingerprint.wdl and CheckFingerprint.changelog.md do not have matching versions
Arrays.wdl and Arrays.changelog.md do not have matching versions
Some WDLs or changelog files need updating. See output for details.
validation_failed
github-actions[bot] commented 1 month ago

Remember to squash merge!

github-actions[bot] commented 1 month ago

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.
github-actions[bot] commented 1 month ago

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.