Updates the transfer_data.py utility to add new options:
--zip_fastqs puts all Fastqs into a ZIP archive before copying to the destination (using the manage_fastqs.py zip option)
--max_zip_size sets a size limit on the Fastq ZIP archive, creating multiple ZIPs which individually will be no larger than this limit
--no_fastqs excludes all Fastqs from transfer (for example, if only QC files are required)
Implementation of these changes requires a new function listdir() in the fileops module (list contents of a local or remote directory).
Additionally the configuration for file transfer destinations (defined in the config file by [destination:...] sections) has been extended to include settings for ZIP archives.
The PR also overhauls the error checking (if an operation fails) and fixes a minor bug with specifying Fastq subdirectories.
Updates the
transfer_data.pyutility to add new options:--zip_fastqsputs all Fastqs into a ZIP archive before copying to the destination (using themanage_fastqs.py zipoption)--max_zip_sizesets a size limit on the Fastq ZIP archive, creating multiple ZIPs which individually will be no larger than this limit--no_fastqsexcludes all Fastqs from transfer (for example, if only QC files are required)Implementation of these changes requires a new function
listdir()in thefileopsmodule (list contents of a local or remote directory).Additionally the configuration for file transfer destinations (defined in the config file by
[destination:...]sections) has been extended to include settings for ZIP archives.The PR also overhauls the error checking (if an operation fails) and fixes a minor bug with specifying Fastq subdirectories.