search

proteomicsunitcrg / qcloud2-pipeline

QCloud2 pipeline
5 stars 1 forks source link

Zip interruption error at ftp2nf.sh causes a critical change at the queue file order #66

Closed rolivella closed 3 years ago

rolivella commented 3 years ago

less /users/pr/rolivella/mybackups/ftp2nf/ftp2nf.log

[INFO] *** START MANAGING FILE blank_QC03_210717155026_7d1adc32-1a73-4ced-ad59-bdf55f89ba84_QC03_d5e6706b251bd79a1453488162adbe3e.raw... [INFO] CHECKSUM COMPARISION... [INFO] Received file blank_QC03_210717155026_7d1adc32-1a73-4ced-ad59-bdf55f89ba84_QC03_d5e6706b251bd79a1453488162adbe3e.raw checksum is OK so it's going to be zipped and moved to Nextflow incoming folder... [INFO] ZIPPING file... zip warning: /users/pr/qcloud/scratch/ba3846b9-948c-4790-bd15-edf32d7d330f/blank_QC03_210717155026_7d1adc32-1a73-4ced-ad59-bdf55f89ba84_QC03_d5e6706b251bd79a1453488162adbe3e.zip not found or empty adding: blank_QC03_210717155026_7d1adc32-1a73-4ced-ad59-bdf55f89ba84_QC03_d5e6706b251bd79a1453488162adbe3e.raw

zip error: Interrupted (aborting)

rolivella commented 3 years ago

The point is that the script is not designed to process to many large files that we receive at almost the same time. When that happens, this timeout kills the process and the unzip gets interrupted:

*/5 * * * * timeout 240 /users/pr/qcloud/bash/ftp2nf.sh 'QcCrG!18' 'v1@b007M9369dcS' >> /users/pr/qcloud/logs/ftp2nf.log 2>&1

SOLUTION (after summer holidays): decouple ftp2nf.sh into two scripts: 1) Dowload from SFTP: ftp2isis.sh 2) Unzip from scratch and move to incoming: isis2nf.sh