mixcr_RNASeq persistent error

[slsevilla]

Problem:

• Xinyu is getting a persistent error with a subset of samples related to rule mixcr_RNASeq.

Example log info:

cat /data/khanlab2/processed_DATA/ngs_pipeline_SJ031111=SJEPD031111_D1=20220911_20221116_151637.log | grep "mixcr_RNASeq"

cat /data/khanlab2/processed_DATA/log/mixcr_RNASeq.52735119.e

Relevant error code:

Java HotSpot(TM) 64-Bit Server VM warning: Ignoring option AggressiveOpts; support was removed in 12.0
slurmstepd: error: *** JOB 52735119 ON cn2414 CANCELLED AT 2022-11-17T04:25:01 DUE TO TIME LIMIT ***
++ kill -TERM 19078
++ touch /gpfs/gsfs10/users/khanlab2/processed_DATA/.snakemake/tmp.yznfwbmd/46.jobfailed
+ trap - SIGTERM
+ wait 19078
+ exit 143

Reviewing samples from early Nov (12) the average processing time was about less than an hour. Start and end times are shown for a random subset of samples

for f in /data/khanlab2/processed_DATA/log/*mixcr_RNASeq.525*.e; do echo $f; cat $f | grep "Nov"; done

/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52568629.e
[Sat Nov 12 13:46:48 2022]
[Sat Nov 12 14:06:05 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52568645.e
[Sat Nov 12 13:46:50 2022]
[Sat Nov 12 14:21:13 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52568646.e
[Sat Nov 12 13:46:49 2022]
[Sat Nov 12 14:28:52 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52568684.e
[Sat Nov 12 13:46:49 2022]
[Sat Nov 12 14:00:55 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52568688.e
[Sat Nov 12 13:46:53 2022]
[Sat Nov 12 14:11:14 2022]

Later in the month (Nov ) this time increases to 2 hours. Again, start and end times are shown for a random subset of samples:

for f in /data/khanlab2/processed_DATA/log/*mixcr_RNASeq.527*.e; do echo $f; cat $f | grep "Nov"; done

/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52717122.e
[Wed Nov 16 15:20:24 2022]
[Wed Nov 16 17:31:47 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52717123.e
[Wed Nov 16 15:20:21 2022]
[Wed Nov 16 18:09:31 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52717126.e
[Wed Nov 16 15:20:21 2022]
[Wed Nov 16 17:44:03 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52717129.e
[Wed Nov 16 15:20:21 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52717137.e
[Wed Nov 16 15:20:27 2022]
[Wed Nov 16 17:26:18 2022]
/data/khanlab2/processed_DATA/log/mixcr_RNASeq.52717197.e
[Wed Nov 16 15:20:22 2022]
[Wed Nov 16 17:42:53 2022]

For the samples that failed, the maximum 6 hours of processing time is being reached.

[Wed Nov 16 22:24:38 2022]
CANCELLED AT 2022-11-17T04:25:01

Solution:

• Currently the cpus-per-task is set to --cpus-per-task=16. All samples would benefit from increasing this to --cpus-per-task=32 which will likely help with processing of the majority of samples. If this still fails, time will need to be increased as well.

[slsevilla]

completed with commit f2cbe1eccd87a1363f807cf5e9731dbbe0607aad