search

usegalaxy-no / galaxyadmin

A repository for managing the work of the usegalaxy.no GalaxyAdmin team
0 stars 0 forks source link

big Trinity job #73

Closed tothuhien closed 1 year ago

tothuhien commented 1 year ago

Job [228409] run Trinity (toolshed.g2.bx.psu.edu/repos/iuc/trinity/trinity/2.9.1+galaxy2) stuck on the step bash -c cat left.fa right.fa > both.fa because the 2 input fast files are so big (138G each). This might be the reason that made the disk access quite slow, and uploading data took so long time. The job ended with error of passing limited time (> 6 days), but still in CG status

tothuhien commented 1 year ago

Copy from @kjetilkl comment on Slack: There is a Trinity job which has been running on the [slurm.usegalaxy.no](http://slurm.usegalaxy.no/) node for 24 hours now, and it is occupying 170 GB of the node's 200 GB memory (according to "tool_destination.yaml" and Slurm). This has resulted in 16 HISAT2 jobs being stuck in the queue, since we previously configured HISAT2, TopHat2, Bowtie2 and RNA STAR to only run on this compute node (which has a larger CVMFS cache for reference datasets). However, since the reference dataset size for most of these tools is around 4 GB, and thus smaller than the 8 GB CVMFS caches on the other compute nodes, I have decided to revert back the "tool_destinations.yaml" configuration to allow these tools to run on all compute nodes again.