Fails with many input files

alanocallaghan commented 2 years ago

I think this may be a general snakemake issue rather than with this profile specifically, but I'm not a python expert.

I have a rule that runs c. 100k jobs and uses all of them together as input to a new rule. Because the input is expanded to 100k items, and the job properties (including input) are written to the jobscript, this means that the jobscript is way over the file size limit for sbatch. This means you get the error :

sbatch: error: Batch job submission failed: Pathname of a file, directory or other parameter too long

My naive solution was to try removing the bit where the job properties are written to the jobscript, but this doesn't work because that's where the properties are later read from to set cluster parameters by slurm-utils. Other than that I'm not quite sure what to do, other than maybe deling some of the properties and then re-writing the jobscript just before submission.

jdblischak commented 2 years ago

My naive solution was to try removing the bit where the job properties are written to the jobscript

@Alanocallaghan In case it could be helpful, I put together an example with my smk-simple-slurm profile that uses your idea of a custom jobscript to submit a job with 150k input files:

https://github.com/jdblischak/smk-simple-slurm/tree/main/examples/many-input-files