sgkit-dev / bio2zarr

Convert bioinformatics file formats to Zarr
Apache License 2.0
27 stars 7 forks source link

Document dexplode/dencode usage on different schedulers #147

Open jeromekelleher opened 6 months ago

jeromekelleher commented 6 months ago

Here's how you do it with xargs (for my own reference; in reality you'd use the worker processes option):

 seq 0 (NPARTS -1) |  xargs -t -I % -P NUM_PROCS vcf2zarr dexplode-partition [ICF_DIR] %
benjeffery commented 6 months ago

On LSF I have a bash script:

N=$LSB_JOBINDEX
vcf2zarr dexplode-partition [ICF_DIR] $((N-1))

Then submit with bsub -J job_name[1-(NPARTS+1)]

jeromekelleher commented 6 months ago

Should we add a "--one-based" option to dexplode-partition and dencode-partiton? It's fiddly having to translate to zero-based in the shell for these things that want to go 1...N