MultiNode (Slurm) - Allow distributing large datasets creation across multiple Slurm nodes

[iejMac]

merge in starting_shard_id PR and use that in here.

node 1 should start at the last possible generated shard of node 0