Currently each batch job outputs to a seperate file. Ideally they should all add their contributions to the same file in order to reduce the storage cost. A file locking system would have to be added and the functions memorySafeAtB and in blmm_batch.py and memorySafeReadAndSumAtB in blmm_concat.py would have to be modified accordingly.
Currently each batch job outputs to a seperate file. Ideally they should all add their contributions to the same file in order to reduce the storage cost. A file locking system would have to be added and the functions
memorySafeAtB
and inblmm_batch.py
andmemorySafeReadAndSumAtB
inblmm_concat.py
would have to be modified accordingly.