This is a proof of concept script how to use the futures api of dask to support the writing of results into a trajectory while still working on frames in parallel. It won't scale out a lot. It also assumes that the single_frame function returns a namedtuple with coordinates. The buffer can potentially blow up so use with care.
def dask_execution(single_frame, trajectory, client)
# have writer in main process
with Writer as w:
futures = client.map(single_frame, trajectory)
idx = 0
for f in as_completed(futures):
res = f.result()
buf[res.frame] = res.xyz
# check if we got a result to write
while True:
# write everything we can *in order*
if idx in buf:
w.write(buf[idx])
del buf[idx]
idx += 1
# we can't write anything so break
else:
break
# write leftover frames
for i in sorted(buf):
w.write(buf[i])
This is a proof of concept script how to use the futures api of dask to support the writing of results into a trajectory while still working on frames in parallel. It won't scale out a lot. It also assumes that the
single_framefunction returns a namedtuple with coordinates. The buffer can potentially blow up so use with care.