Handle image IO in a separate thread

[russelldj]

I haven't benchmarked it, but as we optimize the rendering and aggregation code, I expect that IO will become a bottleneck. It would be helpful to look into whether we can read/write the images asynchronously in another thread.

[russelldj]

This would probably be a queue where you push in the main thread and pop in the writing thread.

[russelldj]

An additional step would be to do IO in a temp directory. In previous experiments we found that the /ofo-share is slower than writing to the user space. So it might be good to write to the user folder and then batch copy them to /ofo-share.

[asidhu0]

The speed of label read, write, and copy was tested to/from user space and ofo-share using the images in /ofo-share/scratch-david/organized-str-disp-MVMT-experiments/per_site_processing/delta/03_training_data/renders. The results shown below display that the label operations were fairly fast and thus threading is not necessary since IO is not the main bottleneck.

(1) Read/write to ofo-share Average read time per time label: 0.000451 seconds Average write time per time label: 0.009240 seconds Total time taken to read 708 files: 0.3190 Total time taken to write 708 files: 6.5419

(2) Read/write to user space Average read time per time label: 0.000416 seconds Average write time per time label: 0.000180 seconds Total time taken to read 708 files: 0.2942 Total time taken to write 708 files: 0.1272

(3) Copy from ofo-share to ofo-share Average copy time per time label: 0.016808 seconds

(4) Copy from user space to ofo-share Average copy time per time label: 0.015213 seconds

(5) Copy from ofo-share to user space Average copy time per time label: 0.000186 seconds

(6) Copy from user space to user space Average copy time per time label: 0.000135 seconds