desihub / gpu_specter

Scratch work for porting spectroperfectionism extractions to GPUs
BSD 3-Clause "New" or "Revised" License
2 stars 3 forks source link

Add multi frame extraction script #50

Closed dmargala closed 3 years ago

dmargala commented 3 years ago

This PR adds bin/desi-extract-exposure for extracting all 30 frames in a single DESI exposure. Frames are divided among nodes and then processed sequentially.

The following figure demonstrates how frames are split among nodes for an example run on corigpu using 5 nodes, 4 GPUs per node, and 8 tasks per node:

image

The effective work rate is ~100 frames per node-hour including startup time.

Another example using just 2 nodes:

image

The effective work rate is now ~120 frames per node-hour including startup time (which is absorbed by longer overall runtime in this case).

dmargala commented 3 years ago

I updated the PR to enable more flexible distribution of work. The exposure extraction script now creates frame processing groups based on (node, CUDA_VISIBLE_DEVICES) instead of just node (and using all available GPUs).

Here is the 2 nodes example again but now splitting the work so that each gpu is processing a different frame (~120 frames per node hour, ~30 frames per gpu-hour):

image

And here is the 5 node example again, where each frame is processed using 2 GPUs (~117 frames per node hour, ~29 frames per gpu hour):

image

And for fun, here is an example the demonstrates a different frame to node mapping. Using 7 nodes, where 2 nodes process 5 r-frames each and the other 5 nodes process 2 b-frames and 2 z-frames (~106 frames per node-hour, ~26 frames per gpu hour).

image