bidhya / verse

0 stars 0 forks source link

Adaptive choose number of rows for Slurm Job #9

Closed bidhya closed 6 months ago

bidhya commented 6 months ago

Due to uneven distribution of land mass, we get a variable number of valid pixels (ie, not the NODATA). This caused over or underestimation of Slurm runtimes.

For Milan nodes with 126 cores and 500 GB ram, we can safely fit between 50,000 to 100,000 pixels per run without incurring error. Cores are reduced to 80 or 90 though.

Moreover, it takes longer time for optimizer in lower latitudes, again, likely because optimizer is hard at work finding snow when there is none!

Proposal here is to adapt the number of pixels submitted for Blender for maximum efficiency and throughput defined as follows:

bidhya commented 6 months ago

8890818d0e4e23ccee7c2b7da6985ebaf6a9710a fixes this.