Collect NERSC Usage Estimates for Oct 11 DOE Response

[richardxdubois]

Lali Chatterjee has asked for an update on how much NERSC time we plan to use from our current allocation (30M KNL NERSC-hrs). She is targeting groups with 2M+ hrs to redistribute time from. Response is due COB Wed, Oct 11.

This issue would collect inputs for that response.

[katrinheitmann]

• 1M for generating extragalactic catalog for protoDC2 (downscaled catalog for DC2), will start as soon as NERSC is back on

[jchiang87]

• 6M for phosim simulations of the protoDC2 field for all six bands and 10 years. This is based on Salman's estimate of 70M cori II hours for phosim generation of the full DC2 dataset, i.e., 70M*(25 sq deg/300 sq deg).

[rmandelb]

• 6M for GalSim atmosphere tests - specifically convergence and accuracy tests for the atmospheric PSFs depending on what DFT settings were used. These will be used to define on the best DFT settings to use for a DFT vs. photon-shooting (imsim and PhoSim) comparison. The 6M assumes: ~5 realizations of a 3x3 raft at the default DFT settings, and then a grid of values of the 4 DFT settings that are relevant. We included a factor of 6 slowdown on KNL that Jim just confirmed is needed when running GalSim (even after rebuilding it for KNL), based on running code provided by Josh Meyers.

Edited to say that we can reduce this test by a factor of 5 in expense, so with the factor of 5 slowdown Richard mentioned, it's still 6M NERSC KNL hours.

[richardxdubois]

For Jim's estimate - I'm carrying in my head that protoDC2 is 2% of DC2. His calculation is for 25 sq deg. What am I missing?

[richardxdubois]

For Rachel's estimate - this is in good old CPU hrs... we have to apply x2.5 to get to NERSC-hrs, and James found that he could only fill 1/2 a KNL node with ImSim jobs. So Rachel's 6M CPU-hrs floats up to 30M NERSC-hrs - which is all we got.

[katrinheitmann]

I think it's me who can't do the math correctly without a supercomputer anymore ;-) I think I might out of 5x5 just 5 and divided 5/300 which is obviously not right. Sorry for that. I think I did that late at night.

On 10/10/17 4:25 PM, Richard Dubois wrote:

For Jim's estimate - I'm carrying in my head that protoDC2 is 2% of DC2. His calculation is for 25 sq deg. What am I missing?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/ComputingInfrastructure/issues/50#issuecomment-335612927, or mute the thread https://github.com/notifications/unsubscribe-auth/AMQ9jJgNxWiYnioz5at6HS11iq3SaJD7ks5sq-DDgaJpZM4PzGFf.

[rmandelb]

Never fear, @richardxdubois , I'm iterating with the others about a way to reduce the cost of that test. Will edit my original comment once I get confirmation from the others that the new plan will work (or, an alternate suggestion from them).

[richardxdubois]

If we are to have a small DC1 re-do with PhoSim 3.7, how much is enough to benchmark? @jchiang87, @salmanhabib

[salmanhabib]

I would guess about 5M NERSC hours depending on how much work we want to do.

On 10/10/17 6:08 PM, Richard Dubois wrote:

If we are to have a small DC1 re-do with PhoSim 3.7, how much is enough to benchmark? @jchiang87 https://github.com/jchiang87, @salmanhabib https://github.com/salmanhabib

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/ComputingInfrastructure/issues/50#issuecomment-335634081, or mute the thread https://github.com/notifications/unsubscribe-auth/AInM9ZjpsA4aTQK4Oj5Z-EZ_QVxUfWpzks5sq_j9gaJpZM4PzGFf.

[richardxdubois]

So far we have a total of 1+6(*)+6+5 = 18M NERSC-hrs.

(*) @jchiang87 is checking with @salmanhabib whether his 6M for PhoSim is CPU-hrs or NERSC-hrs.

[richardxdubois]

@salmanhabib - are there any requests that might have been for AY18 that can be done in CY2017? @katrinheitmann thought there might be one at least.

[salmanhabib]

There's one for about 1M NERSC-hours, running modified gravity simulations (UT Dallas) on the KNLs. I'll send you the info separately.

On 10/11/17 11:43 AM, Richard Dubois wrote:

@salmanhabib https://github.com/salmanhabib - are there any requests that might have been for AY18 that can be done in CY2017? @katrinheitmann https://github.com/katrinheitmann thought there might be one at least.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/ComputingInfrastructure/issues/50#issuecomment-335872236, or mute the thread https://github.com/notifications/unsubscribe-auth/AInM9ZqqhkHvGeshyEEYZwilYVQMcTIbks5srPAfgaJpZM4PzGFf.

[reneehlozek]

@rbiswas4 estimated that we could ask for around 200K NERSC-hours to develop analysis pipelines for the Observing strategy task force.

[jchiang87]

@richardxdubois I went through Salman's numbers for the DC2 phosim NERSC estimate and 70M NERSC hours checks out. So I think the 6M NERSC hours for protoDC2 phosim production is ok too.

[richardxdubois]

So it looks like we're asking Lali for about 20M NERSC-hrs to tide us until the new allocation year....

[cwwalter]

Did we include DM reprocessing with v14?

[richardxdubois]

re DM of v14 - no. I think that would add 1.5M or so. And a lot of space we don't have... But I can add it to the list.

[richardxdubois]

I've sent in the response to Lali.

[drphilmarshall]

Thanks Richard, thanks all. I think our return will be well appreciated - and it's good to have a plan for the remainder of the calendar year!

[richardxdubois]

Lali has responded that she will remove 8M hrs from our allocation, and has done so. Our allocation is now 23.3M NERSC-hrs.