Neural Network submission to the challenge including 3x2 and FoM or SNR

EiffL commented 4 years ago

This PR presents a complete solution to the current challenge, including trained models, plots, and result metrics. It's an update of #4 for the new form of the challenge.

I haven't done any optimization of the model yet, and there is a high probability that I still have bugs somewhere, but wanted to share some of the results.

Here are the caveats:

I have only tried on riz so far
I have replaced the SNR cut on the catalog by an imag < 25.5 cut, see #8
I'm using a linear power spectrum because I haven't implement halofit yet in the jax-cosmo library

When optimized on the total 3x2 SNR, the neural network generally tries to build disjoint bins: NeuralNetwork_bins_4metric_SNR_riz

When optimizing on the FoM, it doesn't care that much and seems to like weird solutions where one large bin has contributions at both low and high redshift: NeuralNetwork_bins_4metric_FOM_riz

plots can be found in the plots folder, and results of the metric in the example folder. Note that I have noticed the FoM values out of cosmosis to be quite finicky so I don't trust them too much.

EiffL commented 4 years ago

Here are my results with the new Buzzard dataset, only on riz:

First the numbers. This is when optimizing for the 3x2pt SNR:

NeuralNetwork run_2 {'bins': 2, 'metric': 'SNR', 'colors': True, 'errors': True} {'SNR_ww': 241.63026428222656, 'SNR_gg': 940.3321533203125, 'SNR_3x2': 942.7998046875, 'FOM_3x2': 243.771484375, 'FOM_DETF_3x2': 15.72159194946289} 
NeuralNetwork run_3 {'bins': 3, 'metric': 'SNR', 'colors': True, 'errors': True} {'SNR_ww': 248.8756561279297, 'SNR_gg': 1102.00390625, 'SNR_3x2': 1104.1590576171875, 'FOM_3x2': 973.4478759765625, 'FOM_DETF_3x2': 26.389142990112305} 
NeuralNetwork run_4 {'bins': 4, 'metric': 'SNR', 'colors': True, 'errors': True} {'SNR_ww': 249.84535217285156, 'SNR_gg': 1282.79638671875, 'SNR_3x2': 1283.5755615234375, 'FOM_3x2': 1571.1767578125, 'FOM_DETF_3x2': 43.85069274902344} 
NeuralNetwork run_6 {'bins': 6, 'metric': 'SNR', 'colors': True, 'errors': True} {'SNR_ww': 251.74142456054688, 'SNR_gg': 1534.844970703125, 'SNR_3x2': 1535.3115234375, 'FOM_3x2': 3477.149169921875, 'FOM_DETF_3x2': 62.80626678466797}

and this is when optimizing for the FoM

NeuralNetwork run_2 {'bins': 2, 'metric': 'FOM_DETF', 'colors': True, 'errors': True} {'SNR_ww': 229.99960327148438, 'SNR_gg': 875.3194580078125, 'SNR_3x2': 880.5097045898438, 'FOM_3x2': 879.8873901367188, 'FOM_DETF_3x2': 19.864736557006836} 
NeuralNetwork run_3 {'bins': 3, 'metric': 'FOM_DETF', 'colors': True, 'errors': True} {'SNR_ww': 241.28668212890625, 'SNR_gg': 1048.0006103515625, 'SNR_3x2': 1049.4024658203125, 'FOM_3x2': 1502.3333740234375, 'FOM_DETF_3x2': 40.038536071777344} 
NeuralNetwork run_4 {'bins': 4, 'metric': 'FOM_DETF', 'colors': True, 'errors': True} {'SNR_ww': 251.23098754882812, 'SNR_gg': 1225.9722900390625, 'SNR_3x2': 1226.718994140625, 'FOM_3x2': 2399.321044921875, 'FOM_DETF_3x2': 57.30086135864258} 
NeuralNetwork run_6 {'bins': 6, 'metric': 'FOM_DETF', 'colors': True, 'errors': True} {'SNR_ww': 252.17022705078125, 'SNR_gg': 1488.3408203125, 'SNR_3x2': 1488.90576171875, 'FOM_3x2': 4554.12744140625, 'FOM_DETF_3x2': 70.29737091064453}
NeuralNetwork run_8 {'bins': 8, 'metric': 'FOM_DETF', 'colors': True, 'errors': True} {'SNR_ww': 252.5953826904297, 'SNR_gg': 1566.8817138671875, 'SNR_3x2': 1567.531005859375, 'FOM_3x2': 5813.3740234375, 'FOM_DETF_3x2': 84.68102264404297} 
NeuralNetwork run_10 {'bins': 10, 'metric': 'FOM_DETF', 'colors': True, 'errors': True} {'SNR_ww': 253.09963989257812, 'SNR_gg': 1743.9248046875, 'SNR_3x2': 1744.4259033203125, 'FOM_3x2': 7897.42431640625, 'FOM_DETF_3x2': 95.17367553710938} 
NeuralNetwork run_12 {'bins': 12, 'metric': 'FOM_DETF', 'colors': True, 'errors': True} {'SNR_ww': 253.20083618164062, 'SNR_gg': 1806.3873291015625, 'SNR_3x2': 1806.93505859375, 'FOM_3x2': 8557.2333984375, 'FOM_DETF_3x2': 103.61077880859375}

And here are what these bins look like:

When optimizing for the 3x2pt SNR: $NeuralNetwork_{'bins': 2, 'metric': 'SNR', 'colors': True, 'errors': True}_riz$ $NeuralNetwork_{'bins': 3, 'metric': 'SNR', 'colors': True, 'errors': True}_riz$ $NeuralNetwork_{'bins': 4, 'metric': 'SNR', 'colors': True, 'errors': True}_riz$ $NeuralNetwork_{'bins': 6, 'metric': 'SNR', 'colors': True, 'errors': True}_riz$
When optimizing for the 3x2pt DETF FoM: $NeuralNetwork_{'bins': 2, 'metric': 'FOM_DETF', 'colors': True, 'errors': True}_riz$ $NeuralNetwork_{'bins': 3, 'metric': 'FOM_DETF', 'colors': True, 'errors': True}_riz$ $NeuralNetwork_{'bins': 4, 'metric': 'FOM_DETF', 'colors': True, 'errors': True}_riz$ $NeuralNetwork_{'bins': 6, 'metric': 'FOM_DETF', 'colors': True, 'errors': True}_riz$

EiffL commented 4 years ago

For reference, here are my final Buzzard results:

riz:

NeuralNetwork run_FOM_5 {'bins': 5, 'metric': 'FOM_DETF', 'output_dir': 'models_buzzard', 'colors': True, 'errors': True} {'SNR_ww': 250.12115478515625, 'SNR_gg': 1300.8900146484375, 'SNR_3x2': 1301.9532470703125, 'FOM_3x2': 4522.55810546875, 'FOM_DETF_3x2': 66.7544174194336} 
NeuralNetwork run_FOM_10 {'bins': 10, 'metric': 'FOM_DETF', 'output_dir': 'models_buzzard', 'colors': True, 'errors': True} {'SNR_ww': 253.08575439453125, 'SNR_gg': 1760.962158203125, 'SNR_3x2': 1761.4442138671875, 'FOM_3x2': 7662.20947265625, 'FOM_DETF_3x2': 98.19822692871094} 
NeuralNetwork run_5 {'bins': 5, 'metric': 'SNR', 'output_dir': 'models_buzzard', 'colors': True, 'errors': True} {'SNR_ww': 250.41868591308594, 'SNR_gg': 1439.7318115234375, 'SNR_3x2': 1440.2572021484375, 'FOM_3x2': 1937.485595703125, 'FOM_DETF_3x2': 51.96028518676758} 
NeuralNetwork run_10 {'bins': 10, 'metric': 'SNR', 'output_dir': 'models_buzzard', 'colors': True, 'errors': True} {'SNR_ww': 251.58102416992188, 'SNR_gg': 1880.4410400390625, 'SNR_3x2': 1880.8570556640625, 'FOM_3x2': 6889.970703125, 'FOM_DETF_3x2': 79.61094665527344}

griz:

NeuralNetwork run_FOM_5 {'bins': 5, 'metric': 'FOM_DETF', 'output_dir': 'models_buzzard', 'colors': True, 'errors': True} {'SNR_ww': 258.38958740234375, 'SNR_gg': 1384.36376953125, 'SNR_3x2': 1384.8883056640625, 'FOM_3x2': 3703.44677734375, 'FOM_DETF_3x2': 74.07708740234375} 
NeuralNetwork run_FOM_10 {'bins': 10, 'metric': 'FOM_DETF', 'output_dir': 'models_buzzard', 'colors': True, 'errors': True} {'SNR_ww': 261.1523742675781, 'SNR_gg': 1865.298828125, 'SNR_3x2': 1865.6043701171875, 'FOM_3x2': 8270.875, 'FOM_DETF_3x2': 112.05186462402344} 
NeuralNetwork run_5 {'bins': 5, 'metric': 'SNR', 'output_dir': 'models_buzzard', 'colors': True, 'errors': True} {'SNR_ww': 254.9384765625, 'SNR_gg': 1484.89501953125, 'SNR_3x2': 1485.2786865234375, 'FOM_3x2': 1937.9420166015625, 'FOM_DETF_3x2': 47.334476470947266} 
NeuralNetwork run_10 {'bins': 10, 'metric': 'SNR', 'output_dir': 'models_buzzard', 'colors': True, 'errors': True} {'SNR_ww': 259.73089599609375, 'SNR_gg': 2053.77685546875, 'SNR_3x2': 2054.05224609375, 'FOM_3x2': 5891.92724609375, 'FOM_DETF_3x2': 88.70637512207031}

These list results for training with the FoM DETF or 3x2 SNR as the loss function

EiffL commented 4 years ago

And same thing for DC2:

riz:

NeuralNetwork run_FOM_5 {'bins': 5, 'metric': 'FOM_DETF', 'output_dir': 'models', 'colors': True, 'errors': True} {'SNR_ww': 354.7750244140625, 'SNR_gg': 1270.3101806640625, 'SNR_3x2': 1272.4361572265625, 'FOM_3x2': 4266.70849609375, 'FOM_DETF_3x2': 117.60552215576172} 
NeuralNetwork run_FOM_10 {'bins': 10, 'metric': 'FOM_DETF', 'output_dir': 'models', 'colors': True, 'errors': True} {'SNR_ww': 357.55303955078125, 'SNR_gg': 1693.6142578125, 'SNR_3x2': 1695.31591796875, 'FOM_3x2': 11150.6083984375, 'FOM_DETF_3x2': 161.53375244140625} 
NeuralNetwork run_5 {'bins': 5, 'metric': 'SNR', 'output_dir': 'models', 'colors': True, 'errors': True} {'SNR_ww': 335.0947570800781, 'SNR_gg': 1384.146240234375, 'SNR_3x2': 1387.193359375, 'FOM_3x2': 2961.429931640625, 'FOM_DETF_3x2': 44.63212966918945} 
NeuralNetwork run_10 {'bins': 10, 'metric': 'SNR', 'output_dir': 'models', 'colors': True, 'errors': True} {'SNR_ww': 354.5575866699219, 'SNR_gg': 1830.9071044921875, 'SNR_3x2': 1832.388427734375, 'FOM_3x2': 10417.8525390625, 'FOM_DETF_3x2': 131.1063995361328}

griz:

NeuralNetwork run_FOM_5 {'bins': 5, 'metric': 'FOM_DETF', 'output_dir': 'models', 'colors': True, 'errors': True} {'SNR_ww': 363.4723815917969, 'SNR_gg': 1350.76708984375, 'SNR_3x2': 1352.3494873046875, 'FOM_3x2': 4233.681640625, 'FOM_DETF_3x2': 132.9571990966797} 
NeuralNetwork run_FOM_10 {'bins': 10, 'metric': 'FOM_DETF', 'output_dir': 'models', 'colors': True, 'errors': True} {'SNR_ww': 368.1811218261719, 'SNR_gg': 1849.5284423828125, 'SNR_3x2': 1850.74072265625, 'FOM_3x2': 11092.52734375, 'FOM_DETF_3x2': 188.56341552734375} 
NeuralNetwork run_5 {'bins': 5, 'metric': 'SNR', 'output_dir': 'models', 'colors': True, 'errors': True} {'SNR_ww': 358.8480529785156, 'SNR_gg': 1437.14501953125, 'SNR_3x2': 1438.4747314453125, 'FOM_3x2': 3254.669189453125, 'FOM_DETF_3x2': 104.16570281982422} 
NeuralNetwork run_10 {'bins': 10, 'metric': 'SNR', 'output_dir': 'models', 'colors': True, 'errors': True} {'SNR_ww': 366.8392028808594, 'SNR_gg': 1972.53662109375, 'SNR_3x2': 1973.5740966796875, 'FOM_3x2': 9922.68359375, 'FOM_DETF_3x2': 165.6302032470703}

LSSTDESC / tomo_challenge

Neural Network submission to the challenge including 3x2 and FoM or SNR #9