search

ErikOsinga / master_research_project

Repository for the Master Research Project: "Data compression for weak lensing studies with the upcoming Euclid mission with Information Maximizing Neural Networks"
0 stars 0 forks source link

Getting the imnn to work with more cosmological parameters and more redshift bins #7

Closed mjvakili closed 5 years ago

mjvakili commented 5 years ago

We should move on two multiple parameters (\Omega_c, \sigma8) and then to three parameters (\Omega_c, \sigma8, w)

Once we get that to work, we can continue the work with more redshift bins.

ErikOsinga commented 5 years ago

Sounds like a good idea, did you have any luck with getting the emulator to work yet? We're definitely going to need this for >1 parameter

mjvakili commented 5 years ago

I'm still working on getting the emulator to work. In the meantime, can you run the following tests:

(1) with multiple tomographic bins and \Omega_c

(2) Single tomographic bin: no interpolation but more restrictive prior for \Omega_c (in order to avoid exploring regions of the parameter space where CLASS is slow)

ErikOsinga commented 5 years ago

(1) I am now doing 3 tomographic bins as a start, the free parameter is Omega_m

I rescale the data by taking the negative logarithm of the data, and then flatten it before feeding it into a dense network with 2 layers with 256 nodes each.

The data visualized: data_visualization_train_1

The network learning curve:

variables_vs_epochs_1

It seems the network has no problem with this setup.

ErikOsinga commented 5 years ago

(2) A more restrictive prior does not work, because the integration hangs for seemingly random cosmologies, not necessarily cosmologies that are far away from the fidicial parameters that I am using

mjvakili commented 5 years ago

Okay then it seems like we have to fix the emulator. Can you also run the ABC to find the posteriors on Omegam?

On Sun, Mar 24, 2019, 18:27 ErikOsinga notifications@github.com wrote:

(2) A more restrictive prior does not work, because the integration hangs for seemingly random cosmologies, not necessarily cosmologies that are far away from the fidicial parameters that I am using

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ErikOsinga_master-5Fresearch-5Fproject_issues_7-23issuecomment-2D475980670&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=Pk-f6hwIZYSuypCTaVlcdQ&m=gXDwbBshzRj_vGBdf1Na4n-pcVjynNyMJYaJ8-y5eJQ&s=TDMLv0tJzYR-QqDdEAuhm2rHXhlfTPHXiyKZYmjYONU&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACJBwAMiRr7k-5FnjNYEyIh3qJoZL6QQakks5vZ7V7gaJpZM4cCga4&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=Pk-f6hwIZYSuypCTaVlcdQ&m=gXDwbBshzRj_vGBdf1Na4n-pcVjynNyMJYaJ8-y5eJQ&s=I5UuS0EbeJwsrjmZ4ohNvY5SnOG6W7m9NMgkLf0zfGI&e= .

ErikOsinga commented 5 years ago

Yes! Below is the result of ABC. I should note that since we are doing simple linear interpolation it is not very informative, because further and further away from the fiducial parameter the data will be worse and worse. In this way it is also hard to interpret if the network has correctly converged or whether the output summary simply varies linearly because the input simulations now vary linearly with the parameters

ABC_1_1D_backup2

mjvakili commented 5 years ago

While initializing the cosmology with CCL, can you set the 'nonlinear' keyword to halofit to see if that makes any difference?

ErikOsinga commented 5 years ago

There is no 'nonlinear' keyword in pyccl.Cosmology, the only keywords that are not simply parameters are transfer_function='boltzmann_class', matter_power_spectrum='halofit', baryons_power_spectrum='nobaryons', mass_function='tinker10', halo_concentration='duffy2008', emulator_neutrinos='strict',

If you mean matter_power_spectrum = 'halofit', this is the default, but when using the emulator for the transfer function the matter_power_spectrum = 'emu' is required or else this error is thrown:

ValueError: matter_power_spectrum 'halofit' can't be used with transfer_function 'emulator'.

mjvakili commented 5 years ago

Okay great! Can you set the transfer function to bbks?

On Tue, Mar 26, 2019, 17:53 ErikOsinga notifications@github.com wrote:

There is no 'nonlinear' keyword in pyccl.Cosmology, the only keywords that are not simply parameters are transfer_function='boltzmann_class', matter_power_spectrum='halofit', baryons_power_spectrum='nobaryons', mass_function='tinker10', halo_concentration='duffy2008', emulator_neutrinos='strict',

If you mean matter_power_spectrum = 'halofit', this is the default, but when using the emulator for the transfer function the matter_power_spectrum = 'emu' is required or else this error is thrown:

ValueError: matter_power_spectrum 'halofit' can't be used with transfer_function 'emulator'.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ErikOsinga_master-5Fresearch-5Fproject_issues_7-23issuecomment-2D476743934&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=Pk-f6hwIZYSuypCTaVlcdQ&m=8EtBdBWm-abn3fVpZ8hjdoAsy0-tA23-SSMeLtVbmE0&s=zds2T9GMM8hxIKIYb2T82yPBi2A__9qvd0-p4qEOS1s&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACJBwLdqz0AraZvK46a-2DmfbItXH2cP8Uks5valB-2DgaJpZM4cCga4&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=Pk-f6hwIZYSuypCTaVlcdQ&m=8EtBdBWm-abn3fVpZ8hjdoAsy0-tA23-SSMeLtVbmE0&s=PV0vF54OSa85JktqDJfb5d0wC04T5p9RLzfOUOUl_Rw&e= .

ErikOsinga commented 5 years ago

The angular power spectrum is quite different for this transfer function than for the default (boltzmann_class)

This is with boltzmann_class (default): errorbars_13

This is for bbks:

errorbars_14

mjvakili commented 5 years ago

It shouldn't look like this.

I looks like it's only calculating the linear part of the powerspectrum and not nonlinear part.

On Wed, Mar 27, 2019, 14:51 ErikOsinga notifications@github.com wrote:

The angular power spectrum is quite different for this transfer function than for the default (boltzmann_class)

This is with boltzmann_class (default): [image: errorbars_13] https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_12582441_55081155-2D9ddc6680-2D509f-2D11e9-2D8e57-2D5b5f4f6f71df.png&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=Pk-f6hwIZYSuypCTaVlcdQ&m=qliqtBf769cXCdk3gGvN6yB_5qrVyYYCh5w0p1ekkfw&s=1e8VF6pFUTkl72l8k_lGlMff3CRoW-ZSk6NRdsCEAEM&e=

This is for bbks:

[image: errorbars_14] https://urldefense.proofpoint.com/v2/url?u=https-3A__user-2Dimages.githubusercontent.com_12582441_55081188-2Dadf44600-2D509f-2D11e9-2D9924-2Dd9af356fce35.png&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=Pk-f6hwIZYSuypCTaVlcdQ&m=qliqtBf769cXCdk3gGvN6yB_5qrVyYYCh5w0p1ekkfw&s=pfGjhOxjgELHzHU00cPH2dacX7v7PzvI235K4Le1-P0&e=

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ErikOsinga_master-5Fresearch-5Fproject_issues_7-23issuecomment-2D477164029&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=Pk-f6hwIZYSuypCTaVlcdQ&m=qliqtBf769cXCdk3gGvN6yB_5qrVyYYCh5w0p1ekkfw&s=iDLZHxatANOC5GqStk_HiSgyt14niR824uYcUpLzwR4&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACJBwOuTiqtQ-5FgILj8YP9fATLnrvFZ-5Fhks5va3dFgaJpZM4cCga4&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=Pk-f6hwIZYSuypCTaVlcdQ&m=qliqtBf769cXCdk3gGvN6yB_5qrVyYYCh5w0p1ekkfw&s=OdYqAvncBMFBmN4eeh-kxtVYRQaFPPNcZ484v--a7BE&e= .

mjvakili commented 5 years ago

Closing this since this is already resolved