retrain speculator

[changhoonhahn]

The current version of the speculator FSPS emulator is inaccurate in the SFH basis coefficient parameter space that we're interested in (see speculator_accuracy.ipynb). We need to update the speculator FSPS emulator and validate its accuracy.

[changhoonhahn]

The speculator package now has a demo for training speculator: speculator_training_demo.ipynb.

• [ ] Retrain emulator following its instructions.
• [ ] Validate retrained emulator

[changhoonhahn]

Scripts for generating speculator training set for simple (calzetti) dust model has been implemented and deployed on NERSC (d7bd33ea94c987f61690582579c767a274284bc7):

• speculator_training.py
• deploy_speculator_trainingset.py

[changhoonhahn]

@kgb0255 lets try to keep track of the speculator training using the task list I've added here: https://github.com/changhoonhahn/gqp_mc/blob/master/run/todo.speculator_training.md

As you train speculator models with different training set sizes, pca components, etc, lets fill out the table to keep track.

[changhoonhahn]

simpledust, 300 training batches, 20 PCAs, 200 pca training batches: ~4% accuracy for 99 percentile

see https://github.com/changhoonhahn/gqp_mc/blob/ae611fa0c5471e82fda90aaa6f31b843b2477b43/nb/training_desi_simpledust_speculator.ipynb for details

[kgb0255]

simpledust, 300 training batches, 20 PCAs, 200 pca training batches: ~4% accuracy for 99 percentile

simpledust, 300 training batches, 40 PCAs, 200 pca training batches: ~5% accuracy for 99 percentile

[kgb0255]

I've uploaded the collection of dPCA plots here.

[changhoonhahn]

PCAs in intervals

Since we've been struggling to train speculator for the entire wavelength range (2305 - 11025), let's try to divide it into three wavelength ranges and train a NN separately for each one.

Here are the wavelength ranges I've settled on:

1. 2305 < wave < 4500
2. 4500 < wave < 6500
3. 6500 < wave < 11025 These were chosen based on the accuracy level at different wavelengths (e.g. we struggle to get good accuracy at < 4500) and so that each bin has similar dimensions.

I've added two notebooks to train and validate speculator in separate wavelength bins:

• training_desi_complexdust_speculator_wavebins.ipynb: for training the speculator model for a specific wavelength bin
• validate_trained_desi_complexdust_speculator_wavebins.ipynb: for validating the separate wavelength bin speculator model

[changhoonhahn]

Not sure why I only included wave > 3000 in the validation plot in the past, but we definitely speculator to reach ~1% accuracy over the entire 2305 - 11025 wavelength. The lower and higher wavelengths, make things worse...

[changhoonhahn]

Turns out the situation is not so dire. Here's an updated validation plots with a larger test set (1e5 test samples): The dip we see above is due to numerical precision issues.

Our mean fractional error isn't at <1% yet, but at 1.5% so we're almost there!

*these plots were generated using the new validation script because the RAM available on colab couldn't handle the new test set.

[changhoonhahn]

<1% achieved!

Finally, we've achieved <1% accuracy for wavebin0. Using Npca=50, Ntrain=5e6, and a [256, 256, 256, 256] architecture did the trick.

<1% for 99 percentile.

For consistency, I'm running wavebin1 and wavebin2 with the same architecture.

[changhoonhahn]

Ntrain=5,000,000, Npca=[50, 30, 30], architecture=4x256

[changhoonhahn]

Ha! I mixed up log base 10 with natural log, so we actually do way better...