grd349
commented
4 years ago Hi @HinLeung622
I agree with pretty much everything you say! It looks like C is doing best. It looks like (from what you have said) all three could do with training for longer. I have some suggestions but we can discuss them here if you need now.
- [ ] I suggest increasing the learning rate. Roughly speaking I increase the learning rate until it breaks because it is too high then I back it off a little. This might make your models train faster.
- [ ] I suggest running for longer.
- [ ] I suggest adding more layers ... let's not mess around, maybe run 8 layers to see what happens.
- [ ] The regularisation is probably architecture dependent. Having more neurons increases complexity so the l2 optimal value will probably be different. Try getting a mild over fit and then we can increase the regularisation.
Having said all of that - the aim here is for you to get a feel for what is going on , what helps, and what doesn't. And in that sense I feel like you are making excellent progress. How do you feel the 'understanding' is coming along?
@Harry-Westwood - how are you getting on. Does the above make any sense?
HinLeung622
@grd349 Basically, the main question of this issue is, what should I look for in the test-training results, when deciding if I should use a bigger or smaller architecture.
Context: I have been running NN fitting tests on the small grid. The following three fit results are all ran under the same NN settings, except the number of neurons in each layer: inputs = mass, age, feh, MLT (feh and MLT do not vary in small grid) outputs = luminosity, Teff, delta nu 4 hidden layers 0.0001 learning rate 0.9999 beta_1 0.999 beta_2 0.0001 l2 regularizer 50k epochs all 9211 datapoints, with batch size = 9211
NN A has 32 neurons in each dense layer, and this is the resulting HR diagram (left = NN, right = grid):
NN B has 128 neurons per layer, and HR diagram:
NN C has 256 neurons per layer, and HR diagram:
HR diagram of NN A was quite jaggered at regions where it should be straight, while the larger architecture B and C are able to smooth that part out. In general, I would say C has the most close-looking shape to the proper HR diagram at the right.
Here is a table documenting some numerical results from the three NNs:
(both losses are in MAE, run time is only approximate. The function for calculating dex was not written when model A was ran, so no data there.) Judging from just the resulting losses and dex, it seems that the more complex the architecture is, the more accurate is the result. Judging from the history plots (learning loss and validation loss vs epoch), the validation loss have been closely following the drop in learning loss in growing epochs in all three cases, suggesting against overfitting. So, does this mean that I should keep on increasing the complexity of the architecture as I have not hit the spot where it is just big enough but still smaller than to overfit? I also don't know how much I should be concerned about run time.
It should also be noted that the dex in both fit B and C (and certainly A if it were to be measured) for luminosity and delta nu are still not low enough to hit the target 0.005 dex set 2 weeks ago, but since judging from the history plots the MAE was still steadily declining (not flattened off), I believe more training can get the dex to decrease to perhaps under 0.005 dex.