search

tnc-br / ddf-isoscapes

3 stars 0 forks source link

Xavier Initialization #184

Open erickzul opened 12 months ago

erickzul commented 12 months ago

Per https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotNormal An initializer that Draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(2 / (fan_in + fan_out)) where fan_in is the number of input units in the weight tensor and fan_out is the number of output units in the weight tensor.

Makes output of training consistent with a constant seed.

Comparison with latest model: https://colab.sandbox.google.com/drive/1iq6j2n9Su8ahl6DW08wigKgbi6RpiFvR image image

Tested training several times:

Val loss: 0.9596271514892578
Train loss: 0.7228686809539795
Test loss: 1.8758741617202759
dO18 RMSE: 2.1917561291995247

Val loss: 0.9596271514892578
Train loss: 0.7228686809539795
Test loss: 1.8758741617202759
dO18 RMSE: 2.1917561291995247

Val loss: 0.9596271514892578
Train loss: 0.7228686809539795
Test loss: 1.8758741617202759
dO18 RMSE: 2.1917561291995247
review-notebook-app[bot] commented 12 months ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

rothn commented 12 months ago

I'm confused-- an RMSE of 2 seems high for what we were getting, but from the graphs Xavier init appears to perform better as I would expect. What am I missing? @erickzul