search

NREL / nfp

Keras layers for end-to-end learning with rdkit and pymatgen
Other
57 stars 28 forks source link

training not smooth #6

Closed jgmeyerucsd closed 2 years ago

jgmeyerucsd commented 5 years ago

I'm training a regression task that is very noisy in loss with my network tacked onto the end of the message passing part. The same training has very smooth loss with standard 1024-bit extended connectivity fingerprint and dense network with a couple hidden layers.

In your paper you describe transfer learning to apply this network to related problems where less data is available. Can you please share that model and weights? Alternatively, is there a way to freeze the early layers after they learn a chemical representation and then tune the bottom dense layers? any other ideas about how to make the loss more smooth? loss

pstjohn commented 5 years ago

Can you please share that model and weights?

Sure, but I'm not sure that would help with arbitrary chemical regression problems. We show in that paper that the transfer learning performance tends to correlate with how well the two problems are related. But those models are in this repo: https://github.com/pstjohn/opv_ml_models

is there a way to freeze the early layers

Yup, this would use the standard Keras syntax of freezing layers: https://keras.io/getting-started/faq/#how-can-i-freeze-keras-layers

any other ideas about how to make the loss more smooth?

How large is your training / validation set? This could be a data set size issue, where your validation set is too small, and thus the loss is fairly noisy. You could also try using dropout or a smaller network to see if that improves things. Also make sure that the predictions for the training set are actually working, otherwise you might have miss-specified something along the way.