Can we transfer learning?

[hori1537]

Can we use a pre-trained model or do transfer learning?

[corochann]

I think yes, if you can write advanced application code. What do you want to do more concretely?

[hori1537]

I can train a simple model about solubility with reference to train_own_dataset.py and saved the model to pickle.

Then, how do I try to train another model, for example, tox, lipophilicity, or whatever, with transfer learning from a model about solubility?

I know the way of transfer learning with Keras, freeze, but not familiar with chainer.

[corochann]

It just depends on how you want to run transfer learning.

One example is following:

0. Train with base model (solubility)
1. Instantiate and load pre-trained model --> get predictor.
2. Define new predictor with new_predictor = GraphConvPredictor(predictor.conv, MLP(out_dim=class_num, hidden_dim=n_unit)) It keeps the Graph Conv layer to be same with pretrained model, but uses new MLP layer for new label prediction.
3. Train with new predictor

[jckkvs]

Thanks! I will try it.

[hori1537]

You said 1. get predictor

However. following train_own_dataset.py, I got the pickle of regressor not predictor.

def parse_arguments():
    ./// omission
    return parser.parse_args()

def main():
    ./// omission
    # Set up the predictor.
    predictor = set_up_predictor(
        args.method, args.unit_num,
        args.conv_layers, class_num, label_scaler=scaler)
    metrics_fun = {'mae': F.mean_absolute_error, 'rmse': rmse}
    regressor = Regressor(predictor, lossfun=F.mean_squared_error,
                          metrics_fun=metrics_fun, device=device)
     run_train(regressor,  .///)

    regressor.save_pickle(model_path, protocol=args.protocol)

if __name__ == '__main__':
    main()

As also, in predict_own_dataset.py, we load the pickle of regressor.

regressor = Regressor.load_pickle(model_path, device=device)

How can I get the predictor ?

[corochann]

You can get by predictor = regressor.predictor

Another way is to save predictor in the first pre-training step using serializer, like serializers.save_npz('predictor.npz', predictor) Please refer chainer document for details.

[hori1537]

Great thanks.

As you said, I tried that

predictor = regressor.predictor
new_predictor = GraphConvPredictor(predictor.conv, MLP(out_dim=class_num, hidden_dim=n_unit))

However,

AttributeError: 'GraphConvPredictor' object has no attribute 'conv'

occured.

So I changed from predictor.conv to predictor.

predictor = regressor.predictor
new_predictor = GraphConvPredictor(predictor, MLP(out_dim=class_num, hidden_dim=n_unit))

I don' know why, but it works well.

[corochann]

https://github.com/pfnet-research/chainer-chemistry/blob/master/chainer_chemistry/models/prediction/graph_conv_predictor.py#L33

Please refer code. You can use predictor.graph_conv to extract only graph convolution network part.

[hori1537]

I understood why my last code was not good (GraphConvPredictor(predictor,...)

I changed the code and will evaluate the performance w/o pre-trained model.

new_predictor = GraphConvPredictor(predictor.graph_conv, MLP(out_dim=class_num, hidden_dim=n_unit))

Thanks.

[corochann]

Seems ok! Can we close the issue then?