Fine-tuning arcface

[abdikaiym01]

Can you help with the question of how to fine-tune a model that has already been trained on 5M faces, and now it has to be trained again on 100k faces? It may be necessary to use some technique like knowledge distillation?

[abdikaiym01]

For example, how to overcome the problem like catastrophic forgetting?

[leondgarse]

If I understand correctly, you wanna fine-tune a model using a different dataset, right? Basically, you can use the basic_model for this fine-tune, and train a new NormDense header on this new dataset, like:

from tensorflow import keras
import losses, train, models, myCallbacks

data_path = '{dataset_path}'
eval_paths = ['{eval_path}']

basic_model = "{path_to_your_previous_basic_model}.h5"
tt = train.Train(data_path, save_path='keras_test.h5', eval_paths=eval_paths,
                basic_model=basic_model, batch_size=512, random_status=0,
                lr_base=0.025, lr_decay=0.5, lr_decay_steps=16, lr_min=1e-5)

""" Train a new NormDense header by freeze the basic_model using `"bottleneckOnly": True` """
optimizer = "adam"
tt.lr_scheduler = myCallbacks.CosineLrScheduler(lr_base=0.025, first_restart_step=3)
tt.train({"loss": losses.ArcfaceLoss(scale=64), "epoch": 4, "optimizer": optimizer, "bottleneckOnly": True}, 0)

""" Train on the new dataset """
import tensorflow_addons as tfa
optimizer = tfa.optimizers.SGDW(learning_rate=0.1, momentum=0.9, weight_decay=5e-5)
tt.lr_scheduler = myCallbacks.CosineLrScheduler(lr_base=0.025, first_restart_step=16)
tt.train({"loss": losses.ArcfaceLoss(scale=64), "epoch": 17, "optimizer": optimizer}, 0)

Pick lr_base / optimizer / epoch / random_status and others on your own needs.