Open Bhammin opened 2 years ago
Same in NAML model. I resolved like below.
elif metric == "logloss":
# avoid logloss nan
# preds = [max(min(p, 1.0 - 10e-12), 10e-12) for p in preds]
preds = [np.clip(p, 10e-12, 1.0 - 10e-12) for p in preds]
preds = np.concatenate(preds)
labels = np.concatenate(labels)
logloss = log_loss(np.asarray(labels), np.asarray(preds))
res["logloss"] = round(logloss, 4)
Description
When using the notebook nrms_MIND.ipynb if I try to include logloss as a metric (update nrms.yaml), I am getting a failure when training the model. Please see below image of the exact failure when calling
model.fit(train_news_file, train_behaviors_file, valid_news_file, valid_behaviors_file)
The failure happens in
cal_metric(labels, preds, metrics)
. I think the issue is that the field preds data type is a list of numpy arrays with different lengths. For example: [[.1,.2],[.1,.1,.2],[.8]].I was able to calculate logloss by making the following change to
cal_metric(labels, preds, metrics)
Since I am only playing with NRMS model at the moment, I'm not sure if this fix would work for all models.
In which platform does it happen?
Jupyter Lab running in Ubuntu 20.04.4 LTS (Focal Fossa). Using Python version 3.8.10 with tensorflow version 2.8.0.
How do we replicate the issue?
Using https://github.com/microsoft/recommenders/blob/main/examples/00_quick_start/nrms_MIND.ipynb, update the hyper parameters metric to include logloss. hparams should like this
HParams object with values {'support_quick_scoring': True, 'dropout': 0.2, 'attention_hidden_dim': 200, 'head_num': 20, 'head_dim': 20, 'filter_num': 200, 'window_size': 3, 'vert_emb_dim': 100, 'subvert_emb_dim': 100, 'gru_unit': 400, 'type': 'ini', 'user_emb_dim': 50, 'learning_rate': 0.0001, 'optimizer': 'adam', 'epochs': 3, 'batch_size': 32, 'show_step': 10, 'title_size': 30, 'his_size': 50, 'data_format': 'news', 'npratio': 4, 'metrics': ['logloss', 'group_auc', 'mean_mrr', 'ndcg@5;10'], 'word_emb_dim': 300, 'model_type': 'nrms', 'loss': 'cross_entropy_loss', 'wordEmb_file': './data/nrms/mind-small/utils/embedding.npy', 'wordDict_file': './data/nrms/mind-small/utils/word_dict.pkl', 'userDict_file': './data/nrms/mind-small/utils/uid2index.pkl'}
Expected behavior (i.e. solution)
The validation logloss should be calculated.
Other Comments
Thank you for providing quick start guides for many models!