[Task] Add regression tests for session-based models - Githubissues

NVIDIA-Merlin / models

Merlin Models is a collection of deep learning recommender system model reference implementations

https://nvidia-merlin.github.io/models/main/index.html

Apache License 2.0

262 stars 50 forks source link

[Task] Add regression tests for session-based models #1047

Open bschifferer opened 1 year ago

bschifferer commented 1 year ago

Description

We want to add a CI test, that collects metrics to understand if performance changes between releases.

First step:

Add regression tests for session based transformer model https://github.com/NVIDIA-Merlin/models/blob/main/examples/usecases/transformers-next-item-prediction.ipynb

@radekosmulski shared ideas on it here: https://nvidia.slack.com/archives/CVBDJUPEZ/p1679617586380369

We will collect:

Runtime NVTabular
Runtime Training Model with TensorFlow
NDCG Performance of Session-based model

I close other tickets related to it:

https://github.com/NVIDIA-Merlin/Merlin/issues/223 : I think that is solved based on sample PR)
https://github.com/NVIDIA-Merlin/Merlin/issues/233 : We changed priority to session-based models