[RecSys`20] MEANTIME: Mixture of Attention Mechanisms with Multi-Temporal Embeddings for Sequential Recommendation

[russellkim]

The main goal of reading paper is not just understanding it. Try to understand the key concept, but we need to get new ideas and research directions from the paper.

Paper information:

• title : MEANTIME: Mixture of Attention Mechanisms with Multi-Temporal Embeddings for Sequential Recommendation
• authors : Cho, Sung Min, Eunhyeok Park, and Sungjoo Yoo
• venue : RecSys 2020
• pdf link : http://arxiv.org/abs/2008.08273
• github link : https://github.com/SungMinCho/MEANTIME
• Abstract Recently, self-attention based models have achieved state-of-the-art performance in sequential recommendation task. Following the custom from language processing, most of these models rely on a simple positional embedding to exploit the sequential nature of the user's history. However, there are some limitations regarding the current approaches. First, sequential recommendation is different from language processing in that timestamp information is available. Previous models have not made good use of it to extract additional contextual information. Second, using a simple embedding scheme can lead to information bottleneck since the same embedding has to represent all possible contextual biases. Third, since previous models use the same positional embedding in each attention head, they can wastefully learn overlapping patterns. To address these limitations, we propose MEANTIME (MixturE of AtteNTIon mechanisms with Multi-temporal Embeddings) which employs multiple types of temporal embeddings designed to capture various patterns from the user's behavior sequence, and an attention structure that fully leverages such diversity. Experiments on real-world data show that our proposed method outperforms current state-of-the-art sequential recommendation methods, and we provide an extensive ablation study to analyze how the model gains from the diverse positional information.

Summary: problems to address, key ideas, quick results

https://docs.google.com/document/d/1CFeDWG1MFdPsqoS8Hi2RJok2-eDaQkgC4fVC-0Ml8FE/edit?usp=sharing

Questions about the paper?

What do you like?

• They utilize fully timestamp information and use relative position concept with modified multi-head attention.
• It is interesting to use Sin/Log/Exp for representing various time patterns.
• They release well-structured source code with other baselines.

What you don't like?

How to improve?

Any new ideas?

• Redefine the Relative distance matrix.
• Devise Item embedding method with a timestamp

Reproducing results (if any)