2019 - arXiv - Time2Vec: Learning a Vector Representation of Time

[VaghehDashti]

Title: Time2Vec: Learning a Vector Representation of Time Year: 2019 Venue: arXiv

Introduction: Time is an important feature in many applications involving events that occur synchronously and/or asynchronously. The input for problems involving time can be considered as a sequence where, rather than being identically and independently distributed (iid), there exists a dependence across time (and/or space) among the data points.

Main problem: How to create a problem/model agnostic vector representation of time?

Example:

• Predicting daily sales for a company based on the date (and other available features)
• Predicting the song a person is interested in listening to based on their listening history
• Predicting the optimum team for a given project based on the candidate’s working history

Input: Data points with timestamps

Output: A vector representation of time

Motivation: A general-purpose model-agnostic representation for time that can be potentially used in any architecture

Previous works and their gaps:

• Some of the previous works use RNNs, assuming that inputs are synchronous, but in practice they fail at effectively capturing time as a feature
• Some of the previous works either directly use time as a feature or use hand-crafted features of time that requires domain expertise
• Some other works propose specific architectures for their problem, e.g. a time gate for RNNs, and therefore fail to generalize
• RNN based approaches seem to fail for long sequences as well

Contribution of this paper: Time2vec’s output improves the model performance on several datasets and tasks. Their output vector also has 3 important properties: Periodicity, Invariance to time scaling, and simplicity

Proposed Method: Their approach is related to time decomposition techniques that encode a temporal signal into a set of frequencies. However, instead of using a fixed set of frequencies as in Fourier transforms, they allow the frequencies to be learned. Their model decomposes time into sine functions and a linear function for interpolation and extrapolation.

Experiments: They conducted experiments on a synthetic dataset and 5 real-world datasets, showing a boost in performance in most cases, and for the other cases it didn’t have an impact on the performance of the models

Code: NA

Gaps of this work: They experimented with removing the linear function (non-periodic) to see the impact, which showed that having both periodic and non-periodic activation functions will improve performance, however, they didn’t conduct experiments on a fully non-periodic representation of time for datasets/problems that are not periodic, but time is still an important factor, for example, in a recommender system, a user may never repeat the same behaviour in a periodic manner.

[hosseinfani]

@VaghehDashti I believe this is the complete and accepted work: https://github.com/BorealisAI/DE-SimplE

[hosseinfani]

@VaghehDashti Also, Representation Learning for Dynamic Graphs: A Survey worth reading.

[hosseinfani]

The sequence can be either synchronous, i.e. sampled at regular intervals, or asynchronous, i.e. sampled at different points in time.