search

Futurewei-io / blue-marlin

Blue Marlin is a critical web infrastructure for advertising based monetization. It is a cloud platform that adds intelligence to a plain Ad System.
Apache License 2.0
5 stars 4 forks source link

Zero-valued ground truth are excluded in loss computing during training #269

Open jimmylao opened 2 years ago

jimmylao commented 2 years ago

@ranga Please refer to function called "calc_loss" defined in model.py lines 258 & 259 are used to exclude zero-value ground truth in loss computing during training

rangaswamymr commented 2 years ago

OK. Thanks @jimmylao . But, is it right to mask these zero values in training? specially when data is missing for more than 2 consecutive days. doesn't it give wrong sMAPE in training?

jimmylao commented 2 years ago

@rangaswamymr Good question.

  1. the assumption is that missing values contribute to very small portion of the total distribution -> excluding them will not change the distribution, i.e. missing values are treated as noise signal
  2. There's no ground truth for missing values. If we introduce some fake ground truth for missing values, it will introduce additional error.

In summary, based on 1 & 2, it is validate to exclude missing values in loss/error computing since the resultant loss/error tells the performance of true signals. However, if missing value exclusion will affect the distribution of the total population, say: systematically long period of missing values, then these missing values need to be retrieved (inferred) first before training. The training itself is not responsible for fixing missing values.