Some questions in the paper

[HowardZJU]

Hi, I am an engineer in Alibaba, and these days we are reproducing the MRDR-DL paper in our deployment environment, while the results are not competing. We read and discuss this paper carefully and have some questions as follows:

• The equation (13) seems to be confusing, while p<0.5 contributes to p>0.5?
• The calculation approach in Table 3 seems to be confusing too, where the recalls@4 > 1.
• Google proved that DR has achieved Cramer-Rao condition, which means that the variance of DR estimator reaches the minimal. How to further break through this theorem and reduce the estimation variance?

Overall this is an abstractive paper, we just want to make clear of the details and explore its application in industrial scenario.

[guosyjlu]

Hi, thanks for your attention on our work.

• The only difference between the MRDR estimator and DR estimator in the imputation learning is the weight of the loss of each instance, i.e., 1/p or (1-p)/p**2. Equ.13 is used to compare the size relationship of the two weights. As such, we can intuitively explain what the MRDR estimator is doing, i.e., the MRDR estimator increases the penalty of the clicked events with low propensity (those with p{u,i}<0.5), and decreases the penalty of the rest of the clicked events (those with p{u,i}>0.5).
• We use two rating-prediction-based recommendation datasets for the real-world experiments. We see ratings of 3 or 4 as positive, and 1~3 as negative. Hence, after the preprocessing, there is at least one positive item for each user in the test set. That’s why the recall@k and dcg@k can be larger than 1.
• I think it’s a bias-variance trade-off problem when using the DR estimator. In DR estimator, IPS term guarantees the unbiasedness more but introduces high variance, while EIB term reduces its variance to some degree but introduces high bias. The propensity in IPS term is unobservable; hence, accurate IPS estimation is nearly impossible. In contrast, the error is observable, but building an accurate error imputation model equals to building an accurate CVR estimation model. Our paper is under the assumption that the predicted propensity is accurate. As such, we can reduce the variance of the DR estimator by learning more accurate imputed errors for user-item pairs with high propensity (which correspondingly account for a larger part of the variance), and less accurate imputed errors for user-item pairs with low propensity by introducing different weights.

[HowardZJU]

Thanks for your answering! The solution can be got from this MRDR paper in ICML 2018 in the area of RL off-line evaluation: http://proceedings.mlr.press/v80/farajtabar18a/farajtabar18a.pdf#:~:text=The%20main%20idea%20of%20our%20estimator%2C%20called%20more,samples%20are%20randomly%20missing%20%28Cao%20et%20al.%2C%202009%29, with the same name, mathematical formula with this paper.. My mentee pointed it out and maybe cite it in Sec 3.2 formally will avoid misunderstanding.