Is there any benchmark?

[sweetice]

I want to contribute to this repository with new RL algorithm. However, there is no benchmark in this repository and the AAAI2019 paper.

Could you release some benchmarks ?

[eyounx]

Thanks for your question.

VirtualTaobao itself is a replacement of dataset benchmarks. A recommendation model can be trained in either the supervised learning way or the reinforcement learning way in it, as demonstrated in the ReadMe.

Note that any benchmark was collected under a specified recommendation model, which cannot show the real performance of other models. Consequently, there is no benchmark that can verify the learned model from the VirtualTaobao environment. The only way to verify the model is to test it in the real Taobao environment. As we reported in the AAAI 2019 paper, the online test of a restricted model from VirtualTaobao achieves around 2% improvement from the supervised-learning based model.

Due to the privacy protection, we cannot release the full-size VirtualTaobao trained with full features and full data, but instead, a small one from selected features. This treatment is the same as the public dataset benchmarks. However, we believe this small VirtualTaobao can reveal useful properties of the real Taobao as an interactive environment, beyond any dataset benchmark.