search

steven95421 / KDD_WinnieTheBest

KDD Cup 2020 Challenges for Modern E-Commerce Platform: Multimodalities Recall first place
189 stars 55 forks source link

How many NDCG@5 score can the single model of this method achieve on the valid set ? #1

Closed biandh closed 4 years ago

biandh commented 4 years ago

You are so good! I read the READM.md carefully, but I was still confused, mainly in the following aspects:

  1. Can you give a simple example of the negative sample sampling method?

  2. In this competition, 69 models have been trained based on Mcan and Visual Bert methods. Do these 69 models have any differences, such as parameters, training samples, etc?

  3. Before post-processing, a single model based on Mcan or visual Bert is used to evalute the NDCG@5 on valid.tsv. How much can be achieved?

  4. In the post-processing stage, the valid set is used to train the model. How to evaluate the model?

  5. After post-processing, how many Score can a single model achieve in testA?

Looking forward to your reply ^_^

steven95421 commented 4 years ago

Hi @biandh, We have updated README.md with the answers.

miziha-zp commented 4 years ago

finding that you got magic improvement by finetuning in valid, and got 0.87-0.88 from single model(For VisualBERT, it was around 0.69. As for MCAN, it was around 0.7)....by the way, how did it improve by blending your 69 models?

steven95421 commented 4 years ago

This huge performance boost is not from post-finetuning, but the post-processing part. We found out that the candidate products provided in the dataset have some bias. That is, if one item occurs in multiple candidate pools it is more likely this item is not the actual answer but just some bias.

miziha-zp commented 4 years ago

thx，amazing magic ！