Guidance for deployment to new tasks?

[merlynmarc]

Hello,

I'd like to apply DS-Agent to tasks beyond the 18 that were included in the paper.

Are there any scripts or guidance for deploying to other Kaggle competitions? For example: https://www.kaggle.com/c/seizure-prediction/overview

Thank you, -marc

[guosyjlu]

Hi marc,

Please refer to FAQ (Q2) in README. I just updated the guidance there.

[merlynmarc]

Perfect, thanks Siyuan!

[merlynmarc]

Hi Siyuan,

How do I actually submit DS-Agent's submissions to the Kaggle leaderboard?

For example, I see the airline-reviews test.csv has the answers in it, which its submission.py uses to return the score.

Are these not actually being submitted to Kaggle? (Is this the case for the results in the paper?)

Thanks, -marc

[guosyjlu]

Yes, the implementation of this paper utilizes offline benchmark, which means the predicted results are evaluated offline with the predefined evaluation metric and ground-truth labels. If you want to utilize realistic feedback of Kaggle leaderboard scores, you can customize the submission.py via Kaggle APIs.

[merlynmarc]

OK, thanks Siyuan.

[merlynmarc]

Apologies. One more question: Where do you get the ground-truth labels? If I understand correctly, these are hidden by the competition organizers. Or are you reporting results on the validation set?

Thank you, -marc

[guosyjlu]

We perform offline evaluation throughout the paper, which means the dataset is split into training, validation and testing set by ourselves. For Kaggle competitions that only release the training set, we futher split the training set into new training set, validation set and testing set.