Training/Validation set movies present in Scriptbase test set

[saxenarohit]

Hi,

Thanks for organizing the shared tasks.

It seems 78 out of 216 movies in the test set for the sub-task 2 (scriptbase) are also included in the training set, with an additional 8 in the validation set. I found this by checking the test instaces file shared in the repository scriptbase_test_instances.jsonl.zip

I hope these will be removed in future instances of the shared task. I am also unsure of how much impact this has on the results of the sub-task.

Thanks

[manestay]

Hi @saxenarohit, thanks for catching this potential mistake. Tagging @jigsaw2212 to take a closer look.

[saxenarohit]

Hi @manestay @jigsaw2212, any update on this?

[jigsaw2212]

@saxenarohit @manestay just seeing this - let me take a closer look and get back