Concern about consulting testing sets while training

[ahmed3amerai]

Hi, I managed to partially reproduce the paper results, and I am trying to understand the training script for DARPA TC3. I noticed the script validates each trained model with the testing dataset and deletes any model with poor performance, i.e. Recall less than 0.8 and precision less than 0.7. If my understanding is correct, why does the training script consult testing sets for validation? would it affect the model performance with unseen data? Regards,

[ahmed3amerai]

Following this issue,

• I changed the part that uses the testing set in the training script by commenting lines 253 to 261 in "train_darpatc.py" file, and only using "train_pro()" function once without validating with the testing set.
• I ran the model training three times and the average f-score was 0.6. Moreover, in one of the runs the f-score dropped to 0.003. Do you have any explanation or comments on this instability of the model? Regards,

[Chaves2021]

Hey. Have you been answered? I have the same doubt.

Thanks :)