li3cmz
commented
3 years ago Thank you for your attention to our work!
For Q1, we did have evaluated the scoring model in terms of the discrimination (positive vs negative) accuracy. Actually, both the discrimination accuracy and ranking loss can be the criterion for selecting a well-performing scoring model, and we use the margin ranking loss on the validation dataset as the criterion to select the best model at last.
For Q2, our training loss first rose up, which was always smaller than the margin value, and then nearly fall to 0. And it fluctuates slightly at the same time.
mindojune
Hello! First, let me thank you for the paper and the code! The paper is cogent and inspiring and got me to work on a project that applies the idea in a different domain. I have some questions regarding the training process of the GRADE model using the margin ranking loss. For your information, I'm using my own implementation of the GRADE model minus the graph reasoning module.
(Q1) Have you evaluated the scoring model in terms of the discrimination (positive vs negative) accuracy? When I'm training my model, I'm exactly not sure what kind of performance in this task leads to a well-performing scoring model.
(Q2) In your experience, what does your loss curve look like? I noticed that during my training, the training loss collapses to the margin value, and I've read that this is because the model is defaulting to output highly similar scores for positive and negative examples, resulting in the loss value = margin. I've tried different things to counter this, but I'm not even sure if this is expected or not, so I was hoping to hear more from you about the dynamics of the training process if possible.
Thank you again for your contribution and hard work!