Some questions about the form of training data organization of SCIFACT Dataset

[neverneverendup]

Dear Author,

Your work SCIFACT published on EMNLP in 2020 is very outstanding. Thank you for your work. After reading your paper and observing part of the code, I have some small questions want to consult.

In Section 5 of the paper, when training the Rationale selection model, the organization of the training data you described in the paper is: "For each claim, we use cited abstracts labeled NOINFO, as well as non-rationale sentences from abstracts labeled SUPPORTS and REFUTES as negative examples.” However, in your official code verisci/training/rationale_selection/transformer_scifact.py, I did not see the code to add the sentences in cited abstracts labeled NOINFO as negative examples to the training data.

In addition, when training the Label prediction model,the organization of the training data you described in the paper is ："For cited abstracts labeled NOINFO, we choose the k sentences from the cited abstract with highest TF-IDF similarity to the claim as input rationales. ", and I also did not find the source in your code.

I want to ask where is the correct code location, thank you!

[dwadden]

Thanks for your questions! Responses inline.

In Section 5 of the paper, when training the Rationale selection model, the organization of the training data you described in the paper is: "For each claim, we use cited abstracts labeled NOINFO, as well as non-rationale sentences from abstracts labeled SUPPORTS and REFUTES as negative examples.” However, in your official code verisci/training/rationale_selection/transformer_scifact.py, I did not see the code to add the sentences in cited abstracts labeled NOINFO as negative examples to the training data.

You're correct, this is a minor mistake in the paper. We don't use the NOINFO documents for rationale selection, we only use non-rationale sentences. Thanks for pointing this out and apologies for the confusion.

In addition, when training the Label prediction model,the organization of the training data you described in the paper is ："For cited abstracts labeled NOINFO, we choose the k sentences from the cited abstract with highest TF-IDF similarity to the claim as input rationales. ", and I also did not find the source in your code.

You're right, this is another minor mistake. We choose k random sentences here, not the top-k sentences.

Apologies for any confusion this may have caused you!