Update

[MeMartijn]

Hi @maartenmarx ,

Hoe gaat het met je? Ik hoorde op de bacheloruitreiking dat je in het ziekenhuis was beland. Niets ernstigs hoop ik? Is alles inmiddels goedgekomen? Verder heb ik iets minder leuk nieuws: het paper is helaas afgewezen voor ICWSM 2020. Ik zal in deze thread het volledige commentaar toevoegen.

[MeMartijn]

===================Reviews for your paper follow===================

---

SPC/Associate Editor review

Meta-Review and Roadmap

This paper examines the use of pre-trained word embeddings to predict whether or
not short statements are true.

There is consensus from all the reviewers that this contribution should not be
accepted. The reviewers found shortcomings with both the setup/motivation as well
as the technical approach. The reviewers did not see sufficient technical
contributions from the techniques proposed. In particular, reviewer 3 lists a
number of related work from the NLP literature that the authors should
compare/contrast with.

---

reviewer/PC member 1 review

Overall evaluation

-1 = weak reject

Reviewer's Confidence

4 = Almost certain that my review accurately reflects the paper's merit

Reviewer's methodological expertise

4 = Expert: Substantial experience and expertise in this methodological approach

Paper summary

The authors explore state of the art techniques to classify fake news and report
combinations that provide good accuracy. They motivate the problem by noting that
a large percentage of people today consume their news via the web. They summarize
existing fake news detection approaches. They summarize deep learning literature
such as text embeddings, the different ways of pooling and padding for sentences
and classifiers.

They take the dataset from politifact that has 6 labels and short statements. They
use Flair to obtain contemporary text embeddings such as Elmo, bert, GPT2, xlnet,
GPT, and flair. They provide short descriptions of each of them. They then pad and
pool and use 5 different classifiers to examine the differences.

They identify that picking the right pooling technique with the embedding is
important

They conclude that a combination of BERT and logistic regression achieves the
highest accuracy of 52.96 % that is 4% higher than recent research on the same
dataset using traditional linguistic features

They provide their train, test and validation sets as supplementary information.

Reasons to accept

They provide a summary of fake news classification techniques from the perspective
of use of text embeddings.

Reasons to reject

Their summary is not extensive. They do not provide explanations as to why certain
techniques work better than others for the same dataset. They only use one dataset
whereas as a review work, multiple datasets could be used.

Many grammatical errors such as 
Instead of 12.791 short statements --> 12,791 short statements
runner-up method is more than 4,6%. --> runner-up method is more than 4.6%.
accuracy of 52,09% --> accuracy of 52.09%
accuracy being 51,92% --> accuracy being 51.92%

Comments for authors

As a paper to review and summarize work in classifying fake news using text mining
techniques, it would be useful to identify the differences in the various
techniques from the perspective of fake news itself and have more datasets and
compare dataset characteristics and performance of various algorithms

Originality of work

(blank)

Potential impact of results

not much

Quality of execution

(blank)

Quality of presentation

(blank)

Adequancy of citations

(blank)

Ethical concerns (if any)

(blank)

---

reviewer/PC member 2 review

Overall evaluation

-3 = strong reject

Reviewer's Confidence

3 = Fairly confident that I've adequately considered all aspects

Reviewer's methodological expertise

3 = Knowledgeable: Knowledgeable in this methodological approach

Paper summary

This paper reports an empirical study on utilizing existing pre-trained word
embeddings in predicting the truthfulness of short statements.

Reasons to accept

- NA

Reasons to reject

- Weak motivation
- No technical contribution
- No reliable qualitative findings

Comments for authors

- Weak motivation: while fake detection is a very important task, evaluating the
truthfulness of statements purely based on their content seems not a very
effective approach. Moreover, it is not clear why examining the effectiveness of
different word vectors/embedding methods in evaluating statements' truthfulness is
important problem that worth to study.
- No technical contribution
- No reliable qualitative findings: the experiments are conducted on only a single
dataset, hence the findings are not generalizable. The authors should consider to
use more diverse datasets.

Originality of work

- Limited

Potential impact of results

- NA

Quality of execution

- Limited

Quality of presentation

- Medium

Adequancy of citations

- Medium

Ethical concerns (if any)

- NA

---

reviewer/PC member 3 review

Overall evaluation

-2 = reject

Reviewer's Confidence

3 = Fairly confident that I've adequately considered all aspects

Reviewer's methodological expertise

3 = Knowledgeable: Knowledgeable in this methodological approach

Paper summary

This paper proposes an NLP classifier that employes word embeddings with logistic
regression. The model is evaluated against the "Liar" dataset.
The authors conclude that the best model is a  combination of a Transformer-based
embedding technique with logistic regression.

Reasons to accept

The problem of detecting disinformation is a timely and difficult one. Every
effort in this direction is commendable and potentially highly beneficial.

Reasons to reject

1) The paper appears to oversimplify the problem of disinformation detection.
Classifiers may work for obvious lies in fake news articles, simply by analyzing
the quality of the syntax, looking up specific keywords used for shocking or
impressing the reader, looking for vague phrasing, or even just by looking at the
types of ads displayed on the page. But this does not hold for elaborate
disinformation schemes. How does the detector work when the fake news are produced
by professional and relatively reputable news outlets, like RT?

2) The Related work section merely describes prior work but does not juxtapose it
to the proposed approach.

3) The detection efficacy evaluation of the embedding-based classifiers is
lacking. For example, there is no k-fold cross-validation of the classifier.

4) The accuracy gains achieved by the proposed approach are marginal (~4%). Given
that they are observed on a limited dataset, we do not know how representative the
results are. 53% accuracy although higher than prior approaches is still slightly
better than a coin toss and probably not very useful in practice.

Comments for authors

Nits:
This makes consumers vulnerable for the spread of misinformation or fake news. ->
vulnerable to

Originality of work

The techniques used in this paper are not sufficiently novel to warrant
publication in this conference. NLP and word embedding has been proposed or used
in the literature before. The paper does not clearly state the differences of the
authors' proposal compared to the huge body of existing work. For example,
https://dl.acm.org/citation.cfm?id=3137600
https://ieeexplore.ieee.org/document/8397049
https://arxiv.org/pdf/1708.07104.pdf
https://cs.stanford.edu/~srijan/hoax/

Potential impact of results

(blank)

Quality of execution

(blank)

Quality of presentation

The paper is well-written, although the evaluation section is quite convoluted. It
is hard to isolate the important results.

Adequancy of citations

There are several missing citations to important prior work.

Ethical concerns (if any)

(blank)

---

=============================