Closed shreydesai closed 3 years ago
Thanks for the feedback Shrey.
"I understand that there may be licensing issues associated with Curation releasing the documents along with the abstracts" - You're right. That is the main issue when trying to do a release like this. We've tried to copy the way the Deepmind released the CNN/DM dataset. Maybe someone will host the articles in a similar way. We would be delighted if they did!
I agree the scraper could be improved. It's something I'll keep working on. If you have any ideas for how it could be improved, or submit a pull request, I would be happy to credit you.
@HenryDashwood Does the large version of this corpus also suffer from similar issues? Given that, you are providing a commercial license for the larger dataset, is there some quality control that you have employed on the dataset to resolve some of the above listed issues?
New cleaner dataset with better scraper
Hi Curation,
I would like to point that, in its current form, this dataset is almost unusable. The main problem is with the fact that users have to scrape documents from the original websites. Website changes, URL forwarding, paywalls, etc. inevitably cause a lot of web-related errors that manifest in the documents, which is very difficult to preprocess out. The provided preprocessing script (in its current state) does not do enough to filter out the noise. There are at least two larger problems with this:
1) Pre-trained models, like BERT, are very sensitive to the integrity of the document, and if there are spurious tokens or ill-formed sentences, the model will not be able to form coherent representations of the document. Seemingly minor things like
[gallery ids=\"1318996,1318995,1318988,1318986,1319003\"]
being injected in a document can cause the number of wordpieces to explode. This generally wouldn't be a big problem, but BERT (and most other pre-trained encoders) have a maximum sequence length of 512 wordpieces/subwords, so the encoder will not have a chance to see other sentences if a single sentence saturates the batch.2) Results obtained on this dataset might not be reproducible given that the documents are not released with the original distribution. When I first used the scraper to collect this dataset (around the time when this dataset was published), I was only able to retrieve 39,917 documents, not 40k as is advertised on this repository. If someone were to run the scraper now, the recall may be substantially lower. I understand that there may be licensing issues associated with Curation releasing the documents along with the abstracts, but this is an important point for consideration -- people having different copies of this dataset (along with different content) will not be able to compare results in a scientifically meaningful way.
Finally, in the interest of providing constructive criticism, I'll point out some specific issues I saw in the documents that may be addressed by finer-grained quality control. These errors are not cherrypicked -- they are chosen from a random sample of about 100 documents/summaries.
1) HTML artifacts in the middle of a document
2) Non-document related content appears in the document
3) Tables get squeezed in with the rest of the document -- hard to preprocess out
4) Document/summary pairs are incorrect
5) Hitting a paywall
6) Spurious client-side access errors
7) Summary in English, document is not
8) Lack of sentence separation
9) Summaries are not well-formed (word separation issues)
10) Scraper does not handle errors gracefully