search

jcpeterson / openwebtext

Open clone of OpenAI's unreleased WebText dataset scraper. This version uses pushshift.io files instead of the API for speed.
GNU General Public License v3.0
711 stars 79 forks source link

Filtering extracted results #23

Open Jack000 opened 4 years ago

Jack000 commented 4 years ago

this is more of a question than an issue - I noticed that in my scrape there is a large number of spurious results like:

Sorry, we just need to make sure you're not a robot. For best results, please make sure your browser is accepting cookies.

The actual openwebtext corpus seems pretty clean, so I'm wondering what if any heuristics were used to remove these pages, in order to reproduce the openwebtext corpus?

The corpus page mentions post-filtering using fasttext - is this something that will be added to this project at some point?

Finally, the readme implies that bs4 would be a better extractor than newspaper - is that the case? It's not an option in extract_text.py so it's difficult to compare.

Jack000 commented 4 years ago

actually, I just realized the openwebtext corpus is owned by a completely different group of people. Am I correct in assuming that this is an independent implementation of webtext, and has nothing to do with https://skylion007.github.io/OpenWebTextCorpus/ ?

jcpeterson commented 4 years ago

Yep, different implementation.