Scrape online articles for the demo

spacy-pl / utils

Utilities for development polish language support for Spacy.

6 stars 1 forks source link

Scrape online articles for the demo #47

Closed MateuszOlko closed 5 years ago

MateuszOlko commented 5 years ago

DOD:

[x] Prepare script for scrapping
[x] Articles saved as artefact in dvc

MateuszOlko commented 5 years ago

@Gizzio articles are saved in data/scrapping/articles.json I didn't set up dvc pipeline, because rerunning script takes 30 mins. Also I was afraid of being banned by the sites for too many downloads. Is it possible to setup pipeline without running script? PR: #58