alan-turing-institute / misinformation-crawler

Web crawler to collect snapshots of articles to web archive
MIT License
5 stars 2 forks source link

Azure dependencies of the crawler #360

Open dongpng opened 4 years ago

dongpng commented 4 years ago

I suspect most people won't use an Azure database (especially when they only want to download the annotated articles). However, when trying to run the crawler but having the articles saved in a file (instead of in an Azure database) I still ran into dependencies issues regarding Azure (see #359). Can we make it so that the Azure dependencies are only needed when the crawler is actually going to use the Azure back end?