search

veltzer / pyscrapers

project to produce various useful scrapers
https://veltzer.github.io/pyscrapers
MIT License
24 stars 7 forks source link

Make dependencies optional #1

Closed jayvdb closed 4 years ago

jayvdb commented 4 years ago

I have no interest in installing pornhub-api, nor youtube-dl to be honest. It would be nice to be able to use the other scrapers without installing libraries for all of them.

veltzer commented 4 years ago

I get scrapers from all kinds of people and integrate them. The current way it works is to have all the dependencies installed at install time. I could split the project into core/plugins. But that will require a large engineering effort and now is the wrong time to do it. In the future, when I have twenty more scrapers I will consider it because then I would have a better notion of what is core, what is a scraper plugin, and what is the correct relationship between them.

Example, youtube-dl is more than a 10 year project and has 1164 extractors and still has not split into core/plugins.

If you don't like it you are free to fork the project at an earlier version.