crwlrsoft / crawler

Library for Rapid (Web) Crawler and Scraper Development
https://www.crwlr.software/packages/crawler
MIT License
312 stars 11 forks source link

HttpLoader Separation #132

Closed otsch closed 5 months ago

otsch commented 5 months ago

For being more flexible to build a separate headless browser loader (in an extension package) extract the most basic HTTP loader functionality to a new HttpBaseLoader and important functionality for the headless browser loader to a new HeadlessBrowserLoaderHelper. It's considered a fix, because there's no new functionality, just refactoring existing code for better extendability.

Also add --display-warnings option to test commands.