No dependencies and simple architecture

eklem / browsercrawler

Crawling content from a site within the browser. A basis for i.e. a search solution for static sites.

https://eklem.github.io/browsercrawler/doc/

MIT License

2 stars 0 forks source link

No dependencies and simple architecture #54

Open eklem opened 5 years ago

eklem commented 5 years ago

Change the actual crawler to:

only crawl a url
Return content + URL's based on some configuration (i.e. selection within this tag)

Nothing more. Leave the rest to crawler logic and search-index-housekeeper?

eklem commented 4 years ago

{
  "dependencies": {
    "brfs": "^2.0.2",
    "cheerio": "1.0.0-rc.3",
    "search-index": "0.15.0"
  }
}

brfsI don't think is needed. cheerio is available throughm daq-proc, and search-index is only one option for this crawler, so should not be a dependency. search-index-housekeeper or nowsearch.xyz is a better place to also depend on search-index.