search

wikimedia / html-metadata

MetaData html scraper and parser for Node.js (supports Promises and callback style)
MIT License
165 stars 44 forks source link

using cache to store metadata for frequently used urls #75

Closed sido420 closed 6 years ago

sido420 commented 6 years ago

Thanks for this module.

I wanted to suggest adding an optional caching layer built into the module to make it more suitable for such use cases.

This would also help avoid hitting the target URL multiple items.

mvolz commented 6 years ago

Thanks for the suggestion!

If you want to do this yourself I would suggest using the methods directly (i.e. require('html-metadata').parseAll, rather than the scrape = require('html-metadata') function in the docs) and then building the cache layer yourself.

The default exported method is intended more as a convenience/quick start sort of method, the real goal of the library is just to provide parsing of the metadata, not necessarily dealing with the requesting and caching side of things. Or at least, this is how we use it!