search

chrisakroyd / robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
MIT License
12 stars 8 forks source link

Save/Restore robots.txt cache. #2

Closed chrisakroyd closed 3 years ago

chrisakroyd commented 6 years ago

Implement saving and restoring of the parsed robots.txt file.