We should be able to detect urls that are repeated/similar (e.g. blog posts) and offer to the user to limit these to a certain number. Either with enquirer or with a flag.
e.g.
$ backstop-crawl url --dedupe
$ backstop-crawl url --dedupe=5
$ backstop-crawl url
We detected multiple similar urls: /blog/[...]/
Would you like to limit these? y
How many? 3
danreeves
commented
7 years ago
We should be able to detect urls that are repeated/similar (e.g. blog posts) and offer to the user to limit these to a certain number. Either with enquirer or with a flag.
e.g.