Closed darkcode85 closed 7 years ago
ignore_links
/ignore_links_like
matches the full link (the String form of the URL), so your Regexp is matching against the beginning of the URL not the path. Probably something like spider.ignore_urls_like { |url| url.path.start_with?('/blog/) }
. Hope that helps.
In the documentation says that is possible to skip processing some pages, but I can not find how I can do it, I have tried with
ignore_links
orignore_pages
but nothing sames to work, eg:spider = Spidr.site('.....', ignore_links: [%{^/blog/}]) do |spider|
spider.every_html_page do |page|
//here I still get pages with the /blog url
end
end
How I can ignore some pages based in the URL?