search

mgnoonan / RssFeeder

1 stars 0 forks source link

Filters are not respected when parsing embedded Urls in the FeedBuilder #216

Closed mgnoonan closed 2 years ago

mgnoonan commented 2 years ago

When parsing embedded Urls (typically on populist.press), the crawler will still attempt to hit sites that are in the Filter exclusion list. This slows down the overall crawling time and provides no value.

mgnoonan commented 2 years ago

We no longer crawl the embedded URLs, so closing this.