sjdirect / abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Apache License 2.0
2.23k stars 555 forks source link

Documentation out of date and demo project doesn't compile #209

Closed nzbart closed 4 years ago

nzbart commented 4 years ago

We're upgrading from Abot 1.x to 2., and I'm trying to see how the new 2.x version should work. I tried copying code from the Readme where ShouldCrawlPage is called on PoliteWebCrawler, but it doesn't compile for me because ShouldCrawlPage is inaccessible:

var crawler = new PoliteWebCrawler();

crawler.ShouldCrawlPage((pageToCrawl, crawlContext) => 
{
    var decision = new CrawlDecision{ Allow = true };
    if(pageToCrawl.Uri.Authority == "google.com")
        return new CrawlDecision{ Allow = false, Reason = "Dont want to crawl google pages" };

    return decision;
});

I also tried copying code from the demo project, but it doesn't appear to have been upgraded for Abot 2.x and doesn't compile out of the box. Updating this project to 2.x would be helpful because I could then learn from the changes that you make during the upgrade.

sjdirect commented 4 years ago

There is a project called Abot2.Demo that exercises Abot2 lib code and the Abot.Demo exercises the older version. ShouldCrawlPage has been renamed to ShouldCrawlPageDecisionMaker in Abot2.

sjdirect commented 4 years ago

I'll update the docs, i can see why that is confusing. Thanks for reporting.

sjdirect commented 4 years ago

Updated docs and added warning at the top of Abot.Demo.Program.cs pointing out that demo exercises old code and the newer code is exercised in the Abot2.Demo project.