search

sjdirect / abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Apache License 2.0
2.25k stars 560 forks source link

Handling pagination #222

Closed thebitbrine closed 4 years ago

thebitbrine commented 4 years ago

I am trying to figure out how can I make abot crawl links that have query parameters for pagination (i.e. http://example.com/articles?p=3) I can't seem to find any way to make about crawl those pages too.

sjdirect commented 4 years ago

Hi, there is a config boolean value called IsRespectUrlNamedAnchorOrHashbangEnabled. I believe that does what you are looking for.