Open jetnet opened 5 years ago
I'm not alone :) #596
One solution could be to define two crawlers, one with the URL normalization to always use www
and the other without the www
. Then you would have to test each start URL to figure out under which one they belong.
If you do not know up front all the domains that will be crawled, it could get tricky for sure. We could make this a feature request, but I am not sure what solution could be generic enough. Especially knowing that www
could technically be a subdomain that serves totally different content (even if I have never encountered such a site).
Maybe we could have a smart URL Normalizer where you can indicate your preference (www
or not) and upon seeing a domain for the first time, if it does not suit your preference, it will first test if its alternate version exists before actually doing the normalization (making an extra call). I guess this could work as long as we can assume doing that test once per domain is valid for all URLs on that domain. An example:
www
www
).Could an (optional) feature like that do it you think?
maybe we could simplify the logic like following: I'd add two new options:
<startURLs includeWWW=[true|false] ...
: works as includeSubdomains="true"
where subDomain
is www
only
<startURLs includeSubdomains=[true|false|www]
: I guess, it's self explanatory :)<startURLs includeParentWWW=[true|false] ...
: allow indexing the parent domain, when the current domain starts with www
<startURLs includeParentDomains=[levelN|www] ...
: wherelevelN
: domain level to allow indexing, if the crawler gets redirected to parent domains. Up to top-level (level = 1)www
: allow redirects to the single parent domain, when the current domain starts with www
I assume, it would be not that easy to implement the options includeParentDomains
, when allowing up to top-level domain. I'd be happy with includeParentDomains=[true|false|www]
, when it'd allow the single parent domain only (any or when the current one starts w/ www
).
What do you think? Thanks a lot!
Plenty of good ideas. I just marked this as a feature request.
we need to crawl many Internet sites and encountered an issue with
www
prefix: some sites redirect to their domains withoutwww
, some other way round. Unfortunately, such case cannot be handle by NC in general way (globally): we can normalize URLs bei removingwww
prefix, and, if a site would redicrect towww.some.site
again, the collector would follow, as it is configured to follow sub-domains. But, there will be cases, when a site is available withwww
prefix only (e.g. https://www.pony.at/ does not work withoutwww
), so we will miss such sites again. So, I'm looking for a general solution for that problem. Any ideas - very welcome! Thank you!Common requirements for a crawler: