GenericMetadataFetcher & "HEAD - Method Not Allowed"

[jetnet]

we crawl many sites using the same configuration templates and configured the GenericMetadataFetcher globaly for all sites. Some sites do not allow the HEAD request, and the crawler stops at the very beginning, e.g.:

www.erneuerbare-energien.de: 2020-01-11 12:59:17 INFO -       REJECTED_BAD_STATUS: https://www.erneuerbare-energien.de/ (HttpFetchResponse [crawlState=BAD_STATUS, statusCode=405, reasonPhrase=Method Not Allowed])

Could the crawler be configured in a way to ignore such errors and continue to crawl further? Thank you!

[essiembre]

Can you share your config? I cannot reproduce as I have a Connection reset error instead.

[jetnet]

here's the config w/o the importer stuff, but the issue is reproducible. Thanks!

[essiembre]

I get Not Found when trying to download your config.

[jetnet]

it seems transfer.sh is having some issues at the moment. please try another one: file.io/Uz01ZR

[LeMoussel]

@jetnet , Out of curiosity, how do you use these variables?

set($domain = "www.erneuerbare-energien.de")

set($domain_xn = "www.erneuerbare-energien.de")

because I can't find any other reference of these 2 variables in this configuration file.

[essiembre]

@jetnet still a 404. Can you simply attach it to this ticket?

[jetnet]

@essiembre, attaching is too simple :) let's try another link: 0x0.st/isqT.xml, BTW - the first link above is working now. @LeMoussel, I haven't included the import section into the the configuration as it's not relevant. Those variables are used for replacement during importing, as Norconex cannot handle domains with non ascii characters, e.g. www.müller.de. But it's another issue.

[essiembre]

I was finally able to reproduce. I added a new feature on the metadata fetcher called "skipOnBadStatus". It will skip http-HEAD related activities upon failure and continue, as opposed to rejecting the documenty (issues a warning instead).

Config sample:

  <metadataFetcher 
        class="com.norconex.collector.http.fetch.impl.GenericMetadataFetcher" 
        skipOnBadStatus="true" />

This new configuration option is in the latest HTTP Collector snapshot just released. Please confirm it works for you.

[jetnet]

Thank you very much! It works as expected :)