memorious.logic.http calls normalizer.normalize_url that drops query arguments with no value from URLs.
Specifying normalize_url: False in the YAML configuration file only prevents the first application of normalize_url in ContextHttp.request(), but does not prevent its second application in ContextHttpResponse.url().
This breaks crawling on a number of websites.
pudo
commented
4 years ago
memorious.logic.httpcallsnormalizer.normalize_urlthat drops query arguments with no value from URLs. Specifyingnormalize_url: Falsein the YAML configuration file only prevents the first application ofnormalize_urlinContextHttp.request(), but does not prevent its second application inContextHttpResponse.url(). This breaks crawling on a number of websites.