I'm scraping various pages using x-ray using a simple script to get the title. However, I get an 400 Error Bad Request (or sometimes 413) from many different domains. These errors often come with the message Request Header Or Cookie Too Large or similar.
Your environment
version of node: * v9.6.1
version of npm: * 5.7.1
Steps to reproduce
Run the following function on many links from the same website. I'm experiencing this with anything from small domains like www.theecologist.org to big domains like www.google.com.
Subject of the issue
I'm scraping various pages using x-ray using a simple script to get the title. However, I get an 400 Error Bad Request (or sometimes 413) from many different domains. These errors often come with the message
Request Header Or Cookie Too Large
or similar.Your environment
Steps to reproduce
Run the following function on many links from the same website. I'm experiencing this with anything from small domains like
www.theecologist.org
to big domains likewww.google.com
.Expected behaviour
The script should return the title.
Actual behaviour
Instead of the title, the script returns error 400 and 413's.