Improve compatibility with more default headers?

Hey @lmammino 👋 hope things are going well!

In our experience with using tall against a large number of domains, some domains will refuse to send back a response without a certain set of headers.

Some of these headers could be set automatically by tall as default options to improve out-of-the-box compatibility (there are libraries to keep up to date with latest Accepts and User-Agents values):

const accepts = [
  // Chrome 112
  'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
  // Firefox 113
  'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
];

const userAgents = [
  // Chrome 112
  'Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Mobile Safari/537.36',
  // Firefox 113
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/113.0'
]

tall(url, {
  headers: {
    'Accept-Encoding': 'gzip, deflate, br',
    Connection: 'keep-alive',
    Accept: accepts[Math.floor(Math.random() * (accepts.length - 1))],
    'User-Agent':
      userAgents[Math.floor(Math.random() * (userAgents.length - 1))],
  },
});

What do you think about setting these as default headers options?

lmammino / tall

Improve compatibility with more default headers? #59