laurengarcia / url-metadata

NPM module: Request a url and scrape the metadata from its HTML using Node.js or the browser.
https://www.npmjs.com/package/url-metadata
MIT License
166 stars 44 forks source link

UserAgent options has is not considered in version 2.2.1. #12

Closed AshfaqMemon closed 5 years ago

AshfaqMemon commented 5 years ago

When I am passing userAgent option, it is not getting passed while requesting for the URL. Please look into this.

I am passing userAgent as below:

var options = {
    userAgent: 'my_custom_useragent'
};
const urlMetadata = require('url-metadata');
urlMetadata(url, options);

In this case my_custom_useragent is ignored.

laurengarcia commented 5 years ago

Hi there, Just looked into this and it appears to be working, your issue may be in a different layer in your code. I triple checked at 3 levels: as the userAgent option is passed into the request module from this url-metadata module, and where it is propagated out from the request module (the module that this package depends on to make the call).

I checked via consoling out the data in the command line in both modules ...and just to be sure, i also fired up WireShark locally to verify that the call was going out with the user agent set properly.

Screenshots below:

  1. from terminal showing both modules parsing userAgent correctly
  2. WireShark showing that the User-Agent header is indeed going out with the request from my local command line interface

Feel free to reopen if you have further issues or encounter the bug in some way that i was not able to reproduce.

request-options request-via-wireshark