url-inspector

Synopsys

npx url-inspector <url>

Description

Get normalized metadata about what a URL mainly represents.

This is a Node.js module.

Sources of information:

• HTTP response headers
• embedded tags in binary formats (using exiftool)
• OpenGraph, Twitter Cards, schema.org, json+ld, title and meta tags in HTML pages
• oEmbed endpoints
• if a URL is mainly wrapping a media, that media might be inspected too

Inspection stops when enough information has been gathered, or when a maximum number of bytes (depending on media type) have been downloaded.

Format

• url: url of the inspected resource
• title: title of the resource, or filename, or last component of pathname with query
• description: optional longer description, without title in it, and only the first line.
• site: the name of the site, or the domain name
• mime: RFC 7231 mime type of the resource (defaults to Content-Type) The inspected mime type could be more accurate than the http header.
• ext: The file extension, only derived from the mime type. Safe to be used as file extension.
• what: what the resource represents page, image, video, audio, file
• type: how the resource is used: link, image, video, audio, embed. Example: if what:image and mime:text/html, and no html snippet is found, type will be 'link'.
• html: the html representation of the resource, according to type and use.
• script: url of a script that must be installed along with the html representation.
• date (YYYY-MD-DD format) creation or modification date
• author: optional credit, author (without the @ prefix and with _ replaced by spaces)
• keywords: optional array of collected keywords (lowercased words that are not in title words).
• size: optional Content-Length as integer; discarded when type is embed
• icon: optional link to the favicon of the site
• width, height: optional dimensions as integers
• duration: optional hh:mm:ss string
• thumbnail: optional a URL to a thumbnail, could be a data-uri for embedded images
• source: optional a URL that can go in a 'src' attribute; for example a resource can be an html page representing an image type. The URL of the image itself would be stored here; same thing for audio, video, embed types.
• error: optional an http error code, or string

Install

url-inspector currently requires those external libraries/tools:

• exiftool
• libcurl (and libcurl-dev if node-libcurl needs to be rebuilt)

Both programs are well-maintained, and available in most linux distributions.

Usage

import Inspector from 'url-inspector';

const opts = {
 ua: "Mozilla/5.0", // override ua, defaults to somewhat modern browser
 nofavicon: false, // disable additional requests to get a favicon
 nosource: false, // disable main embedded media sub-inspection
 file: true, // local files inspection is only enabled by default when using CLI
 meta: {} // user-entered metadata, to be merged and normalized
 providers: null // custom providers (module path or array)
};

const inspector = new Inspector(opts);

const obj = await inspector.look(url);

Inspector throws http-errors instances.

By default oembed providers are

• found from a curated list of providers
• found from a custom list, required from opts.providers
• discovered in the inspected web pages

It is possible to add custom providers in the options, by passing an array or a path to a module exporting an array.

See src/custom-oembed-providers.js for examples.

To normalize an already existing metadata object, including url rewriting done by providers, and other changes in fields, do:

await inspector.norm(obj);

url-inspector uses node-libcurl to make http requests, and exposes it as:

const req = await Inspector.get(urlObj);

where req.abort() stops the request, req.res is the response stream, and res.statusCode, res.headers are available.

Proxy support

url-inspector configures http(s) proxies through proxy-from-env package and environment variables (http_proxy, https_proxy, all_proxy, no_proxy):

Read proxy-from-env documentation.

License

Open Source, see ./LICENSE.