apify / got-scraping

HTTP client made for scraping based on got.
422 stars 32 forks source link

Export all the hooks from the package #101

Closed foxt451 closed 10 months ago

foxt451 commented 10 months ago

I had a use-case when I wanted to extend the exported Got instance further and override just the tls hook (the last one). But doing like below will override the beforeRequest property as a whole, removing all other useful hooks (insecureParserHook, ...) that I don't want to override.

export const extendedGotScraping = gotScraping.extend({
    hooks: {
        beforeRequest: [
            myNewTlsHook,
        ],
    },
});

So I had to manually copy paste the source code of all other hooks (basically, the whole lib), and do so:

import { gotScraping } from 'got-scraping';
import { insecureParserHook } from './insecure-parser.js';
import { sessionDataHook } from './storage.js';
import { http2Hook } from './http2.js';
import { proxyHook } from './proxy.js';
import { browserHeadersHook } from './browser-headers.js';
import { tlsHook } from './clever-tls.js';

export const extendedGotScraping = gotScraping.extend({
    hooks: {
        beforeRequest: [
            insecureParserHook,
            sessionDataHook,
            http2Hook,
            proxyHook,
            browserHeadersHook,
            myNewTlsHook,
        ],
    },
});

it would be very convenient, if one could just do like this:

import { gotScraping, hooks } from 'got-scraping';
export const extendedGotScraping = gotScraping.extend({
    hooks: {
        beforeRequest: [
            insecureParserHook: hooks.insecureParserHook,
            sessionDataHook: hooks.sessionDataHook,
            ...
            myNewTlsHook,
        ],
    },
});

Or alternatively:

import { gotScraping, hooks } from 'got-scraping';
export const extendedGotScraping = gotScraping.extend({
    hooks: {
        beforeRequest: [
            ...hooks.beforeRequest.slice(0, -1)
            myNewTlsHook,
        ],
    },
});

Maybe even combine the two approaches