unblocked-web / double-agent

A test suite of common scraper detection techniques. See how detectable your scraper stack is.
MIT License
135 stars 10 forks source link

easily run one or multiple runners #55

Closed GlenDC closed 2 years ago

GlenDC commented 2 years ago

Problem Statement

The public code base's runner can only runner secret-agent out of the box. The next public version probably will only be able to run just Hero.

There is an implementation available for puppeteer but it is more of an example and not really useable within the runner context as-is.

Proposal

For my own fork of the runner module of this codebase I implemented an interface as follows runner/interfaces/runner.ts:

import IAssignment from '@double-agent/collect-controller/interfaces/IAssignment';

interface IRunnerFactory {
    startFactory(): Promise<void>;
    spawnRunner(assignment: IAssignment): Promise<IRunner>;
    stopFactory(): Promise<void>;
}

interface IRunner {
    run(assignment: IAssignment): Promise<void>;
    stop(): Promise<void>;
}

export { IRunnerFactory, IRunner };

I refactored the puppeteer and secret-agent assignments:

Within my runAssignments i also use commandor such that I can for example provide a flag to enable one or multiple runners using the enum:

enum Runner {
  Puppeteer = 'puppeteer',
  SecretAgent = 'secret-agent',
}

I implemented this as an initial job. This way I can also easily add my own stacks and run them using the runner.

GlenDC commented 2 years ago

If you like this approach I can make a patch for this repository to make it work for you as well.

calebjclark commented 2 years ago

@GlenDC, I love it! Yes, a PR of your patch would be much appreciated.

Thanks for helping us.