CheshireCaat / playwright-with-fingerprints

Anonymous automation via playwright with fingerprint replacement technology.
MIT License
133 stars 9 forks source link

Error: Can't send data because WebSocket is not opened #38

Open Wizzzz opened 5 months ago

Wizzzz commented 5 months ago

Hello, I'm opening this issue because I have a problem with playwright-with-fingerprint. After a few uses, I get an error when generating a browser and I have to stop my nodejs and restart it.

I've been trying to fix the error myself for a while now, but it's impossible, so I'm relying on you.

Here's the error in question: Error: Can't send data because WebSocket is not opened. at exports.throwIf (C:\Users\Administrator\Desktop\project\node_modules\websocket-as-promised\src\utils.js:4:11) at WebSocketAsPromised.send (C:\Users\Administrator\Desktop\project\node_modules\websocket-as-promised\src\index.js:252:5) at SocketService.send (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\services\socket.js:89:14) at BasRemoteClient._send (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:223:25) at BasRemoteClient.send (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:254:17) at BasRemoteClient._startThread (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:275:10) at C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:190:12 at new Promise (<anonymous>) at BasRemoteClient.runFunction (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:189:21) at C:\Users\Administrator\Desktop\project\node_modules\browser-with-fingerprints\src\plugin\connector\index.js:31:16

To solve the problem, I have to close my program and restart it.

I've noticed that this error occurs if I have other programs that use playwright-with-fingerprint, but if I don't have any other nodes that use the plugin, I never get the error.

Here's my class for managing browsers:

` import { FetchOptions } from "browser-with-fingerprints"; import fs from "fs"; import path from "path"; import { BrowserContext } from "playwright-core"; import { plugin } from "playwright-with-fingerprints";

import { Logger } from "../../logger"; import { Proxy } from "../../proxy"; import { BrowserOptions } from "../../types/browser"; import { capsolverDefaultConfig, CapsolverOptions } from "../../types/capsolver"; import { nopechaDefaultConfig, NopechaOptions } from "../../types/nopecha"; import { sleep } from "../../utils";

type InstanceOptions = { browserOptions: BrowserOptions; capsolverOptions?: CapsolverOptions; c; nopechaOptions?: NopechaOptions; proxy?: Proxy; };

type PageOptions = { blockImageDomains?: string[]; };

export class BrowserInstance { private browser: BrowserContext | null = null; readonly instanceOptions: InstanceOptions;

constructor(instanceOptions: InstanceOptions) {
    this.instanceOptions = instanceOptions;

    if (this.instanceOptions.capsolverOptions) {
        const configPath = path.join(__dirname, "..\\..\\extensions\\capsolver\\assets\\config.json");
        const config = { ...capsolverDefaultConfig, ...this.instanceOptions.capsolverOptions };
        fs.writeFileSync(configPath, JSON.stringify(config));
    } else if (this.instanceOptions.nopechaOptions) {
        const manifestPath = path.join(__dirname, "..\\..\\extensions\\nopecha\\manifest.json");
        const config = { ...nopechaDefaultConfig, ...this.instanceOptions.nopechaOptions };
        const manifest = JSON.parse(fs.readFileSync(manifestPath, "utf8"));
        manifest.nopecha = config;
        fs.writeFileSync(manifestPath, JSON.stringify(manifest));
    }

    // ignore unhandled playwright-with-fingerprint exceptions
    if ((global as any).ignoreUncaughtException !== true) {
        Logger.error("Uncaught exceptions are not being handled");
        (global as any).ignoreUncaughtException = true;
        (global as any).browserError = 0;
        process.on("uncaughtException", (err) => {
            console.error("uncaughtException:", err);
            (global as any).browserError++;
            if ((global as any).browserError > 15) {
                Logger.error("Too many browser exceptions, exiting process");
                process.exit(1);
            }
        });
    }
}

private async startBrowser() {
    // set extensions path
    const capsolverPath = path.join(__dirname, "..\\..\\extensions\\capsolver");
    const nopechaPath = path.join(__dirname, "..\\..\\extensions\\nopecha");

    const device: FetchOptions = {
        tags: this.instanceOptions.browserOptions.tags || ["Microsoft Windows", "Chrome"],
    };
    plugin.setRequestTimeout(90000);

    // if chrome device, set last version
    if (device.tags && device.tags.includes("Chrome")) {
        const versions = await plugin.versions("extended");
        const browserVersion = versions[0]["browser_version"];
        const reducedBrowserVersion = parseInt(browserVersion.split(".")[0]);

        device.minBrowserVersion = reducedBrowserVersion;
        device.maxBrowserVersion = reducedBrowserVersion;
    }

    // fetch fingerprint (use minx/max browser version bc impossible use useBrowserVersion)
    const fingerprint = await plugin.fetch(this.instanceOptions.browserOptions.fingerprintSwitcherKey, device);
    const args = this.instanceOptions.browserOptions.args || [];
    plugin.useFingerprint(fingerprint);

    // loads captcha solvers extensions
    if (this.instanceOptions.capsolverOptions) args.push(`--load-extension=${capsolverPath}`);
    else if (this.instanceOptions.nopechaOptions) args.push(`--load-extension=${nopechaPath}`);

    // set proxy
    if (this.instanceOptions.proxy)
        plugin.useProxy(this.instanceOptions.proxy.fullAddress(), {
            detectExternalIP: true,
            changeGeolocation: true,
            changeWebRTC: true,
            changeTimezone: true,
            changeBrowserLanguage: true,
        });

    // launch browser
    this.browser = (await plugin.launchPersistentContext(this.instanceOptions.browserOptions.profilePath, {
        headless: this.instanceOptions.browserOptions.headless,
        args,
    })) as BrowserContext;
}

public async init() {
    if (this.browser) throw "Browser is already initialized";

    // blocks instance creation if one is already in progress
    let isPossibleToCreateBrowser = false;
    if (!fs.existsSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`)) fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, "0");

    do {
        const lastBrowserGeneration = parseInt(fs.readFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, "utf8"));
        if (Date.now() - lastBrowserGeneration > 90000) {
            fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, Date.now().toString());
            isPossibleToCreateBrowser = true;
        } else await sleep(1000);
    } while (!isPossibleToCreateBrowser);

    try {
        const isBrowserCreated = await Promise.race([
            sleep(90000).then(() => false),
            this.startBrowser()
                .then(() => true)
                .catch(async (error) => {
                    console.error(error);
                    (global as any).browserError++;
                    if ((global as any).browserError > 15) {
                        Logger.error("Too many browser exceptions, exiting process");
                        process.exit(1);
                    }
                    await sleep(90000);
                    return false;
                }),
        ]);

        if (!isBrowserCreated) throw "[BROWSER INSTANCE] Timeout or error during browser creation";

        if (!this.browser) throw "[BROWSER INSTANCE] Browser failed to initialize";
    } catch (error) {
        if (this.browser) await this.close();
        throw error;
    } finally {
        await sleep(10000);
        fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, "0");
    }
}

public async newPage(options: PageOptions = {}) {
    if (!this.browser) throw "[BROWSER INSTANCE] Browser is not initialized";

    const page = await this.browser.newPage();

    if (options.blockImageDomains && options.blockImageDomains.length > 0) {
        await page.route("**/*", async (route) => {
            const request = route.request();

            if (request.resourceType() === "image") {
                const url = request.url();
                for (const domain of options.blockImageDomains!) if (url.includes(domain)) return await route.abort();
            }

            return await route.continue();
        });
    }

    return page;
}

public async close() {
    if (!this.browser) throw "[BROWSER INSTANCE] Browser is not initialized";

    await this.browser.close();
}

} `

Thank advance !

CheshireCaat commented 5 months ago

Use separate work folders for each process, this can help to avoid such errors:

Wizzzz commented 3 months ago

Hello, I'm coming back to you because I've implemented what you told me, I have a folder with 15 engines and a database that allows me to make it impossible to take 2 engines simultaneously, unfortunately, I have the following problem: image

It's important to remember that this isn't multi-threading, but rather several nodes with different tasks, so it's not the same process that launches the different browsers.

Here is my code to launch a browser: ` private async startBrowser() { // set extensions path const capsolverPath = path.join(dirname, '..\..\extensions\capsolver'); const nopechaPath = path.join(dirname, '..\..\extensions\nopecha');

    // retrieve engine data
    await axios
        .get('http://localhost:4242/', { params: { duration: this.instanceOptions.taskDuration } })
        .then((response: AxiosResponse) => {
            this.workDirId = response.data.uuid;
            plugin.setWorkingFolder(response.data.path);
        })
        .catch(error => {
            Logger.error('Failed to retrieve engine data,  use default path');
        });

    // set tags
    const device: FetchOptions = {
        tags: this.instanceOptions.browserOptions.tags || ['Microsoft Windows', 'Chrome']
    };

    // if chrome device, get and set last version (use min/max browser version bc impossible use useBrowserVersion)
    if (device.tags && device.tags.includes('Chrome')) {
        const versions = await plugin.versions('extended');
        const browserVersion = versions[0]['browser_version'];
        const reducedBrowserVersion = parseInt(browserVersion.split('.')[0]);

        device.minBrowserVersion = reducedBrowserVersion;
        device.maxBrowserVersion = reducedBrowserVersion;
    }

    // if profile set, use it
    if (
        this.instanceOptions.browserOptions.profilePath &&
        fs.existsSync(this.instanceOptions.browserOptions.profilePath)
    ) {
        plugin.useProfile(this.instanceOptions.browserOptions.profilePath, {
            loadFingerprint: true,
            loadProxy: true
        });
    }

    // fetch fingerprint
    const fingerprint = await plugin.fetch(this.instanceOptions.browserOptions.fingerprintSwitcherKey, device);
    const args = this.instanceOptions.browserOptions.args || [];
    plugin.useFingerprint(fingerprint);

    // loads captcha solvers extensions
    if (this.instanceOptions.capsolverOptions) args.push(`--load-extension=${capsolverPath}`);
    else if (this.instanceOptions.nopechaOptions) args.push(`--load-extension=${nopechaPath}`);

    // set proxy
    if (this.instanceOptions.proxy)
        plugin.useProxy(this.instanceOptions.proxy.fullAddress(), {
            detectExternalIP: true,
            changeGeolocation: true,
            changeWebRTC: true,
            changeTimezone: true,
            changeBrowserLanguage: true
        });

    // launch browser
    try {
        let isLaunch = false;
        await Promise.race([
            sleep(20000).then(() => {
                if (!isLaunch) throw '[BROWSER INSTANCE] Timeout during launch';
            }),
            new Promise((resolve, reject) => {
                plugin
                    .launchPersistentContext(this.instanceOptions.browserOptions.profilePath, {
                        headless: this.instanceOptions.browserOptions.headless,
                        args
                    })
                    .then((browser: BrowserContext) => {
                        isLaunch = true;
                        this.browser = browser;
                        resolve(browser);
                    })
                    .catch((error: any) => {
                        reject(error);
                    });
            })
        ]);
    } catch (error) {
        (global as any).browserError++;
        if ((global as any).browserError > 50) {
            Logger.error('Too many browser exceptions, exiting process');
            process.exit(1);
        }
        throw error;
    }
}

public async init() {
    if (this.browser) throw 'Browser is already initialized';

    // blocks instance creation if one is already in progress
    let isPossibleToCreateBrowser = false;
    if (!fs.existsSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`))
        fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, '0');

    do {
        const lastBrowserGeneration = parseInt(
            fs.readFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, 'utf8')
        );
        if (Date.now() - lastBrowserGeneration > 90000) {
            fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, Date.now().toString());
            isPossibleToCreateBrowser = true;
        } else await sleep(1000);
    } while (!isPossibleToCreateBrowser);

    try {
        const isBrowserCreated = await Promise.race([
            sleep(90000).then(() => false),
            this.startBrowser()
                .then(() => true)
                .catch(async error => {
                    console.error(error);
                    await sleep(90000);
                    return false;
                })
        ]);

        if (!isBrowserCreated) throw '[BROWSER INSTANCE] Timeout or error during browser creation';

        if (!this.browser) throw '[BROWSER INSTANCE] Browser failed to initialize';
    } catch (error) {
        if (this.browser) await this.close();
        throw error;
    } finally {
        await sleep(10000);
        fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, '0');
    }
}

`

@CheshireCaat

CheshireCaat commented 3 months ago

@Wizzzz Greetings, try to check the following algorithm, perhaps one of the node processes has not completed and is holding a lock, which causes an error:

Just in case, I recommend that you ALWAYS close the browser using the browser.close() method after completion, because this also clears some of the overlaid locks.

If the problem does not repeat itself, then the processes can still be open when you run your code - there is a timer in the client to automatically turn off the engine after 5 minutes, you can simply force the current process to end upon completion.

Or do additional management before starting - check if there is another process that used the same folder, and if there is, then kill it.

You can also try importing a list of locks from the proper-lockfile package and try to clear them before/after starting manually.

When I have time for fixes and updates, I will try to find another package for locks inside the plugin instead of the current one, because of it there are too many problems and too often.

Anyway, I hope my advice will help you a little.

Wizzzz commented 3 months ago

@CheshireCaat After verification, all engines have a .lock file.

Will deleting the .lock file unblock the situation? Technically, with my DB, it's impossible for 2 processes to use the same engine.

So I can make sure that before starting a browser, I check the existence of the .lock file and if it exists, delete it.

If everything is well managed in my DB, there won't be any problems?

CheshireCaat commented 3 months ago

@Wizzzz you can try to remove lock file, but there is no guarantee that it complete release it for the target proccess. At least, if it will not work, you can try other advices from the my previous comment.

If everything is well managed in my DB, there won't be any problems?

In theory, taking into account the rest of the recommendations, everything should be fine.

Wizzzz commented 3 months ago

@CheshireCaat Okok I'll give it a try and let you know.

Is there a discord or a telegram group with other users of your packages? It might be cool to have one to discuss how we can implement a common solution to face its problems since we are several to have it.

Wizzzz commented 3 months ago

Is there an example of the solution you propose? Because I'm not sure I've understood 100% of the different steps. ;(