Open Wizzzz opened 5 months ago
Use separate work folders for each process, this can help to avoid such errors:
Hello, I'm coming back to you because I've implemented what you told me, I have a folder with 15 engines and a database that allows me to make it impossible to take 2 engines simultaneously, unfortunately, I have the following problem:
It's important to remember that this isn't multi-threading, but rather several nodes with different tasks, so it's not the same process that launches the different browsers.
Here is my code to launch a browser: ` private async startBrowser() { // set extensions path const capsolverPath = path.join(dirname, '..\..\extensions\capsolver'); const nopechaPath = path.join(dirname, '..\..\extensions\nopecha');
// retrieve engine data
await axios
.get('http://localhost:4242/', { params: { duration: this.instanceOptions.taskDuration } })
.then((response: AxiosResponse) => {
this.workDirId = response.data.uuid;
plugin.setWorkingFolder(response.data.path);
})
.catch(error => {
Logger.error('Failed to retrieve engine data, use default path');
});
// set tags
const device: FetchOptions = {
tags: this.instanceOptions.browserOptions.tags || ['Microsoft Windows', 'Chrome']
};
// if chrome device, get and set last version (use min/max browser version bc impossible use useBrowserVersion)
if (device.tags && device.tags.includes('Chrome')) {
const versions = await plugin.versions('extended');
const browserVersion = versions[0]['browser_version'];
const reducedBrowserVersion = parseInt(browserVersion.split('.')[0]);
device.minBrowserVersion = reducedBrowserVersion;
device.maxBrowserVersion = reducedBrowserVersion;
}
// if profile set, use it
if (
this.instanceOptions.browserOptions.profilePath &&
fs.existsSync(this.instanceOptions.browserOptions.profilePath)
) {
plugin.useProfile(this.instanceOptions.browserOptions.profilePath, {
loadFingerprint: true,
loadProxy: true
});
}
// fetch fingerprint
const fingerprint = await plugin.fetch(this.instanceOptions.browserOptions.fingerprintSwitcherKey, device);
const args = this.instanceOptions.browserOptions.args || [];
plugin.useFingerprint(fingerprint);
// loads captcha solvers extensions
if (this.instanceOptions.capsolverOptions) args.push(`--load-extension=${capsolverPath}`);
else if (this.instanceOptions.nopechaOptions) args.push(`--load-extension=${nopechaPath}`);
// set proxy
if (this.instanceOptions.proxy)
plugin.useProxy(this.instanceOptions.proxy.fullAddress(), {
detectExternalIP: true,
changeGeolocation: true,
changeWebRTC: true,
changeTimezone: true,
changeBrowserLanguage: true
});
// launch browser
try {
let isLaunch = false;
await Promise.race([
sleep(20000).then(() => {
if (!isLaunch) throw '[BROWSER INSTANCE] Timeout during launch';
}),
new Promise((resolve, reject) => {
plugin
.launchPersistentContext(this.instanceOptions.browserOptions.profilePath, {
headless: this.instanceOptions.browserOptions.headless,
args
})
.then((browser: BrowserContext) => {
isLaunch = true;
this.browser = browser;
resolve(browser);
})
.catch((error: any) => {
reject(error);
});
})
]);
} catch (error) {
(global as any).browserError++;
if ((global as any).browserError > 50) {
Logger.error('Too many browser exceptions, exiting process');
process.exit(1);
}
throw error;
}
}
public async init() {
if (this.browser) throw 'Browser is already initialized';
// blocks instance creation if one is already in progress
let isPossibleToCreateBrowser = false;
if (!fs.existsSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`))
fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, '0');
do {
const lastBrowserGeneration = parseInt(
fs.readFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, 'utf8')
);
if (Date.now() - lastBrowserGeneration > 90000) {
fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, Date.now().toString());
isPossibleToCreateBrowser = true;
} else await sleep(1000);
} while (!isPossibleToCreateBrowser);
try {
const isBrowserCreated = await Promise.race([
sleep(90000).then(() => false),
this.startBrowser()
.then(() => true)
.catch(async error => {
console.error(error);
await sleep(90000);
return false;
})
]);
if (!isBrowserCreated) throw '[BROWSER INSTANCE] Timeout or error during browser creation';
if (!this.browser) throw '[BROWSER INSTANCE] Browser failed to initialize';
} catch (error) {
if (this.browser) await this.close();
throw error;
} finally {
await sleep(10000);
fs.writeFileSync(`${process.env.TMP as string}\\lastBrowserGeneration.txt`, '0');
}
}
`
@CheshireCaat
@Wizzzz Greetings, try to check the following algorithm, perhaps one of the node processes has not completed and is holding a lock, which causes an error:
Just in case, I recommend that you ALWAYS close the browser using the browser.close()
method after completion, because this also clears some of the overlaid locks.
If the problem does not repeat itself, then the processes can still be open when you run your code - there is a timer in the client to automatically turn off the engine after 5 minutes, you can simply force the current process to end upon completion.
Or do additional management before starting - check if there is another process that used the same folder, and if there is, then kill it.
You can also try importing a list of locks from the proper-lockfile
package and try to clear them before/after starting manually.
When I have time for fixes and updates, I will try to find another package for locks inside the plugin instead of the current one, because of it there are too many problems and too often.
Anyway, I hope my advice will help you a little.
@CheshireCaat After verification, all engines have a .lock file.
Will deleting the .lock file unblock the situation? Technically, with my DB, it's impossible for 2 processes to use the same engine.
So I can make sure that before starting a browser, I check the existence of the .lock file and if it exists, delete it.
If everything is well managed in my DB, there won't be any problems?
@Wizzzz you can try to remove lock file, but there is no guarantee that it complete release it for the target proccess. At least, if it will not work, you can try other advices from the my previous comment.
If everything is well managed in my DB, there won't be any problems?
In theory, taking into account the rest of the recommendations, everything should be fine.
@CheshireCaat Okok I'll give it a try and let you know.
Is there a discord or a telegram group with other users of your packages? It might be cool to have one to discuss how we can implement a common solution to face its problems since we are several to have it.
Is there an example of the solution you propose? Because I'm not sure I've understood 100% of the different steps. ;(
Hello, I'm opening this issue because I have a problem with playwright-with-fingerprint. After a few uses, I get an error when generating a browser and I have to stop my nodejs and restart it.
I've been trying to fix the error myself for a while now, but it's impossible, so I'm relying on you.
Here's the error in question:
Error: Can't send data because WebSocket is not opened. at exports.throwIf (C:\Users\Administrator\Desktop\project\node_modules\websocket-as-promised\src\utils.js:4:11) at WebSocketAsPromised.send (C:\Users\Administrator\Desktop\project\node_modules\websocket-as-promised\src\index.js:252:5) at SocketService.send (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\services\socket.js:89:14) at BasRemoteClient._send (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:223:25) at BasRemoteClient.send (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:254:17) at BasRemoteClient._startThread (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:275:10) at C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:190:12 at new Promise (<anonymous>) at BasRemoteClient.runFunction (C:\Users\Administrator\Desktop\project\node_modules\bas-remote-node\src\index.js:189:21) at C:\Users\Administrator\Desktop\project\node_modules\browser-with-fingerprints\src\plugin\connector\index.js:31:16
To solve the problem, I have to close my program and restart it.
I've noticed that this error occurs if I have other programs that use playwright-with-fingerprint, but if I don't have any other nodes that use the plugin, I never get the error.
Here's my class for managing browsers:
` import { FetchOptions } from "browser-with-fingerprints"; import fs from "fs"; import path from "path"; import { BrowserContext } from "playwright-core"; import { plugin } from "playwright-with-fingerprints";
import { Logger } from "../../logger"; import { Proxy } from "../../proxy"; import { BrowserOptions } from "../../types/browser"; import { capsolverDefaultConfig, CapsolverOptions } from "../../types/capsolver"; import { nopechaDefaultConfig, NopechaOptions } from "../../types/nopecha"; import { sleep } from "../../utils";
type InstanceOptions = { browserOptions: BrowserOptions; capsolverOptions?: CapsolverOptions; c; nopechaOptions?: NopechaOptions; proxy?: Proxy; };
type PageOptions = { blockImageDomains?: string[]; };
export class BrowserInstance { private browser: BrowserContext | null = null; readonly instanceOptions: InstanceOptions;
} `
Thank advance !