Closed dengbin9009 closed 3 years ago
This error message is probably because your system is overloaded. See here for more information. I would recommend instead of launching task 1000 times which will internally will start the driver again and again all the time to only create a new context each time, they are also isolated. Also 10 4 threads is probably too much for your system and it runs out of resources or file handles. (50MB driver + 300-400MB for chromium) 40 = at least 18GB of memory and your CPU is then also on quite a high load.
@mxschmitt
Also 10 4 threads is probably too much for your system and it runs out of resources or file handles. (50MB driver + 300-400MB for chromium) 40 = at least 18GB of memory and your CPU is then also on quite a high load.
I have tried to create only 4 * 1 threads, the error also arises.
I would recommend instead of launching task 1000 times which will internally will start the driver again and again all the time to only create a new context each time, they are also isolated.
Every time the task ends, I will close the corresponding launch. Do I need to build only one browser and create a new context for every task? or build only one browser&context and create a new page for every task?
Having a single sync_playwright and browser per thread. Then inside of the thread you should create each time a new context, this should also speed up your execution and reduce the memory/CPU overhead.
@mxschmitt Thx, I try it, and then I will reply to the result!
@mxschmitt en... I originally added proxy to launch, and everything is normal. Now I add proxy to the context, but the following error appears.
Browser needs to be launched with the global proxy. If all contexts override the proxy, global proxy will be never used and can be any string, for example "launch({ proxy: { server: 'per-context' } })"
proxies = {"server": xxxxxxx, "bypass": "*","username": "xxxxxxx", "password": "xxxxxx"}
browser = browserType.launch(
headless=True,
proxy=proxies
)
context = browser.new_context(
locale='en-US',
proxy=proxies
)
But I saw in source code that it can be used in this way:
def new_context(
self,
*,
viewport: ViewportSize = None,
no_viewport: bool = None,
ignore_https_errors: bool = None,
...
default_browser_type: str = None,
proxy: ProxySettings = None,
....
) -> "BrowserContext":
If you want to use the proxy on the context level, when you want to have e.g. different proxy servers on each context, then you need to set an arbitrary value e.g. per-context
on the server proxy object. Its not totally clear, maybe we should remove it.
So what do I need to do to fulfill my needs: Use different agents to visit a website every time, more than 3000 times.
Can you provide an demo? Many thx!
Just pass in browsertype.launch any arbitrary proxy, it won't get used when you always specify one at context level too. e.g.
chromium.launch(proxy={"server": "foobar"})
that should then work.
Closing for now since it seems answered. I improved the error message with the proxy in an upstream PR https://github.com/microsoft/playwright/pull/6350
An error occurred after using playwright about 2500 consecutive times:
my code is :
How can i solve this issue.