Open MMaster opened 1 month ago
How much RAM does your system have?
Could you try to add --shm-size=2G
in the docker run
command?
Currently the VM has 8GB RAM and half of it is free. I sure can increase the shared memory size for that, but I'm not really sure how that should help. I tried it and even the very first run failed.
The browser starts successfully 1 time and then fails to start 3 times on 4 different runs (it was doing the same thing with headless browser when X was not part of the docker image).
I've fixed the issue by simply doing try except on the start call and doing max 5 retries before giving up and it works, so memory doesn't seem to be the issue here.
Ok. That's strange because I can't replicate the issue. I launched the script 5 times in a row and never had the issue:
Are you sure you are running the latest version of the script?
Yeah I am on latest version.
The related issues that you marked are actually 2 separate issues. The first one happens even if the browser starts successfully and returns the tokens - I suspect it is because the async event loop is not properly stopped, but instead sys.exit is called. The second one may be related, but the solutions there don't really apply since I don't have any zombie processes, the docker container stops after the run with no leftovers and also the corrupted user data doesn't apply since it's immutable docker image.
Anyway I noticed even on stackoverflow that some people have this issue randomly with nodriver with no reliable solution.
Ok I was able to reproduce the issue on a VM with just 2 cores and 1GB of RAM.
Ok I was able to reproduce the issue on a VM with just 2 cores and 1GB of RAM.
fyi: Yesterday it stopped happening completely on the original VM. But it happened on dedicated machine with 128 GB RAM and 10 cores / 20 threads. It's completely random for me.
Ok I have narrowed down the issue, nodriver doesn't wait enough time before giving up trying to connect chromium instance: https://github.com/ultrafunkamsterdam/nodriver/blob/main/nodriver/core/browser.py#L340-L346
I have pushed a dirty patch in the Dockerfile for waiting more time: https://github.com/iv-org/youtube-trusted-session-generator/commit/0551c92227287aaa32d3a559635dde8f7ea9b5a1#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R22. I validated that it works fine on my VM with 2vCPU and 1GB of RAM.
But I'm waiting for an official implementation, for which I have created a PR:
Hi, even with the dirty patch and also when increasing the sleep time further, I still get the same error. VM resources are not the issue, it has much more than 2vCPU and 1GB of RAM.
Did anyone encounter this issues in other settings/has fixes for them? Thanks!
When running the docker container the browser fails to start quite often throwing the following exception:
Catching the exception and trying again after 3 seconds until it succeeds seems to fix this issue.