search

claabs / epicgames-freegames-node

Automatically login and find available free games the Epic Games Store. Sends you a prepopulated checkout link so you can complete the checkout after logging in. Supports multiple accounts, login sessions, and scheduled runs.
https://hub.docker.com/r/charlocharlie/epicgames-freegames
MIT License
1.39k stars 92 forks source link

Memory Leak #189

Closed sigboe closed 9 months ago

sigboe commented 2 years ago

Describe the bug

Memory Leak

Screenshots or Logs

I can't share anything my server hangs.

Additional context

I only have 8 Gigs of ram, I use about half while ideling My whole server hangs, but I can SSH into the server, albeit everything is very slow. The docker service is responding, but maybe it takes 10 minutes for a docker command to complete. Docker stats shows this docker container is using all the RAM of my server. docker kill epicgames-freegames fixes the problem, but takes 10 to 15 minutes to resolve

claabs commented 2 years ago

Similar to #180

Can you elaborate on your use case here? The only thing that's caused a memory leak is the process spawning a bunch of browser instances. A browser should really only spin up on first launch to log in, and then at most once per month after that.

Are you starting it up, and then not completing the captcha? Because that could cause many processes to run in parallel due to the cron jobs (which could probably be fixed).

sigboe commented 2 years ago

There was no notice of unsolved captchas, I've put a limit of 512MB ram on the docker container, so when it happens again it should make it easier to troubleshoot. Maybe 512MB is to little? Can it go over that in normal use?

My cron job I set to go once a day, at an arbitrary time different from the default config file

sigboe commented 2 years ago

Another thing to note, this happened during the week when the docker container was not able to fetch dead by daylight. So it maybe tried to fetch that game 6 times

claabs commented 2 years ago

I thought I'd note, here are some commits I made to hopefully help resolve this:

elbrodark commented 2 years ago

hi i dont know if it helps but i got the same problem it stucks at checking free games for XX@gmail.com

CONTAINER ID   NAME                  CPU %     MEM USAGE / LIMIT    MEM %     NET I/O           BLOCK I/O         PIDS
d20ef31a1855   epicgames-freegames   100.37%   1.964GiB / 2GiB      98.20%    5.26MB / 258kB    1.11GB / 53.5MB   2071
Run on startup: true
[2022-01-03 09:07:37.615 +0100] INFO : Started epicgames-freegames-node
    COMMIT_SHA: "09ca50a6eeb1847ba751281eb33feba9cf4608d7"
    BRANCH: "master"
    DISTRO: "alpine"
[2022-01-03 09:07:38.074 +0100] INFO : Checking free games for xxx1@gmail.com
[2022-01-03 09:07:45.502 +0100] INFO : Successfully refreshed login
    user: "xxx1@gmail.com"
[2022-01-03 09:07:56.210 +0100] INFO : Available free games
    user: "xxx1@gmail.com"
    availableGames: [
      "Tomb Raider GAME OF THE YEAR EDITION",
      "Shadow of the Tomb Raider: Definitive Edition",
      "Rise of the Tomb Raider: 20 Year Celebration"
    ]
[2022-01-03 09:07:57.197 +0100] INFO : Unpurchased free games
    user: "xxx1@gmail.com"
    purchasableGames: [
      "Tomb Raider GAME OF THE YEAR EDITION",
      "Shadow of the Tomb Raider: Definitive Edition",
      "Rise of the Tomb Raider: 20 Year Celebration"
    ]
[2022-01-03 09:07:57.197 +0100] INFO : Purchasing Tomb Raider GAME OF THE YEAR EDITION
[2022-01-03 09:08:38.074 +0100] INFO : Checking free games for xxx2@gmail.com
[2022-01-03 09:09:05.343 +0100] WARN  (TimeoutError): waiting for selector `button[data-cy='setAccessibilityCookie']:not([disabled])` failed: timeout 30000ms exceeded
    TimeoutError: waiting for selector `button[data-cy='setAccessibilityCookie']:not([disabled])` failed: timeout 30000ms exceeded
        at new WaitTask (/usr/app/node_modules/puppeteer/src/common/DOMWorld.ts:806:28)
        at DOMWorld.waitForSelectorInPage (/usr/app/node_modules/puppeteer/src/common/DOMWorld.ts:655:22)
        at Object.internalHandler.waitFor (/usr/app/node_modules/puppeteer/src/common/QueryHandler.ts:78:19)
        at DOMWorld.waitForSelector (/usr/app/node_modules/puppeteer/src/common/DOMWorld.ts:511:25)
        at Frame.waitForSelector (/usr/app/node_modules/puppeteer/src/common/FrameManager.ts:1274:47)
        at Page.waitForSelector (/usr/app/node_modules/puppeteer/src/common/Page.ts:3210:29)
        at getHcaptchaCookies (/usr/app/src/puppet/hcaptcha.ts:57:43)
        at runMicrotasks (<anonymous>)
        at processTicksAndRejections (internal/process/task_queues.js:95:5)
        at PuppetPurchase.purchaseShort (/usr/app/src/puppet/purchase.ts:142:29)
[2022-01-03 09:09:05.532 +0100] WARN : Setting the hCaptcha accessibility cookies encountered an error. Continuing without them...
[2022-01-03 09:09:38.077 +0100] INFO : Checking free games for xxx3@gmail.com
(node:53) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:53) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:53) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:53) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit
[2022-01-03 09:10:38.074 +0100] INFO : Checking free games for xxx4@gmail.com
[2022-01-03 09:11:38.091 +0100] INFO : Checking free games for xxx5@googlemail.com

PID   USER     TIME  COMMAND
    1 root      0:00 tini -- docker-entrypoint.sh
   10 root      0:00 {docker-entrypoi} /bin/sh /usr/local/bin/docker-entrypoint.sh
   53 root      0:23 node /usr/app/dist/src/index.js
   64 root      1:10 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  136 root      0:37 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  171 root      0:26 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  203 root      0:25 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  236 root      0:19 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  268 root      0:18 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  300 root      0:18 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  335 root      0:15 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  367 root      0:14 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  389 root      0:14 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  434 root      0:12 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  466 root      0:11 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  477 root      0:11 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  503 root      0:11 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  567 root      0:10 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  600 root      0:09 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  605 root      0:09 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  637 root      0:08 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  701 root      0:08 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  734 root      0:07 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  740 root      0:07 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  764 root      0:07 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  840 root      0:06 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  844 root      0:06 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  867 root      0:05 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  888 root      0:06 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
  933 root      0:06 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1007 root      0:05 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1019 root      0:04 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1049 root      0:04 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1082 root      0:04 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1107 root      0:04 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1184 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1215 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1226 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1230 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1263 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1363 root      0:03 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1370 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1400 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1408 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1439 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1544 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1551 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1577 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1587 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1615 root      0:02 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1723 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1737 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1760 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1766 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1790 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1909 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1915 root      0:00 /bin/sh
 1919 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1932 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1947 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 1962 root      0:01 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2105 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2144 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2151 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2152 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2160 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2295 root      0:00 /usr/lib/chromium/chrome --extra-plugin-dir=/usr/lib/nsbrowser/plugins --user-data-dir=/root/.config/chromium --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disabl
 2321 root      0:00 ps axu
claabs commented 2 years ago

It looks like you're seeing the puppeteer hang issue (#164), and triggering the fix I put in to retry launching the browser (https://github.com/claabs/epicgames-freegames-node/issues/164#issuecomment-1002649474). It retries up to 30 times for each account, so I see how you ran into all the zombie chromium processes.

A couple action items on this:

  1. If you're on not arm64, you should try the Debian image (:bullseye-slim) to see if that fixes the browser hang.
  2. I should reduce the retry count for launching the browser to something like 5.
  3. I should try to prune the zombie Chromium processes at runtime rather than waiting for tini to do it at exit.
claabs commented 2 years ago

Oh also, if you're seeing this regularly, you should probably set a memory limit on the container to prevent it from freezing your server. https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory

-m 2g should be fine

claabs commented 2 years ago

I added the Chromium process pruning and lower amount of retries in https://github.com/claabs/epicgames-freegames-node/commit/1078cf8f3868bbe1f34c62997749ec45744c1a52

Hopefully that should handle it for now.

sigboe commented 2 years ago

@claabs I haven't had any issues after going to 2gigs and purchasing at least two games, always updated on the :latest tag