search

RoguedBear / codezinger-due-date-scraper

this will probably get my account banned.
GNU General Public License v3.0
2 stars 0 forks source link

program crashes somewhere when the browser is launched and chromium starts to consume very high cpu #6

Closed RoguedBear closed 2 years ago

RoguedBear commented 2 years ago

orphan chromium processes still remain active and consume cpu as if the browser isn't closed.

workaround for now:

RoguedBear commented 2 years ago

update: when this happens, the bot doesn't post anything on discord. possible source of error could be these lines since: https://github.com/RoguedBear/codezinger-due-date-scraper/blob/7401be2e0db262c5137badf2e560efa05ae89d89/scrap_data.py#L173-L176

since they're not under try/except

relevant log files: image red is the last log and since when the chromium high cpu usage is started green is when i manually restarted the container

RoguedBear commented 2 years ago

processed running when cpu usage is high: image

Last log message: image

RoguedBear commented 2 years ago

possible workarounds:

RoguedBear commented 2 years ago

exact LOC where the freezing happens (confirmed via logs): https://github.com/RoguedBear/codezinger-due-date-scraper/blob/7401be2e0db262c5137badf2e560efa05ae89d89/scrap_data.py#L176

Logs when the program freezes:

crond: USER root pid   9 cmd /entrypoint.sh
codezinger-due-date-scraper  Copyright (C) 2022  RoguedBear, Ya-s-h
    This program comes with ABSOLUTELY NO WARRANTY; see COPYING
    This is free software, and you are welcome to redistribute it
    under certain conditions; see COPYING
loaded short names
Starting browser...
browser started

Logs when the program runs normally

crond: USER root pid   9 cmd /entrypoint.sh
codezinger-due-date-scraper  Copyright (C) 2022  RoguedBear, Ya-s-h
    This program comes with ABSOLUTELY NO WARRANTY; see COPYING
    This is free software, and you are welcome to redistribute it
    under certain conditions; see COPYING
loaded short names
Starting browser...
browser started
logging in.
Logged in. allegedly
sorting
loading more
Processing stuff
[<pyppeteer.element_handle.ElementHandle object at 0x7f1d26000640>, <pyppeteer.element_handle.ElementHandle object at 0x7f1d26001030>]
2
Scraped 2 questions(50%)
awaiting browser close
Browser closed
Processing new events...
Sent 0 new webhook messages
Finished, exiting...
RoguedBear commented 2 years ago

update: unable to reproduce the problem when running current version and previous version repeatedly, but --disable-gpu does help in drastically reducing CPU usage.

graph: image

RoguedBear commented 2 years ago

running the program for 2 weeks, commit https://github.com/RoguedBear/codezinger-due-date-scraper/commit/a64681c2b01c03d78f1711d9987c3053e9fe66d3 does seem to have fixed the issue