The YT parser keeps crashing due to the huge amount of memory used by jsdom when virtualizing the document for the given entry. Replace it with linkedom seems to solve the issue.
Changelog
fix(shared): replace 'jsdom' with 'linkedom' to prevent the parser from crashing for memory allocation failure
Test Plan
Ensure to at least 50 collected htmls in your db to be sure to stress the parser enough.
docker-compose up -d mongodb
yarn pm2 start platforms/yttrex/backend/ecosystem.dev.config.js # the yt:parser:env is started with max memory of 2GB
yarn yt:backend reset-htmls-processed #reset the htmls.processed to `false` and update the `savingTime`
You can also debug this with vscode by spinning up the parserv:debug process and then attaching the debugger from the proper tab.
Summary
The YT parser keeps crashing due to the huge amount of memory used by
jsdomwhen virtualizing thedocumentfor the given entry. Replace it withlinkedomseems to solve the issue.Changelog
fix(shared): replace 'jsdom' with 'linkedom' to prevent the parser from crashing for memory allocation failure
Test Plan
Ensure to at least 50 collected htmls in your db to be sure to stress the parser enough.
You can also debug this with vscode by spinning up the
parserv:debugprocess and then attaching the debugger from the proper tab.