Open virophagesp opened 4 months ago
What documentation is my checklist referring to?
this is currently a basic setup for multi-threading the scanning of comments using only 2 CPU cores I plan to add for loops for core scaling later The reason why is because If multiple additional threads are made, large amounts of data shared between them will cause lag, and they share alot of data If this is merged then this will be a major problem to fix
@ThioJoe I think multithreading is done, what do you think?
found a bug that causes it to freeze, I will reopen when its fixed and tested for other bugs
i also found out i forgot to implement shared memory which might be the cause this might take a while
i implemented shared memory but now there is way too much ram usage and a unrelated bug i am currentlly fixing
i fixed the memory leak but there is still the issue of freezing on certain videos after 1 thread finishes
I am reopening this since it is mostly fixed I also found the freezing issue to be uncommon
I found a variety of more videos with the freezing issue I have found it to be caused by certain comments with replies not taking in the pageToken input and causing an infinite loop What separates them from most comments with replies that dont cause this issue is unclear I plan to investigate more this week and hopefully fix it
I found out that it is for any comment with more than 100 replies But also it isnt caused by my changes at all
@ThioJoe this is ready now
Can you explain why there's 1 reference to YOUTUBE[1]
It is only for multi threading, I tried using it like it was before but errors were caused so unfortunately there needs to be 2 of them, maybe if other things get multithreaded it will have other uses
Related Issue/Addition to code
Old-Multi-Threading-Attempt
Type of change
Proposed Changes
Why is this change needed?
It's needed because it speeds up scanning alot ### Additional Info - the threading module does not use multiple cpu cores, which is why the old attempt was too slow ### Checklist: - [x] My code follows the style guidelines of this project and I have read [CONTRIBUTING.md](/CONTRIBUTING.md) - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] Any dependent changes have been merged and published in downstream modules - [x] I have checked my code and corrected any misspellings