Closed kelson42 closed 2 years ago
I don’t know what is the status here but I made a new attempt a now it dies with the followinf error
yes it's an incident bug of trying to fix this issue ; you might have noticed the recipe is on :dev
. Don't need to re-run it for now, it's being run off zimfarm with debug logs and --keep
for inspection.
OK, now that we have logs, it seems that the scraper crashes when adding one file to the ZIM ; that file being the 3.2GB version of this 12h video.
It crashes while adding it as the e-penser_fr_all_2021-01.zim.tmp
is 2.1G large.
zimwriterfs has no problem ziming that folder, creating a 15GB file.
This is not a youtube issue anymore. will pursue this on scraperlib/pylibzim. Looking at the code, it looks like poor memory mgmt on scraperlib but this part was changed with the port to libzim_next. Will test libzim_next on it.
As for the 247 code, my guess is that this is due to multithreading but I can't explain it exactly.
This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.
It passed !!! https://farm.openzim.org/pipeline/46e8c752a8094ea5cb7d5716
@rgaudin Should we close the ticket?
Yes, apparently, issue was in libzim and libzim update fixed it. There are about a 100 videos. Logs mentions 10 missing videos as those are private.
https://farm.openzim.org/pipeline/78ee12e5de4bba2ad0908ff5/debug