Open tao opened 3 years ago
I tried this with --workers 4
on my 10 year old quad-core Debian machine with 4 gigs of RAM. I was able to generate 4218 pages in ~20 mins without this error... so it's probably just my MacBook Air isn't suitable for such a large site.
I'll keep experimenting with it but the information may be useful if you want to add a warning or try / catch for this kind of error.
Sadly, the same error can still happen. I am ran into it locally on a MacBook Pro after approx. 450 pages (4 workers).
But I am quite surprised about your performance (4218 pages in 20 min). What kind of structure do those pages have? Complex/simple, with/without images.
I was not able to try-catch the error. It just broke the worker processes. Did you manage to do so?
Maybe related: https://github.com/spatie/fork/issues/22
Too much payload to deserialize here: https://github.com/statamic/ssg/blob/HEAD/src/Generator.php#L290
If you have only 2 workers, the closure will consist of half of the pages. If you increase the number of workers, the payload will decrease.
I suspect that it is a conceptual bug (wrong library).
Edit: On the other hand, the return value might not be supposed to carry other data. I was storing sitemap information here. The error disappeared after moving the sitemap logic out of the closure.
Thanks for the research. That's sounding like it could be the problem.
We could potentially send lists of references to entries/terms/etc to the fork closures instead of the objects themselves, then re-fetch the items within the closures.
I'm bumping into this a lot... has anyone started anything to solve it?
After generating about ~3800 pages I receive this error:
I got this with
php please ssg:generate --workers=2
first and decided to try a classicphp please ssg:generate
but I received the same error too. After this latest update: if you don't specify the workers but still have spatie/fork code installed does it still call the fork code but runs it on 1 thread by default?During the async call with two workers, it didn't crash immediately but a little while later this happened:
I can try uninstall spatie/fork code and test the site generation again if that helps, and I'll see if I can try this on another machine with more ram and report back.