OvermindDL1
commented
5 years ago The 'crop' step still takes a fraction of a second with getting over a thousand chunks in later save files now so it might be good to just leave it without multiprocess. There's something screwy there (I'm wondering if multithread processes are being left unclosed elsewhere, they need to be closed before they are joined on or they become zombie processes that later multithread usage can use and fail on since it re-uses existing processes for that, not looked deep enough into the code to see all the usages yet).
L0laapk3
slikts
PirateIncognito
workthread.join()is still freezing on occasion, unsure what the workthread is doing or why it's freezing as nothing is being logged, it is thewaitKillworkthread, and factorio did already get killed and it did fully die (no remnants of its process is still running).I'm not entirely sure of the purpose of that workthread as it is spawned up and then immediately joined on.
When I canceled it after leaving it at that point for 20 minutes it printed this when I Ctrl+c'd it:
Interestingly I only hit Ctrl+c once, I'm guessing it's printing
KeyboardInterruptfor each subprocess. At this point, however, it was frozen again with no further output, so after waiting a while I hit Ctrl+c again and it printed this and finally died:A few interesting things to notice:
First, when Ctrl+c was hit it tried running
taskkil /im factorio.exe, which is a Windows command (the heck?! Python should not need those.)...It seems the subprocesses were all stuck waiting on messages, the only one different was:
For note
mp.cpu_count()is returning 16 (the number of cores in this system).Some testing shows that in
crop.pythis section:I put a
print(repr(files))just after thefor _ in range(len(files)):line and it was only called once, printing out:Upon which it seems to permanently freeze at
progressQueue.get(True).It only seems to be happening on 'night' as well, these is the log up to that point:
At which point it freezes. Interesting that the file list doesn't output for the day crop...
Replacing the
workfunction to justreturn Falsedoes not change anything, setting output shows that it looks like it's not even being called... interesting...Tried calling
map_asyncand evenmapin a variety of different ways and no callback executes, everything is getting locked on a semaphore and never being released.Just to get it 'working' for now, commenting out all the multiprocess stuff in crop.py (haven't run into any issues with it in zoom.py yet for whatever reason) and just calling
workdirectly in a loop onfilesand that works, only takes a second as well with about a hundred chunks (zoom is the one that takes a few seconds) and works fine.