Komodo Edit 11.0 - Multiprocessing Print to command output fails to print correctly and in time.

[Zebrafish007]

Short Summary

The printing standard output to command output fails for several substeps during a run of the the below script. This makes it harder to follow what goes on in realtime during tracking tasks for a kernel script during multiprocessing.

Steps to Reproduce

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os, sys, platform, shutil, glob
import time, multiprocessing

def mp_worker((inputs, the_time)):
    print " Process %s\tWaiting %s seconds" % (inputs, the_time)
    time.sleep(int(the_time))
    print " Process %s\tDONE" % inputs

def mp_handler():                           # Non tandem pair processing
    p = multiprocessing.Pool(2)
    p.map(mp_worker, data)

def mp_handler_tandem():
    subdata = zip(data[0::2], data[1::2])
    for task1, task2 in subdata:
        p = multiprocessing.Pool(2)
        p.map(mp_worker, (task1, task2))

data = (['a', '2'], ['b', '3'], ['c', '1'], ['d', '4'],
        ['e', '1'], ['f', '2'], ['g', '3'], ['h', '4'])

if __name__ == '__main__':

    print 'mp_handler():'
    mp_handler()

    time.sleep(2)

    print '\nmp_handler_tandem():'
    mp_handler_tandem()
    print '---'
    time.sleep(2)

Expected results

1. mp_handler():
2. Process a Waiting 2 seconds
3. Process b Waiting 3 seconds
4. Process a DONE
5. Process c Waiting 1 seconds
6. Process c DONE
7. Process d Waiting 4 seconds
8. Process b DONE
9. Process e Waiting 1 seconds
10. Process e DONE
11. Process f Waiting 2 seconds
12. Process f DONE
13. Process g Waiting 3 seconds
14. Process d DONE
15. Process h Waiting 4 seconds
16. Process g DONE
17. Process h DONE
19. mp_handler_tandem():
20. Process a Waiting 2 seconds
21. Process b Waiting 3 seconds
22. Process a DONE
23. Process b DONE
24. Process c Waiting 1 seconds
25. Process d Waiting 4 seconds
26. Process c DONE
27. Process d DONE
28. Process e Waiting 1 seconds
29. Process f Waiting 2 seconds
30. Process e DONE
31. Process f DONE
32. Process g Waiting 3 seconds
33. Process h Waiting 4 seconds
34. Process g DONE

35. Process h DONE

Actual results

Edit 11.0:

1. Process h Waiting 4 seconds
2. Process h DONE
3. Process g Waiting 3 seconds
4. Process g DONE
5. Process f Waiting 2 seconds
6. Process f DONE
7. Process d Waiting 4 seconds
8. Process d DONE
9. Process b Waiting 3 seconds
10. Process b DONE
11. Process d Waiting 4 seconds
12. Process d DONE
13. Process h Waiting 4 seconds
14. Process h DONE
15. mp_handler():

16. mp_handler_tandem():

Edit 10.2.3:

1. Process h Waiting 4 seconds
2. Process h DONE
3. Process g Waiting 3 seconds
4. Process g DONE
5. Process f Waiting 2 seconds
6. Process f DONE
7. mp_handler():
9. mp_handler_tandem():

---

Platform Information

Komodo Edit or IDE? Komodo Edit Komodo Version? 10.2.3 build 17708 and 11.0 build 18119 Operating System (and version)? Windows 10 (10.0.15063)

Python 2.7.13 and/ or 3.6.

Additional Information

Last known startpoint in error log when I ran this code:

Edit 11.0: [2017-11-10 13:59:37,970] pystderr.log

[mitchell-as]

Hi, multi-threading is hard, especially with input and output. You need to call sys.stdout.flush() after your multi-threaded print statements. Then you will get the same output as you would if you ran invoked python [FILENAME] directly.

[Zebrafish007]

Hi Mitchell,

Cheers. wasn't aware something like that could be done. Clearly not a bug.