Parallel fabricate on windows fork-bombs.

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Set 'parallel_ok=True' with any fabricate script.

What is the expected output? What do you see instead?
1. Parallel goodness.

What version of the product are you using? On what operating system?
1. Tip of tree.

Please provide any additional information below.
main(parallel_ok=True, jobs=1) begins creating the same command over-and-over, 
never stopping.

Original issue reported on code.google.com by jaros...@gmail.com on 7 Jan 2013 at 11:20

[GoogleCodeExporter]

I can't reproduce this with the following script:

import fabricate
def build():
    fabricate.run('dir', '/w', shell=True)
fabricate.main(parallel_ok=True, jobs=1)

Can you attach (a minimal version) of the script you're using to reproduce this?

Original comment by benh...@gmail.com on 8 Jan 2013 at 1:37

[GoogleCodeExporter]

As Lex Trotman noted on fabricate-users, parallel jobs only works with strace 
runner, which is not available on Windows. This is automatically detected in 
the code, however -- line 921 of fabricate.py checks that the system supports 
strace:

        self.parallel_ok = parallel_ok and is_strace and _pool is not None

Original comment by benh...@gmail.com on 8 Jan 2013 at 1:38

[GoogleCodeExporter]

    from fabricate import *
    import os, shutil

    def build():
        for i in xrange(10):
            run('dir', '/w', shell=True)

    def clean():
        autoclean()

    main(parallel_ok=True, jobs=2)

Causes the fork bomb. Windows 8; Python 2.7.3 x64.

Also, have you considered the use of the SDK took, 'Tracker.exe'. You can find 
it in 

    c:\Program Files (x86)\Microsoft SDKs\Windows\v8.0A\bin\NETFX 4.0 Tools\Tracker.exe

Although it exists in other SDKs (7.0), and is redistributable. The usage of 
tracker is like this:

    tracker.exe /if <intermediate-files> /c <command-line>

This results in a set of files printed to <intermediate-files> of the form:

    xxx.read.tlog
    yyy.write.tlog

Where each line in the *.tlog files after the first line are the files accessed 
(either read or written).

Original comment by jaros...@gmail.com on 8 Jan 2013 at 10:30

[GoogleCodeExporter]

I see, explicitly specifying jobs > 1 tries to use multiprocessing even on 
Windows. Wonder why that is -- we'll look into it. Thanks.

Oh, and thanks for the pointer to Microsoft's tracker.exe -- that looks great, 
basically like strace for Windows! What do you mean by "redistributable" -- the 
copyright notice on my version says "(C) Microsoft All rights reserved".

Original comment by benh...@gmail.com on 9 Jan 2013 at 7:16

[GoogleCodeExporter]

Usually, MS executables that aren't marked as redistributable may not be 
redistributed outside of MS official distribution. Executables marked as 
redistributable can be packaged outside of official distribution.

I suppose the proper strategy would be to try to detect the existence of 
tracker.exe in SmartRunner, and not deal with distribution headaches.

I've attached a hacked-up version of a 'TrackerRunner' as a patch file. This 
still craps out on Windows with parallel jobs, though; not sure why. Works well 
enough for my simple projects, though.

Original comment by jaros...@gmail.com on 9 Jan 2013 at 7:28

[GoogleCodeExporter]

Original comment by jaros...@gmail.com on 9 Jan 2013 at 7:30

Attachments:

• fabricate.py.diff

[GoogleCodeExporter]

Very cool, thanks. To keep things separate, could you add those TrackerRunner 
patch to issue 3?

You indicated in comment 3 that tracker.exe is marked as redistributable. Where 
does it say that, or how did you know?

Original comment by benh...@gmail.com on 9 Jan 2013 at 7:38

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I don't know that if it is redistributable ... I'll try to find out tomorrow, 
and move the patch, then, as well. I'm no longer on a real computer.

Original comment by jaros...@gmail.com on 9 Jan 2013 at 7:42

[GoogleCodeExporter]

I have been working on parallel building for windows. I now have it working 
with a modified version of the Tracker patch.

I think fork issue that initially generated it issue may be down to a special 
case in windows when using the multiprocessing module. You must protect the 
entry point of the build script with this piece of code.

if __name__ == "__main__":
    freeze_support()
    main(parallel_ok=True)

I do not fully understand what this does, but it boils down to Windows no 
having an os.fork() function and the way fork is emulated on a windows system.

Anyway, look out for a git push soon with the modifications that make fabricate 
parallel on windows.

Original comment by simon.al...@gmail.com on 3 May 2013 at 11:09

• Changed state: Started