Fix run_isolated.py error message in case of zombie child process outlives the test, on Windows on success

[GoogleCodeExporter]

Example run where a zombie child process outlives the test process

--- CUT HERE ---
SUCCESS: all tests passed.
 INFO    run_isolated(253): rmtree(E:\b\build\slave\Win7_Tests__2_\isolate-2014-07-08nhc_fn)
Failed to delete E:\b\build\slave\Win7_Tests__2_\isolate-2014-07-08nhc_fn. 
Maybe the test has subprocess outliving it. Sleep 2 seconds.
Failed to delete E:\b\build\slave\Win7_Tests__2_\isolate-2014-07-08nhc_fn. 
Maybe the test has subprocess outliving it. Sleep 4 seconds.
Failed to delete E:\b\build\slave\Win7_Tests__2_\isolate-2014-07-08nhc_fn. 
Maybe the test has subprocess outliving it. Sleep 6 seconds.
Traceback (most recent call last):
  File "E:\b\build\slave\Win7_Tests__2_\build\src\tools\swarming_client\isolate.py", line 1521, in <module>
    sys.exit(main(sys.argv[1:]))
  File "E:\b\build\slave\Win7_Tests__2_\build\src\tools\swarming_client\isolate.py", line 1514, in main
    return dispatcher.execute(OptionParserIsolate(version=__version__), argv)
  File "E:\b\build\slave\Win7_Tests__2_\build\src\tools\swarming_client\third_party\depot_tools\subcommand.py", line 242, in execute
    return command(parser, args[1:])
  File "E:\b\build\slave\Win7_Tests__2_\build\src\tools\swarming_client\isolate.py", line 1268, in CMDrun
    run_isolated.rmtree(outdir)
  File "E:\b\build\slave\Win7_Tests__2_\build\src\tools\swarming_client\run_isolated.py", line 257, in rmtree
    shutil.rmtree(root)
  File "E:\b\depot_tools\python276_bin\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "E:\b\depot_tools\python276_bin\lib\shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "E:\b\depot_tools\python276_bin\lib\shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "E:\b\depot_tools\python276_bin\lib\shutil.py", line 250, in rmtree
    os.remove(fullname)
WindowsError: [Error 5] Access is denied: 
u'E:\\b\\build\\slave\\Win7_Tests__2_\\isolate-2014-07-08nhc_fn\\out\\Release\\n
acl64.exe'
--- CUT HERE ---

Instead, it should try to:
- Continue deleting as much as possible to get the exact list of locked files.
- Print out the whole list of files locked.
- Print out an error message that a child process outlive the parent process.
- (Bonus point) Track the culprit processes.

Original issue reported on code.google.com by maruel@chromium.org on 8 Jul 2014 at 12:06

[GoogleCodeExporter]

Original comment by maruel@chromium.org on 8 Jul 2014 at 12:09

[GoogleCodeExporter]

From https://code.google.com/p/chromium/issues/detail?id=392108

Can the isolate script use Handle command line program 
(http://technet.microsoft.com/en-us/sysinternals/bb896655.aspx) to find the 
process that's holding on to it? Since this is a NaCl file, it's a browser test 
so the script might also want to print the command line flags of all the parent 
processes as well (since the parent one will likely be the one with the test 
name as a gtest_filter argument)

Original comment by jam@chromium.org on 21 Jul 2014 at 6:32

[GoogleCodeExporter]

No need to use it, a CL is already on-going; 
https://codereview.appspot.com/112420047/

Original comment by maruel@chromium.org on 21 Jul 2014 at 6:41

[GoogleCodeExporter]

The crux was implemented in af9d837484984559f58d4f774e345ef0093c955a which 
fixes this bug. The rest is to be implemented in issue 120.

Original comment by maruel@chromium.org on 6 Aug 2014 at 3:19

• Changed state: Fixed

[GoogleCodeExporter]

This is still happening, i.e. see 
https://chromium-swarm.appspot.com/user/task/14861c2a9e83901

c:\users\chrome~2\appdata\local\temp\scoped_dir5084_14580\D5B2BC~1\Default\Cache
\data_0 - Access is denied.
c:\users\chrome~2\appdata\local\temp\scoped_dir5084_14580\D5B2BC~1\Default\Cache
\data_1 - Access is denied.
c:\users\chrome~2\appdata\local\temp\scoped_dir5084_14580\D5B2BC~1\Default\Cache
\data_2 - Access is denied.
c:\users\chrome~2\appdata\local\temp\scoped_dir5084_14580\D5B2BC~1\Default\Cache
\data_3 - Access is denied.
c:\users\chrome~2\appdata\local\temp\scoped_dir5084_14580\D5B2BC~1\Default\Cache
\index - Access is denied.

Original comment by jam@chromium.org on 11 Sep 2014 at 12:12

[GoogleCodeExporter]

Original comment by jam@chromium.org on 11 Sep 2014 at 12:42

• Changed state: Accepted

[GoogleCodeExporter]

This was improved in 41036ec833891e4cf59050e4ac0c5d4986e48a75 by John.

Original comment by maruel@chromium.org on 2 Dec 2014 at 7:52

• Changed state: Fixed