Permission denied failure on Windows

[jketema]

Hi,

I'm trying to get cvise running on Windows, but LinesPass::0 fails with the below error. Any idea what is going on? I have full permissions to read from and write to c:\\Users\\xxx\\Desktop\\help

Traceback (most recent call last):
  File "c:\Users\xxx\Desktop\cvise\tools\cvise\bin\cvise", line 484, in <module>
    reducer.reduce(pass_group, skip_initial=args.skip_initial_passes)
  File "C:\Users/xxx/Desktop/cvise/tools/cvise/share\cvise\cvise.py", line 163, in reduce
    self._run_additional_passes(pass_group['first'])
  File "C:\Users/xxx/Desktop/cvise/tools/cvise/share\cvise\cvise.py", line 186, in _run_additional_passes
    self.test_manager.run_pass(p)
  File "C:\Users/xxx/Desktop/cvise/tools/cvise/share\cvise\utils\testing.py", line 543, in run_pass
    self.state = self.current_pass.new(self.current_test_case, self.check_sanity)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users/xxx/Desktop/cvise/tools/cvise/share\cvise\passes\lines.py", line 56, in new
    self.__format(test_case, check_sanity)
  File "C:\Users/xxx/Desktop/cvise/tools/cvise/share\cvise\passes\lines.py", line 35, in __format
    shutil.copy(test_case, backup.name)
  File "C:\Users\xxx\AppData\Local\Programs\Python\Python312\Lib\shutil.py", line 423, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\Users\xxx\AppData\Local\Programs\Python\Python312\Lib\shutil.py", line 262, in copyfile
    with open(dst, 'wb') as fdst:
         ^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: 'c:\\Users\\xxx\\Desktop\\help\\tmpu3ys457d'

[marxin]

Hello.

It seems to me as an unusual temporary directory (expected would be C:\Users\USERNAME\AppData\Local\Temp). Can you please check the usual one?

Unfortunately, I don't have a Windows machine I can test it on.

[jketema]

It seems to me as an unusual temporary directory

Yeah, now you point that out, that is odd. c:\\Users\\xxx\\Desktop\\help is the directory in which the source file and script calling the compiler live. %TEMP% and %TMP% do point to C:\Users\xxx\AppData\Local\Temp. I'll have a look to see why it's not picking up on that.

[jketema]

Hi again,

It seems that Windows 11 doesn't like a file to be opened twice. The following fails with the same error:

import tempfile
import shutil

with tempfile.NamedTemporaryFile(mode='w+') as x:
     shutil.copy(some_file_that_exists, x.name)

In addition temp directory is not being used due to the following line: https://github.com/marxin/cvise/blob/00bdd8c1f9824002b10ac5fecd31dff6c5bef671/cvise/passes/lines.py#L16

[marxin]

Interesting! Anyway, it's something we can probably address with the new API provided by Python 3.12: https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile where we can set delete_on_close=False and then call tmp_file.close() in the with block.

Any chance you could test the suggested change on your system, please?

[jketema]

Yes, that works if I close both tmp_file after the flush and backup sometime before shutil.copy(test_case, backup.name).

[jketema]

I also need to make similar changes to the clang and clex passes, otherwise those passes hang.

[marxin]

Great, then let me prepare a proper PR during the weekend that would address that.

[jketema]

Thanks for your help. Here's what I did:

diff --git a/cvise/passes/clang.py b/cvise/passes/clang.py
index 1247c88..1e714f1 100644
--- a/cvise/passes/clang.py
+++ b/cvise/passes/clang.py
@@ -21,7 +21,7 @@ class ClangPass(AbstractPass):

     def transform(self, test_case, state, process_event_notifier):
         tmp = os.path.dirname(test_case)
-        with tempfile.NamedTemporaryFile(mode='w', delete=False, dir=tmp) as tmp_file:
+        with tempfile.NamedTemporaryFile(mode='w', delete=False, dir=tmp, delete_on_close=False) as tmp_file:
             args = [
                 self.external_programs['clang_delta'],
                 f'--transformation={self.arg}',
@@ -36,9 +36,11 @@ class ClangPass(AbstractPass):
             stdout, _stderr, returncode = process_event_notifier.run_process(cmd)
             if returncode == 0:
                 tmp_file.write(stdout)
+                tmp_file.close()
                 shutil.move(tmp_file.name, test_case)
                 return (PassResult.OK, state)
             else:
+                tmp_file.close()
                 os.unlink(tmp_file.name)
                 if returncode == 255 or returncode == 1:
                     return (PassResult.STOP, state)
diff --git a/cvise/passes/clex.py b/cvise/passes/clex.py
index 64b9000..352fc52 100644
--- a/cvise/passes/clex.py
+++ b/cvise/passes/clex.py
@@ -20,14 +20,16 @@ class ClexPass(AbstractPass):

     def transform(self, test_case, state, process_event_notifier):
         tmp = os.path.dirname(test_case)
-        with tempfile.NamedTemporaryFile(mode='w', delete=False, dir=tmp) as tmp_file:
+        with tempfile.NamedTemporaryFile(mode='w', delete=False, dir=tmp, delete_on_close=False) as tmp_file:
             cmd = [self.external_programs['clex'], str(self.arg), str(state), test_case]
             stdout, _stderr, returncode = process_event_notifier.run_process(cmd)
             if returncode == 51:
                 tmp_file.write(stdout)
+                tmp_file.close()
                 shutil.move(tmp_file.name, test_case)
                 return (PassResult.OK, state)
             else:
+                tmp_file.close()
                 os.unlink(tmp_file.name)
                 return (
                     PassResult.STOP if returncode == 71 else PassResult.ERROR,
diff --git a/cvise/passes/lines.py b/cvise/passes/lines.py
index 2eb0d48..6c67c59 100644
--- a/cvise/passes/lines.py
+++ b/cvise/passes/lines.py
@@ -15,9 +15,10 @@ class LinesPass(AbstractPass):
     def __format(self, test_case, check_sanity):
         tmp = os.path.dirname(test_case)

-        with tempfile.NamedTemporaryFile(mode='w+', dir=tmp) as backup, tempfile.NamedTemporaryFile(
-            mode='w+', dir=tmp
+        with tempfile.NamedTemporaryFile(mode='w+', dir=tmp, delete_on_close=False) as backup, tempfile.NamedTemporaryFile(
+            mode='w+', dir=tmp, delete_on_close=False
         ) as tmp_file:
+            backup.close()
             with open(test_case) as in_file:
                 try:
                     cmd = [self.external_programs['topformflat'], self.arg]
@@ -29,6 +30,7 @@ class LinesPass(AbstractPass):
                 if not line.isspace():
                     tmp_file.write(line)
             tmp_file.flush()
+            tmp_file.close()

             # we need to check that sanity check is still fine
             if check_sanity:

[jketema]

There's also a similar problem in the clangbinarysearch pass, which I had missed initially, as I ran with a test case that was already reduced.

[marxin]

Can you please test the pull request I've just created?

[jketema]

On your PR I'm seeing it hanging in ClangPass::remove-unused-function that probably means there's still some file that cannot be accessed. Before any of mine or your changes I also saw this with ClangBinarySearchPass, but that works with your PR, so there must be some subtle difference between the passes. It's not clear to me what though, the code seems pretty much identical.

[marxin]

Thanks for the testing effort! Yeah, the passes seem very much the same. Can you please terminate the program and show me the corresponding back-traces?

[jketema]

The backtrace I'm getting is:

Exception ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\managers.py", line 873, in _decref
    conn = _Client(token.address, authkey=authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 507, in Client
    answer_challenge(c, authkey)
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 935, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 215, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jketema\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 304, in _recv_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt:
00:00:28 INFO Exiting now ...

[jketema]

Note: eventually cvise does continue after printing:

00:05:42 WARNING ClangPass::remove-unused-function has encountered a non fatal bug: pass got stuck

[marxin]

Can you please add more debugging output to the pass:

diff --git a/cvise/passes/clang.py b/cvise/passes/clang.py
index 990184d..c6cbc4b 100644
--- a/cvise/passes/clang.py
+++ b/cvise/passes/clang.py
@@ -33,7 +33,8 @@ class ClangPass(AbstractPass):

             logging.debug(' '.join(cmd))

-            stdout, _, returncode = process_event_notifier.run_process(cmd)
+            stdout, stderr, returncode = process_event_notifier.run_process(cmd)
+            print(stdout[:64], stderr, returncode)
             if returncode == 0:
                 tmp_file.write(stdout)
                 tmp_file.close()

and test it?

[jketema]

The output is:

Error: The counter value exceeded the number of transformation i  1

Where print(cmd) gives:

['C:/Users/xxx/Desktop/cvise/tools/cvise2/libexec\\cvise\\clang_delta.EXE', '--transformation=remove-unused-function', '--counter=505', 'C:\\Users\\xxx\\AppData\\Local\\Temp\\cvise-ClangPass-remove-unused-function-leyd0zb3\\cvise-yp5abkhu\\test.cpp']

And test.cpp contains:

template <class a, class...> bool b = __is_nothrow_constructible(a);
                                        class c;
                                        class B {
                                     template <typename... d> B() noexcept(b<c, d...>);
                                   };
                                        class e : B {
                                  };

[marxin]

Error: The counter value exceeded the number of transformation i  1

Yeah, that's expected output, but I don't see why should it become stuck. Please attach a full reduction log: (with --debug option).

[jketema]

There's very little log data:

...
00:00:15 INFO ===< ClangBinarySearchPass::remove-unused-function >===
00:00:15 DEBUG available transformation opportunities for c++98: 1, took: 0.02 s
00:00:15 DEBUG available transformation opportunities for c++11: 1, took: 0.03 s
00:00:15 DEBUG available transformation opportunities for c++14: 1, took: 0.02 s
00:00:15 DEBUG available transformation opportunities for c++17: 1, took: 0.02 s
00:00:15 DEBUG available transformation opportunities for c++20: 1, took: 0.02 s
00:00:15 DEBUG available transformation opportunities for c++2b: 1, took: 0.02 s
00:00:15 INFO using C++ standard: c++2b with 1 transformation opportunities
00:00:16 DEBUG Creating pass root folder: C:\Users\xxx\AppData\Local\Temp\cvise-ClangPass-remove-unused-function-0sj3kcmo
00:00:16 INFO ===< ClangPass::remove-unused-function >===
00:06:38 WARNING ClangPass::remove-unused-function has encountered a non fatal bug: pass got stuck
00:06:38 DEBUG Please consider tarring up cvise_bug_2 and creating an issue at https://github.com/marxin/cvise/issues and we will try to fix the bug.
00:06:51 DEBUG Creating pass root folder: C:\Users\xx\AppData\Local\Temp\cvise-BalancedPass-curly-ze7mdy62
00:06:51 INFO ===< BalancedPass::curly >===
...

[jketema]

The problem goes away when I close the file in the location where the unlink was at the beginning of the else branch in the clang pass. And similarly for the clex pass.

[marxin]

Oh, got catch!

Apparently, it's something that produces an exception: https://docs.python.org/3/library/os.html#os.remove

On Windows, attempting to remove a file that is in use causes an exception to be raised; on Unix, the directory entry is removed but the storage allocated to the file is not made available until the original file is no longer in use.

Please try the latest version of the #139.

[jketema]

The latest version works for me!

[marxin]

Great, thanks for testing!

[jketema]

Thanks for fixing!