tempfile module misinterprets access denied error on Windows

[8e06f10c-4750-4034-a995-c3e4932e02af]

BPO	22107
Nosy	@pfmoore, @ncoghlan, @tjguk, @bobince, @takluyver, @zware, @serhiy-storchaka, @eryksun, @zooba, @RoccoMatano, @earonesty
Files	tempfile_bad_tempdir.patch tempfile_bad_tempdir_2.patch tempfile_bad_tempdir_3.patch tempfile_bad_tempdir_4.patch master...bjmcculloch_patch-1.diff: fix PermissionError exception handlers

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['3.8', '3.9', '3.10', 'performance', 'library', 'OS-windows'] title = 'tempfile module misinterprets access denied error on Windows' updated_at = user = 'https://bugs.python.org/rupole' ``` bugs.python.org fields: ```python activity = actor = 'bugale bugale' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)', 'Windows'] creation = creator = 'rupole' dependencies = [] files = ['38145', '38151', '39421', '39434', '42704'] hgrepos = [] issue_num = 22107 keywords = ['patch'] message_count = 43.0 messages = ['224304', '236004', '236028', '236029', '236047', '236050', '236052', '236053', '236055', '236058', '236060', '236112', '236118', '236130', '236132', '236133', '236143', '243514', '243608', '243611', '243617', '243626', '243628', '243787', '243788', '243805', '243808', '243812', '243814', '243815', '259559', '264779', '264971', '265040', '282403', '285131', '337735', '340738', '347076', '347095', '353353', '353360', '357075'] nosy_count = 20.0 nosy_names = ['paul.moore', 'ncoghlan', 'rupole', 'tim.golden', 'aclover', 'python-dev', 'takluyver', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'Tor.Colvin', 'rocco.matano', 'V\xc3\xa1clav Dvo\xc5\x99\xc3\xa1k', 'Billy McCulloch', 'Paul Doom', 'earonesty', 'bugale bugale', 'JDM', 'Jonathan Mills'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'resource usage' url = 'https://bugs.python.org/issue22107' versions = ['Python 3.8', 'Python 3.9', 'Python 3.10'] ```

[8e06f10c-4750-4034-a995-c3e4932e02af]

_mkstemp_inner assumes that an access denied error means that it has generated a filename that matches an existing foldername. However, in the case of a folder for which you don't have permissions to create a file, this means it will loop thru the maximum possible number of files. This causes it to hang for several seconds and eventually return a bogus FileExistsError.

Similar behaviour exists in 2.7.7, but it throws an IOError instead.

http://bugs.python.org/issue18849 seems to be where this was introduced.

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

changeset 035b61b52caa has this:-

             return (fd, _os.path.abspath(file))
         except FileExistsError:
             continue    # try again
+        except PermissionError:
+            # This exception is thrown when a directory with the chosen name
+            # already exists on windows.
+            if _os.name == 'nt':
+                continue
+            else:
+                raise

     raise FileExistsError(_errno.EEXIST,
                           "No usable temporary file name found")

Could we simply set a flag saying it's a PermissionError and then raise the appropriate PermissionError or FileExistsError at the end of the loop?

[serhiy-storchaka]

What exceptions are raised on Windows when try to open a file in bad directory?

open('foo').close() open('foo/bar') # what raised? open('nonexistent/bar') # what raised?

If raised the same exceptions as on Linux, then perhaps the following patch fixes this issue.

[serhiy-storchaka]

And what returns os.access for writable directories and non-existent files?

os.mkdir('dir')
os.access('dir', os.W_OK)          # what returns?
os.access('nonexistent', os.W_OK)  # what returns or raises?

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

>>> os.mkdir('dir')
>>> os.access('dir', os.W_OK)
True
>>> os.access('nonexistent', os.W_OK)
False
>>> open('dir/bar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'dir/bar'
>>> open('nonexistent/bar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'nonexistent/bar'

[serhiy-storchaka]

Thank you Mark. Could you please make first test in msg236028 (when first part of the path is a file, not a directory)?

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

>>> open('README').close()
>>> open('README/bar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'README/bar'

[zooba]

Is there a difference if you do open(..., 'w')? It's a different enough operation that it may have a different error.

[serhiy-storchaka]

Is there a difference if you do open(..., 'w')? It's a different enough operation that it may have a different error.

Oh, yes, I forgot the 'w' mode.

Mark, could you please run following test on Windows?

import os
open('foo', 'wb').close()
flags = os.O_RDWR | os.O_CREAT | os.O_EXCL | getattr(os, 'O_NOFOLLOW', 0) | getattr(os, 'O_BINARY', 0)
os.open('foo/bar', flags, 0o600)          # what raised?
os.open('nonexistent/bar', flags, 0o600)  # what raised?

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

>>> open('foo', 'wb').close()
>>> flags = os.O_RDWR | os.O_CREAT | os.O_EXCL | getattr(os, 'O_NOFOLLOW', 0) | getattr(os, 'O_BINARY', 0)
>>> os.open('foo/bar', flags, 0o600)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'foo/bar'
>>> os.open('nonexistent/bar', flags, 0o600)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'nonexistent/bar'

[serhiy-storchaka]

Great. There is only one difference between Windows and Linux, but it affects only error type in tests. Here is a patch with updated test. It should now work on Windows.

[8e06f10c-4750-4034-a995-c3e4932e02af]

os.access doesn't check filesystem permissions, so the patch will not catch the condition that creates the problem.

[serhiy-storchaka]

It is best that we can do. How else we can check filesystem permissions? Only trying to create a file, But we just tried this and it failed.

[8e06f10c-4750-4034-a995-c3e4932e02af]

It doesn't actually do anything, so why do it at all? In order to distinguish why it failed, you might try checking if the file actually exists, and if it is a folder.

[tjguk]

And, just to be clear to Serhiy who I know doesn't use Windows, os.access really is a worthless function in its present form: worse, even, because it can be misleading. I have a long-standing patch to convert it to use AccessCheck but I've never quite had the guts to commit it because I fear the breakage would be too great.

[serhiy-storchaka]

The main issue is not tempfile raises a FileExistsError, but that it hangs for several seconds (for example if the temp dir doesn't exist). The patch allows to fail early and try other temp dir.

os.access() is not enough, we can add os.path.isdir(). Could you please run tests on patched Python on Windows and say what tests are failed?

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

The feedback here https://mail.python.org/pipermail/python-dev/2011-May/111530.html seems positive. It references bpo-2528 which is still open but strikes me as the way forward. Why don't we go for it and nail this issue once and for all?

[serhiy-storchaka]

Added os.path.isdir().

Could anybody please run tests on Windows?

[pfmoore]

\====================================================================== ERROR: test_read_only_directory (test.test_tempfile.TestMkdtemp) ----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 267, in _inside_empty_temp_dir
    yield
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 286, in test_read_only_directory
    self.skipTest("can't set the directory read-only")
  File "C:\Work\Projects\cpython\lib\unittest\case.py", line 645, in skipTest
    raise SkipTest(reason)
unittest.case.SkipTest: can't set the directory read-only

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 289, in test_read_only_directory
    self.assertEqual(os.listdir(tempfile.tempdir), [])
  File "C:\Work\Projects\cpython\lib\contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 269, in _inside_empty_temp_dir
    support.rmtree(dir)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 374, in rmtree
    _rmtree(path)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 354, in _rmtree
    _waitfor(os.rmdir, path)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 301, in _waitfor
    func(pathname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Gustav\\AppData\\Local\\Temp\\tmpe53kiky0'

====================================================================== ERROR: test_read_only_directory (test.test_tempfile.TestMkstempInner) ----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 267, in _inside_empty_temp_dir
    yield
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 286, in test_read_only_directory
    self.skipTest("can't set the directory read-only")
  File "C:\Work\Projects\cpython\lib\unittest\case.py", line 645, in skipTest
    raise SkipTest(reason)
unittest.case.SkipTest: can't set the directory read-only

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 289, in test_read_only_directory
    self.assertEqual(os.listdir(tempfile.tempdir), [])
  File "C:\Work\Projects\cpython\lib\contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 269, in _inside_empty_temp_dir
    support.rmtree(dir)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 374, in rmtree
    _rmtree(path)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 354, in _rmtree
    _waitfor(os.rmdir, path)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 301, in _waitfor
    func(pathname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Gustav\\AppData\\Local\\Temp\\tmp0qwkkr7l'

[serhiy-storchaka]

Thank you Paul. What with updated patch?

[pfmoore]

Works fine with the new patch:

.\rt.bat -x64 -q test_tempfile

C:\Work\Projects\cpython\PCbuild>"C:\Work\Projects\cpython\PCbuild\amd64\python.exe" -Wd -E -bb "C:\Work\Projects\cpython\PCbuild\..\lib\test\regrtest.py" test_tempfile

[1/1] test_tempfile 1 test OK.

[1762cc99-3127-4a62-9baf-30c3d0f51ef7]

New changeset 63f0ae6e218a by Serhiy Storchaka in branch '2.7': Issue bpo-22107: tempfile.gettempdir() and tempfile.mkdtemp() now try again https://hg.python.org/cpython/rev/63f0ae6e218a

New changeset 3a387854d106 by Serhiy Storchaka in branch '3.4': Issue bpo-22107: tempfile.gettempdir() and tempfile.mkdtemp() now try again https://hg.python.org/cpython/rev/3a387854d106

New changeset 1134198e23bd by Serhiy Storchaka in branch 'default': Issue bpo-22107: tempfile.gettempdir() and tempfile.mkdtemp() now try again https://hg.python.org/cpython/rev/1134198e23bd

[serhiy-storchaka]

Unfortunately the patch doesn't fix original issue and looks as this can't be fixed until os.access will be more useful on Windows. But it fixes several related issues. mkstemp() now fails early if parent directory doesn't exist or is a file. gettempdir() and mkdtemp() now try again in case of collision on Windows as well as on Unix. I hope that fixing os.access will automatically fix original issue.

Thanks Mark and Paul for testing on Windows. Thanks Tim for explaining issue with os.access.

[tjguk]

My reluctance to commit the os.access patch is because it will cause such a behaviour change in a function which has been pretty stable for a long while. It'll certainly be more correct, but at the undoubted expense of breaking someone's long-working code.

[pfmoore]

I'm not sure I follow. Isn't the point of this patch to try again in certain cases of a PermissionError, where currently the code breaks out of the loop early? How can the result be worse than the current behaviour? Suerly sometimes (maybe not always) it works now where it previously failed, but that's a good thing?

[eryksun]

Shouldn't it be checking whether file (or filename) is a directory [1]? For example:

    except PermissionError:
        # This exception is thrown when a directory with
        # the chosen name already exists on windows.
        if _os.name == 'nt' and _os.path.isdir(file):
            continue
        # If the directory allows write access, continue 
        # trying names. On Windows, currently this test 
        # doesn't work. The implementation assumes all 
        # directories allow write access, but it really 
        # depends on the directory's discretionary and 
        # system access control lists (DACL & SACL).
        elif _os.access(dir, _os.W_OK):
            continue
        else:
            raise

---

[1]: Windows sets the last error to ERROR_ACCESS_DENIED when a system call returns the NTSTATUS code STATUS_FILE_IS_A_DIRECTORY. This status code is returned by NtCreateFile and NtOpenFile when the target file is a directory, but the caller asked for anything but a directory (i.e. CreateOptions contains FILE_NON_DIRECTORY_FILE).

[serhiy-storchaka]

There is a risk of race condition. One process can create a directory file, then other process fails to create a file with the same name file, then the first process removes directory file, then the second process handles PermissionError. The same is possible also with threads.

[eryksun]

Ok, I think I understand now. You chose an indirect check to avoid the race condition. If we have write access to the directory, then the PermissionError must be because a directory exists with the same name.

Unfortunately os.access doesn't currently tell us this information. Moreover, checking isdir(dir) and access(dir, W_OK) is actually redundant as things currently stand. If isdir returns True that means GetFileAttributes didn't fail, which means access must return True as designed:

    return_value = (attr != INVALID_FILE_ATTRIBUTES) &&
                   ( !(mode & 2) ||
                     !(attr & FILE_ATTRIBUTE_READONLY) ||
                      (attr & FILE_ATTRIBUTE_DIRECTORY)   );

[eryksun]

My reluctance to commit the os.access patch is because it will cause such a behaviour change in a function which has been pretty stable for a long while.

The original behavior could be preserved as the default. A keyword-only argument could enable checking access based on the file security and thread token.

[tjguk]

That is a possibility which hadn't occurred to me. @eryksun, would you mind eyeballing the patch over on bpo-2582? It almost certainly won't apply cleanly as it's been almost two years since I last refreshed it, but you can hopefully gauge the intent.

[cd34197f-d4a4-4e30-9fbe-454f267f097e]

This issue was closed, but I believe the original bug reported was not fixed: trying to create a temporary file in a directory where you don't have write permissions hangs for a long time before failing with a misleading FileExistsError, rather than failing immediately with PermissionError.

I've just run into this on Python 3.5.1 while trying to use tempfile to check if a directory is writable - which I'm doing precisely because os.access() isn't useful on Windows!

I find it hard to believe that there is no way to distinguish a failure because the name is already used for a subdirectory from a failure because we don't have permission to create a file.

[7b4e8ca0-708a-4b6d-8e29-eafae9758a4b]

I've also run into this bug on Windows. In my case, the tempdir path includes directories on a network share, which I lack write access permissions to. Python tries to generate a *lot* of files, and never figures out it should move on to another directory. The attached patch for tempdir.py resolves my issue.

In _get_default_tempdir() and _mkstemp_inner(), you want to know if the filename you tried to create already exists as a directory, not whether the parent directory is a directory – that's handled in _get_default_tempdir().

In mkdtemp(), attempting to create a directory with the same name as an existing directory does not throw a PermissionError, so the code is superfluous.

[serhiy-storchaka]

Billy, no, the parent directory is tested intentionally. If it isn't a directory, we should fail.

May be there is a way to distinguish a failure because the name is already used for a subdirectory from a failure because we don't have permission to create a file, but I haven't access to Windows and can't experiment. Someone motivated and experienced should to do this work.

[eryksun]

The Windows API loses information when mapping kernel status values to Windows error codes. For example, the following status values are all mapped to ERROR_ACCESS_DENIED:

STATUS_INVALID_LOCK_SEQUENCE   0xc000001e
STATUS_INVALID_VIEW_SIZE       0xc000001f
STATUS_ALREADY_COMMITTED       0xc0000021
STATUS_ACCESS_DENIED           0xc0000022
STATUS_PORT_CONNECTION_REFUSED 0xc0000041
STATUS_THREAD_IS_TERMINATING   0xc000004b
STATUS_DELETE_PENDING          0xc0000056
STATUS_FILE_IS_A_DIRECTORY     0xc00000ba
STATUS_FILE_RENAMED            0xc00000d5
STATUS_PROCESS_IS_TERMINATING  0xc000010a
STATUS_CANNOT_DELETE           0xc0000121
STATUS_FILE_DELETED            0xc0000123

Encrypting File System
STATUS_ENCRYPTION_FAILED       0xc000028a
STATUS_DECRYPTION_FAILED       0xc000028b
STATUS_NO_RECOVERY_POLICY      0xc000028d
STATUS_NO_EFS                  0xc000028e
STATUS_WRONG_EFS               0xc000028f
STATUS_NO_USER_KEYS            0xc0000290

STATUS_ACCESS_DENIED is from a failed NtAccessCheck. STATUS_FILE_IS_A_DIRECTORY is from trying to open a directory as a file, because NT doesn't allow accessing the anonymous data stream of a directory, such as "dirname::$DATA", which is the same as trying to open "dirname" as a file. It only allows creating a named data stream for a directory, such as "dirname:streamname:$DATA".

The original status value may still be available, but only by calling the undocumented runtime library function, RtlGetLastNtStatus, which was added in XP (NT 5.1). After a failed system call, the Windows base API calls BaseSetLastNTError, which calls RtlNtStatusToDosError to get the Win32/DOS error code for a given NT status value. This in turn caches the last NT status in the LastStatusValue field of the thread environment block (TEB). RtlGetLastNtStatus gets this value from the TEB.

Possibly PyErr_SetExcFromWindowsErrWithFilenameObjects could capture the most recent kernel status value from RtlGetLastNtStatus(), to add this as a new "ntstatus" attribute of OSError. This wouldn't always be meaningful, since the thread's LastErrorValue (returned by GetLastError) isn't always related to a failed system call, but it can help in cases such as this, to distinguish a genuine denial of access from some other failure, and without suffering from race conditions.

For example, I added a "testdir" subdirectory to a directory and then modified the DACL of the parent directory to deny write/append access for all users. The following experiment checks the NT status using ctypes:

    STATUS_ACCESS_DENIED = ctypes.c_long(0xC0000022).value
    STATUS_FILE_IS_A_DIRECTORY = ctypes.c_long(0xC00000BA).value

    ntdll = ctypes.WinDLL('ntdll')

try: open('testdir', 'w')
except: status_dir = ntdll.RtlGetLastNtStatus()

try: open('test', 'w')
except: status_access = ntdll.RtlGetLastNtStatus()

    >>> status_dir == STATUS_FILE_IS_A_DIRECTORY
    True
    >>> status_access == STATUS_ACCESS_DENIED
    True

Obviously using ctypes isn't recommended, since the status value needs to be captured ASAP after the call fails, so here's an example in C:

    #define UNICODE
    #include <Windows.h>
    #include <fcntl.h>
    #include <stdio.h>

typedef NTSTATUS (NTAPI *RTLGETLASTNTSTATUS)(VOID);

    int main()
    {
        HMODULE hNtdll = GetModuleHandle(L"ntdll");
        RTLGETLASTNTSTATUS RtlGetLastNtStatus = (RTLGETLASTNTSTATUS)
            GetProcAddress(hNtdll, "RtlGetLastNtStatus");

        if (_open("testdir", _O_CREAT | _O_EXCL) == -1)
            printf("status_dir: %#08x\n", RtlGetLastNtStatus());

        if (_open("test", _O_CREAT | _O_EXCL) == -1)
            printf("status_access: %#08x\n", RtlGetLastNtStatus());

       return 0;
    }

output:

status_dir: 0xc00000ba
status_access: 0xc0000022

[eec47d6c-1da6-47af-aaa0-95451fea4ba1]

Can the _mkstemp_inner portion of Billy McCulloch's patch be applied? Due to a large default os.TMP_MAX value (2147483647 - seems to be the current value on Win 7/8.1/10 I have access to), the following will push the CPU to 100% for a very long time when run under a non-elevated shell:

-- import tempfile tempfile.TemporaryFile(dir='C:\Windows') ... wait ...

In _mkstemp_inner() we should be testing for the filename, not parent directory here:

        except PermissionError:
            # This exception is thrown when a directory with the chosen name
            # already exists on windows.
            if (_os.name == 'nt' and _os.path.isdir(dir) and
                _os.access(dir, _os.W_OK)):
                continue

Changing the _os.path.isdir(dir) call to _os.path.isdir(filename) is all that is needed to prevent the death loop and function correctly in cases where Windows os.access(dir, _os.W_OK) claims we have write access when we do not.

[40eb10e5-107a-49fc-8df9-709d3b9f2217]

I just experienced the described problem using Python 3.6.0 (Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32), but i do not understand the current status of this issue: On the one hand it is marked as 'open', which seems to imply that is still not resolved. On the other hand the resolution was set to 'fixed' in May 2015. Should i open a new issue for Python 3.6?

[e38a025c-f063-4852-bd07-66afc81d53fb]

This issue persists as of today (March 2019), in Python 3.7.2 (64 bit) running on Windows 10. I gather from the comments that fixing it is no trivial matter, although I don't fully understand why. The hang of "several seconds" that was originally described is at least 30 seconds on that platform -- I'm not sure when it would clear, as I didn't have the patience to wait it out.

https://stackoverflow.com/questions/55109076/python-tempfile-temporaryfile-hangs-on-windows-when-no-write-privilege

It seems to me that a genuine naming collision would be pretty rare -- at least for the (fairly common) use case I'm dealing with, trying to check where the directory is writeable or not. Would it make sense to at least set a default number of collision-retries that is significantly lower? Say, between 3 and 10, instead of the system max?

[7b4e8ca0-708a-4b6d-8e29-eafae9758a4b]

I stand by the patch file I previously submitted on 2016-05-04. A more detailed analysis / description of my reasoning follows.

Change 1 in _get_default_tempdir: A PermissionError is thrown on Windows if you attempt to create a file whose filename matches an existing directory. As the code currently stands, the if statement checks whether the proposed file's parent directory is a directory (which it is, or a FileNotFoundError would have been thrown), instead of whether the proposed filename conflicts with an existing directory. The edited expression is really a typo that, in the context of the code block, always evaluates True. Here’s what we’re now saying in the if block: when a PermissionError is raised, if we’re on Windows (the only currently supported platform to throw nonsense errors at us) AND the filename we chose simply conflicts with an existing directory AND we supposedly have write access to the parent directory, then we were just unlucky with the chosen name and should try again in the same parent directory. (I say supposedly because Windows seems to erroneously report True on this check, even when we don’t have write access. I wouldn’t be surprised if this last check does something useful in certain contexts, I just don’t know what they are.)

Change 2 in _mkstemp_inner: Same as above for Change 1. While _get_default_tempdir uses this code block to make sure the system tempdir is really writable, and _mkstemp_inner does it so that a file descriptor can be returned, the result and arguments are the same.

Change 3 in mkdtemp: For _get_default_tempdir and _mkstemp_inner, the blocks of code in question are creating temporary files. As such, they need to handle the oddball case for Windows where attempts to create a file with a filename which conflicts with an existing directory name result in a PermissionError. The same block of error handling code is copied in mkdtemp, even though this function never tries to create a file – it only tries to create a directory. As such, in the case that we try to create a directory with a name that already exists (whether as a file or a directory), we wouldn't be dealing with a PermissionError, we'd have a FileExistsError, which is already handled in mkdtemp by the preceding lines. The only way I’ve seen a PermissionError crop up for a call to mkdir on Windows is if the user doesn’t have permission to create filesystem objects in the parent directory. This is the intended usage of a PermissionError, so no special handling needed is required. Remember, a PermissionError shouldn’t happen if mkdtemp is called without a dir kwarg, because the _sanitize_params will invoke _get_default_tempdir, which will check to ensure that the parent directory is writable. As such, this block of code was superfluous, and the patch should not raise PermissionError in user code where it previously was caught.

[dd602496-28cc-4360-9fb4-7b7b694c6e12]

Series of operations needed to answer the questions os.access is not answering on windows:

bool CanAccessFolder( LPCTSTR folderName, DWORD genericAccessRights )
{
    bool bRet = false;
    DWORD length = 0;
    if (!::GetFileSecurity( folderName, OWNER_SECURITY_INFORMATION | GROUP_SECURITY_INFORMATION 
            | DACL_SECURITY_INFORMATION, NULL, NULL, &length ) && 
            ERROR_INSUFFICIENT_BUFFER == ::GetLastError()) {
        PSECURITY_DESCRIPTOR security = static_cast< PSECURITY_DESCRIPTOR >( ::malloc( length ) );
        if (security && ::GetFileSecurity( folderName, OWNER_SECURITY_INFORMATION | GROUP_SECURITY_INFORMATION
                            | DACL_SECURITY_INFORMATION, security, length, &length )) {
            HANDLE hToken = NULL;
            if (::OpenProcessToken( ::GetCurrentProcess(), TOKEN_IMPERSONATE | TOKEN_QUERY | 
                    TOKEN_DUPLICATE | STANDARD_RIGHTS_READ, &hToken )) {
                HANDLE hImpersonatedToken = NULL;
                if (::DuplicateToken( hToken, SecurityImpersonation, &hImpersonatedToken )) {
                    GENERIC_MAPPING mapping = { 0xFFFFFFFF };
                    PRIVILEGE_SET privileges = { 0 };
                    DWORD grantedAccess = 0, privilegesLength = sizeof( privileges );
                    BOOL result = FALSE;

                    mapping.GenericRead = FILE_GENERIC_READ;
                    mapping.GenericWrite = FILE_GENERIC_WRITE;
                    mapping.GenericExecute = FILE_GENERIC_EXECUTE;
                    mapping.GenericAll = FILE_ALL_ACCESS;

                    ::MapGenericMask( &genericAccessRights, &mapping );
                    if (::AccessCheck( security, hImpersonatedToken, genericAccessRights, 
                            &mapping, &privileges, &privilegesLength, &grantedAccess, &result )) {
                        bRet = (result == TRUE);
                    }
                    ::CloseHandle( hImpersonatedToken );
                }
                ::CloseHandle( hToken );
            }
            ::free( security );
        }
    }

    return bRet;
}

[eryksun]

CanAccessFolder is incomplete for the following reasons:

(1) It doesn't account for SeBackupPrivilege (read and execute access) and SeRestorePrivilege (write and delete access). If a create or open call requests backup semantics, these two privileges are checked by the I/O Manager in order to grant access before the request is sent to the filesystem device stack. That said, our os.open() call doesn't use O_OBTAIN_DIR (0x2000), an undocumented flag that allows opening a directory by requesting backup semantics. If it did use this undocumented flag, there would actually be no problem to solve since we would get FileExistsError instead of PermissionError.

(2) It doesn't check the parent directory in case FILE_READ_ATTRIBUTES access (required for GENERIC_READ) or DELETE access (required for GENERIC_ALL) are either not granted or denied to the user for the target directory. These rights are granted by NT filesystem policy if FILE_READ_DATA and FILE_DELETE_CHILD access are granted to the parent directory, respectively. This is a general concern that's not relevant here since we only need to check the directory for write access (i.e. FILE_ADD_FILE).

(3) It doesn't get the LABEL_SECURITY_INFORMATION from the file security, in order to check for no-read-up, no-write-up, and no-execute-up mandatory access. Adding files to a directory is disallowed if its mandatory label denies write-up access and its integrity level is higher than the caller's (e.g. the caller is a standard user at medium integrity level, and the directory is at high or system integrity level).

(4) It doesn't check effective access via OpenThreadToken, in case the thread is impersonating.

(5) It cannot take into account access granted or denied by filesystem filter drivers such as antimalware programs. For this, we need to actually try to open the directory with the requested access via CreateFile. We're granted the required access if CreateFile succeeds, or if it fails with a sharing violation (i.e. winerror 32). A sharing violation isn't an issue since in practice adding a file to a directory is internal to a filesystem; it doesn't count against shared data access.

[4b503c90-0393-4d2b-9f55-8524b213d6fc]

Attempting to answer the question "did this open call fail because the path was a directory" by implication from "do we think we ought to be able to write a file to this directory" is IMO doomed. There's no reliable way to determine whether one should be able to write to a location, short of trying to write to it. There isn't in general and there especially isn't on Windows, for the reasons discussed by Eryk and more.

I believe Billy's patch is an improvement over what we have, in as much as it's specifically checking for the condition we care about to work around a shortcoming of Windows error reporting. I would further remove the use of os.access, which does nothing useful (it reflects the read-only file attribute, which no-one uses, and which doesn't exist for directories anyway).

Yes, there is a race condition if the directory goes away between use and check, but it seems vanishingly unlikely this could happen by accident, and it could only happen on purpose if an attacker can guess the random filenames in which case they already have worse attacks than just making mkstemp fail. In general failure is a much better outcome than hanging for hours on end.

We may be overthinking this. Maybe it is OK to treat all permission errors as maybe-file-exists errors, like bpo-18849 originally did, and just retry without trying to pick apart the entrails.

...just not quite as many as 2147483647 times. Given how unlikely an accidental filename clash is in the first place, I'm thinking a more realistic number of retries might be something like, 2.

os.TMP_MAX probably isn't a good choice in any case, as it indicates the number of names C's tmpnam() can come up with, which (a) is unrelated to the number of names _RandomNameSequence can come up with and (b) we have no particular reason to try to completely exhaust anyway.

[dd602496-28cc-4360-9fb4-7b7b694c6e12]

i would like to point out that the primary reason any of this nonsense exists is because of short filename restrictions.

i've replaces nearly all of my temp file creation code in all of my project to return os.urandom(32).hex() ... which is reliable secure, fast, and doesn't require any fileops

sure, it doesn't work on 8.3 limited systems, but maybe the NamedTemp code should check that *first.... and *if it's OK to use long names... just use them, otherwise revert to existing behavior

[dd602496-28cc-4360-9fb4-7b7b694c6e12]

This is the fist of what I'm using: https://gist.github.com/earonesty/a052ce176e99d5a659472d0dab6ea361

Seems OK for my use cases. There's probably issues with relying on __del__ this way. But it solves the Windows close/reopen problem, too.

[nethe-GitHub]

I can't believe this issue haven't been solved after ten years. And some package really takes tempfile.TemporaryFile() for writability check. Hope Windows users enjoy the 2147483647 loops.

[zooba]

Maybe there really is a reason for PathYetAnotherMakeUniqueName?

[TheRealQuantam]

Ran into this issue yesterday when I tried using a module that used Numba on Windows. Somebody has probably already mentioned this, but I'm not reading this whole thread. The root of the problem is this comment in posixmodule.c:

(Directories cannot be read-only on Windows.)

This is NOT true. While there is no exact (single) equivalent, Windows has multiple permissions that make directories effectively read-only. In this case the permission that is lacking is FILE_ADD_FILE, which allows files to be created in that directory (an alias of FILE_WRITE_DATA, which is the effect it has when applied to files). Related is FILE_ADD_SUBDIRECTORY (an alias of FILE_APPEND_DATA).

After a bit of searching, I found a straightforward solution in a 2004 (cough 10 years before the problem appeared in Python cough) post by old Chen. Here is a hacky workaround I made for my own use. It probably isn't sufficiently robust for production code, but it gets the job done until Python properly fixes it:

import win32api
import win32
import win32file
import pywintypes

orig_access = os.access
def my_access(path, mode, *, dir_fd=None, effective_ids=False, follow_symlinks=True):
    res = orig_access(path, mode, dir_fd = dir_fd, effective_ids = effective_ids, follow_symlinks = follow_symlinks)
    if not res or not mode & os.W_OK or not os.path.isdir(path):
        return res

    try:
        hdl = win32file.CreateFile(
            str(path), 
            2, # FILE_ADD_FILE
            win32file.FILE_SHARE_READ | win32file.FILE_SHARE_WRITE | win32file.FILE_SHARE_DELETE, 
            None, 
            win32file.OPEN_EXISTING, 
            win32file.FILE_FLAG_BACKUP_SEMANTICS, 
            None,
        )
        win32file.CloseHandle(hdl)

    except pywintypes.error as e:
        if e.winerror == 5: # ERROR_ACCESS_DENIED
            return False

    except Exception as e:
        pass

    return True

os.access = my_access```

[zooba]

Changing os.access to detect a directory (which it already does) and testing for FILE_ADD_FILE access as shown by @TheRealQuantam seems reasonable. If someone wants to work up a PR, I'm sure we could move that forward.

I'm not sure whether it would unblock the original issue, but only because I also haven't re-read the entire thread.

[TheRealQuantam]

For an official fix it would probably be best to make os.access test FILE_ADD_FILE | FILE_ADD_SUBDIRECTORY (6), as that's the more conservative option. That is, due to there being multiple permissions in Windows it's impossible for os.access to be 100% accurate in the edge case where a directory has add file but not add subdirectory permissions (or vice-versa); in such cases I'd think it's better to report that the directory is not writable even though files or directories can be created than report that it is writable only for attempts to create things to then fail.

I can confirm that at least in the case of Numba (which used tempfile to create temporary files) my workaround does in fact fix the issue with _mkstemp_inner (the topic of this issue). The attempt to create a temp file in a non-writable directory fails immediately for access denied, and Numba falls back to using the user's temp directory (the proper solution).

[EsbernTK]

I have just encountered this issue, when huggingface_hub tried to create a temporary dir in C:\\Users. In this case it seems that os.access("C:\\Users", os.W_OK) just totally misjudges the process permissions, as it returns True, while the directory requires administrator permissions for write access. Both os.makedirs("C:\\Users\\tmp12345") and fp = open("C:\\Users\\tmpfile.txt", "w") raise a PermissionError: [Errno 13], so it isn't even an edgecase where the process has FILE_ADD_FILE permission but lacks FILE_ADD_SUBDIRECTORY, as @TheRealQuantam suggests.

I also tested os.access("C:\\Windows\\System32", os.W_OK), which also returns True. A pretty shocking result. The only time i've made os.access return False, is when the directory does not exist. From this it seems that os.access does not actually check any permissions, but just if the directory actually exists or not, making its purpose a bit confusing on Windows. For the original error in mkdtemp, i might be missing some security/performance nuance in the original implementation and in this discussion, but if the function is misinterpreting the PermissionError to mean that the directory already exists, then why not check for that before the os.mkdir call. Furthermore, checking if the dir even exists and is a valid dir, before attempting to create a temporary directory inside it.

def mkdtemp(suffix=None, prefix=None, dir=None):
    """User-callable function to create and return a unique temporary
    directory.  The return value is the pathname of the directory.

    Arguments are as for mkstemp, except that the 'text' argument is
    not accepted.

    The directory is readable, writable, and searchable only by the
    creating user.

    Caller is responsible for deleting the directory when done with it.
    """

    prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
    if not _os.path.exists(dir): raise FileNotFoundError(_errno.ENOENT, "No such file or directory: %r" % dir)
    if not _os.path.isdir(dir): raise NotADirectoryError(_errno.ENOTDIR, "Not a directory: %r" % dir)

    names = _get_candidate_names()
    if output_type is bytes:
        names = map(_os.fsencode, names)
    for seq in range(TMP_MAX):
        name = next(names)
        file = _os.path.join(dir, prefix + name + suffix)
        if os.path.exists(file): continue
        _sys.audit("tempfile.mkdtemp", file)
        _os.mkdir(file, 0o700)
        return file
    raise FileExistsError(_errno.EEXIST, "No usable temporary directory name found")

I recognize that I do not understand the performance/stability/security implications of these changes fully, but on the surface it seems like a better approach than to rely on the try except statements from the previous code. Since the purpose of catching the PermissionError was solely to check if the file already exists, then the os.path.exists should take care of this case, while avoiding the edgecases that os.access caused in the previous code. And if _os.mkdir(file, 0o700) raises a PermissionError, it is now safe to assume that it was caused by missing permissions, and not that the file already exists. Of course there is the very unlikely race condition, where another process creates the file between the check and the _os.mkdir(file, 0o700) call. However, as noted earlier in this thread, this is exceedingly unlikely. Though if this is a genuine concern, it can be handled through a try block like before.

try:
    _os.mkdir(file, 0o700)
except FileExistsError:
    continue    # try again
except PermissionError:
    # This exception is thrown when a directory with the chosen name
    # already exists on windows, or if the process does not have proper
    # permissions to create the directory.
    if (_os.name == 'nt' and _os.path.exists(file)):
        continue
    else:
        raise

Again, i've removed the os.access check from the if statement, since its purpose is fulfilled more directly by checking if the file actually exists or not.