Open Sup3rGeo opened 4 years ago
Hi @Sup3rGeo!
atomicwrites
tries to create the temporary directory 'c:\\program files\\python36\\lib\\site-packages\\typhoon\\__pycache__\\tmpu12vgv2e'
using tempfile.mkstemp
, which in turn calls _mkstemp_inner
:
def _mkstemp_inner(dir, pre, suf, flags, output_type):
"""Code common to mkstemp, TemporaryFile, and NamedTemporaryFile."""
names = _get_candidate_names()
if output_type is bytes:
names = map(_os.fsencode, names)
for seq in range(TMP_MAX):
name = next(names)
file = _os.path.join(dir, pre + name + suf)
try:
fd = _os.open(file, flags, 0o600)
except FileExistsError:
continue # try again
except PermissionError:
# This exception is thrown when a directory with the chosen name
# already exists on windows.
if (_os.name == 'nt' and _os.path.isdir(dir) and
_os.access(dir, _os.W_OK)):
continue
else:
raise
return (fd, _os.path.abspath(file))
raise FileExistsError(_errno.EEXIST,
"No usable temporary file name found")
The traceback shows it is hanging on the PermissionError
handling, which does call _os.access(dir, _os.W_OK)
to check if it should keep retrying to create the temporary directory, otherwise it fails. It seems for some reason it is keeping looping there, instead of _os.access(dir, _os.W_OK)
returning False
and we getting the expected PermissionError
.
Strange, unfortunately this is hard to debug. 😕
Hopefully #4730 would help (see also #4755) Edit: now done in https://github.com/pytest-dev/pytest/pull/5864
According to this stackoverflow question, it seems this is a long lasting issue for windows that has not been yet fixed:
https://bugs.python.org/issue22107
Let's hope the items mentioned by @sparrowt above will help workaround the issue for pytest.
Let's hope the items mentioned by @sparrowt above will help workaround the issue for pytest.
It would definitely help, but only for py38 users I'm afraid... 🤔
sadly this issue still exists on Windows 10, Python 3.8.6, pytest 6.1.2 😖
I guess a workaround would be to check for permission before calling into atomic_write
.
I've spent a few minutes trying to write a test for it:
@pytest.mark.skipif(not sys.platform.startswith("win"), reason="Windows only (#5844)")
def test_write_pyc(tmp_path: Path, request) -> None:
cache_dir = tmp_path.joinpath("cache")
cache_dir.mkdir()
try:
cache_dir.chmod(0) # this doesn't work
with pytest.raises(PermissionError):
cache_dir.joinpath("foo").touch()
# call _write_pyc and check it returns False instead of hanging
finally:
cache_dir.chmod(stat.S_IWRITE)
But unfortunately couldn't get cached_dir
to throw a PermissionError
for some reason. I'm leaving this here in case someone has some tips, as I'm short on time right now to investigate further. 👍
Stumbled across this too... There are two issues:
os.fspath(pyc) may be read-only (on linux too?) and this arguments does not go to get_fileobject (this is called with dir empty) and then is replaces it (which is dangerous, since this may be read-only too)
if dir is None:
dir = os.path.normpath(os.path.dirname(self._path))
printing dir after these two lines in atomicwrites/init.py
gives me C:\Program Files\Python38\lib\site-packages\hypothesis__pycache__
with is write protected.
File "C:\Program Files\Python38\lib\site-packages_pytest\assertion\rewrite.py", line 165, in exec_module _write_pyc(state, co, source_stat, pyc) File "C:\Program Files\Python38\lib\site-packages_pytest\assertion\rewrite.py", line 308, in _write_pyc with atomic_write(os.fspath(pyc), mode="wb", overwrite=True) as fp: File "C:\Program Files\Python38\lib\contextlib.py", line 113, in enter return next(self.gen) File "C:\Program Files\Python38\lib\site-packages\atomicwrites__init.py", line 166, in _open with get_fileobject(**self._open_kwargs) as f: File "C:\Program Files\Python38\lib\site-packages\atomicwrites\init__.py", line 190, in get_fileobject
Calling pytest with --assert=plain is a work-around.
Still broken on Win10, Python 3.9.6, pytest-6.2.4 Calling pytest with --assert=plain indeed works.
And in my case pytest was trying to rewrite allure-pytest module under site-packages. I don't think pytest has to rewrite libraries. On my other env with python 3.8 pytest is working Ok, I have found that python itself is owned by system and some folders under site-packages are owned by my user, this is the reason of not failing.
Did some more experiments and found out that:
And I think other workaround is not installing python under program files, but under your user. Didn't test it though.
IMHO pytest should detect if it has write access, and either skip rewriting for modules he doesn't have access to, either write PYCs somewhere else in this case, either don't rewrite libraries within python at all.
Another workaround maybe to use virtual environments (which have the user's rights).
But how do the Linux guys handle this? They should get the same problem...
Ran into this issue but couldn't use virtual environments out of the box since I am using a program that depends on using system-wide installed packages. The solution ended up being installing all my needed packages to the system and then creating a virtual environment with the command virtualenv --system-site-packages
to use all of the system packages. Then I installed pytest on its own to the virtual environment with pip install --ignore-installed pytest
. This forced pytest to not be able to write to system-installed packages. Hopefully this workaround will help someone out. Credit: https://stackoverflow.com/a/19459977
I have also been experiencing this issue (currently on Windows 10, Python 3.9.7, and pytest 7.1.1).
I've observed that it happens after I've used pip
to update packages. My workaround is to delete all the __pycache__
directories in the Python installation directory. Then pytest
starts working even without elevated permissions.
Basically, it seems that if:
C:\Program files
(in a situation that you also have to runpip install
from an elevated command prompt in order to install packages)Then pytest just hangs. CTRL+C shows the following traceback:
I would expect it not to hang, but for it to terminate with a
Permission denied
message instead.Using Windows 10, python 3.6.8 and pytest 5.0.0