search

pyinstaller / pyinstaller

Freeze (package) Python programs into stand-alone executables
http://www.pyinstaller.org
Other
11.8k stars 1.94k forks source link

PermissionError: [Errno 13] Permission denied --onefile FileIO Issue #6467

Closed fatihkabakk closed 2 years ago

fatihkabakk commented 2 years ago

Description of the issue

When creating single exe file in --onefile mode, the file \lib\site-packages\PyInstaller\building\api.py at line 734, with open(self.name, 'ab') as outf: the program raises a PermissionError. I have tried adding while loop and an exception catcher, it looped 509 times before properly opening the file. I suppose that file does not get closed completely before the append process starts. I would make a pull request about this problem, but I'm not completely sure how to solve this problem properly. It can be solved without including sleep, but then program loops so many times (509 as stated before). My fix is added below;

My Program's Import Details (None of Importance I think, see A minimal example program which shows the error)

import threading
from tkinter import *
import tkinter.messagebox as messagebox
from tkinter import filedialog
from PIL import Image, ImageTk
from os.path import expanduser

Context information (for bug reports)

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

Make sure everything is packaged correctly

A minimal example program which shows the error

Stacktrace / full error message

30364 INFO: Appending PKG archive to EXE
Traceback (most recent call last):
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main   
    return _run_code(code, main_globals, None,
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\Scripts\pyinstaller.exe\__main__.py", line 7, in <module>
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\site-packages\PyInstaller\__main__.py", line 124, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\site-packages\PyInstaller\__main__.py", line 58, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\site-packages\PyInstaller\building\build_main.py", line 827, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\site-packages\PyInstaller\building\build_main.py", line 759, in build
    exec(code, spec_namespace)
  File "C:\Users\Fatih\Desktop\txtToCsvConverter\Txt To Csv Converter.spec", line 24, in <module>
    exe = EXE(
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\site-packages\PyInstaller\building\api.py", line 518, in __init__
    self.__postinit__()
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\site-packages\PyInstaller\building\datastruct.py", line 155, in __postinit__
    self.assemble()
  File "C:\Users\Fatih\AppData\Local\Programs\Python\Python39\lib\site-packages\PyInstaller\building\api.py", line 734, in assemble
    with open(self.name, 'ab') as outf:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Fatih\\Desktop\\txtToCsvConverter\\dist\\Txt To Csv Converter.exe'

Please also see https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs for more about what would use to solve the issue.

rokm commented 2 years ago

Do you have an anti-virus program running on the system? This looks like an AV blocking the access to the exe file...

fatihkabakk commented 2 years ago

Do you have an anti-virus program running on the system? This looks like an AV blocking the access to the exe file...

I just have windows-defender, and tried with disabling it, too. However, issue still exists.

fatihkabakk commented 2 years ago

Also, I used this program in the past, and it was working. It got broken after some changes I think.

rokm commented 2 years ago

Also, I used this program in the past, and it was working.

The rest of your system is not exactly frozen in time, either.

It got broken after some changes I think.

If you can pinpoint the breaking change (using git bisect), we can investigate further.

Otherwise, I'm inclined to believe that this is specific to your system/environment; if it was the case of a file not being closed fast enough, as you suspect, then our CI would be falling apart as well, and we'd be likely seeing more reports like this. The ones that we do get, however, typically boil down to AV, e.g., #6463.

But even if this is a legit issue in PyInstaller, I don't think retry loop is a proper solution...

Aarekaz commented 2 years ago

I am also having the permission denied issue:

350 INFO: Python: 3.9.0
350 INFO: Platform: Windows-10-10.0.22000-SP0
352 INFO: wrote C:\Users\anura\Desktop\GUI\gui.spec
359 INFO: UPX is not available.
361 INFO: Extending PYTHONPATH with paths
['C:\\Users\\anura\\Desktop\\GUI']
852 INFO: checking Analysis
2789 INFO: checking PYZ
4160 INFO: checking PKG
4163 INFO: Bootloader c:\users\anura\desktop\gui\venv\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
4163 INFO: checking EXE
4164 INFO: Building EXE because EXE-00.toc is non existent
4164 INFO: Building EXE from EXE-00.toc
4165 INFO: Copying bootloader EXE to C:\Users\anura\Desktop\GUI\build\gui\gui.exe
4175 INFO: Copying icon to EXE
4175 INFO: Copying icons from ['c:\\users\\anura\\desktop\\gui\\venv\\lib\\site-packages\\PyInstaller\\bootloader\\images\\icon-console.ico']
4177 INFO: Writing RT_GROUP_ICON 0 resource with 104 bytes
4177 INFO: Writing RT_ICON 1 resource with 3752 bytes
4178 INFO: Writing RT_ICON 2 resource with 2216 bytes
4178 INFO: Writing RT_ICON 3 resource with 1384 bytes
4178 INFO: Writing RT_ICON 4 resource with 37019 bytes
4179 INFO: Writing RT_ICON 5 resource with 9640 bytes
4179 INFO: Writing RT_ICON 6 resource with 4264 bytes
4180 INFO: Writing RT_ICON 7 resource with 1128 bytes
4188 INFO: Copying 0 resources to EXE
4188 INFO: Emedding manifest in EXE
4190 INFO: Updating manifest in C:\Users\anura\Desktop\GUI\build\gui\gui.exe
4191 INFO: Updating resource type 24 name 1 language 0
4200 INFO: Appending PKG archive to EXE
Traceback (most recent call last):
  File "C:\Users\anura\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\anura\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\anura\Desktop\GUI\venv\Scripts\pyinstaller.exe\__main__.py", line 7, in <module>
  File "c:\users\anura\desktop\gui\venv\lib\site-packages\PyInstaller\__main__.py", line 124, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "c:\users\anura\desktop\gui\venv\lib\site-packages\PyInstaller\__main__.py", line 58, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "c:\users\anura\desktop\gui\venv\lib\site-packages\PyInstaller\building\build_main.py", line 782, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "c:\users\anura\desktop\gui\venv\lib\site-packages\PyInstaller\building\build_main.py", line 714, in build
    exec(code, spec_namespace)
  File "C:\Users\anura\Desktop\GUI\gui.spec", line 23, in <module>
    exe = EXE(pyz,
  File "c:\users\anura\desktop\gui\venv\lib\site-packages\PyInstaller\building\api.py", line 507, in __init__
    self.__postinit__()
  File "c:\users\anura\desktop\gui\venv\lib\site-packages\PyInstaller\building\datastruct.py", line 155, in __postinit__
    self.assemble()
  File "c:\users\anura\desktop\gui\venv\lib\site-packages\PyInstaller\building\api.py", line 719, in assemble
    with open(self.name, 'ab') as outf:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\anura\\Desktop\\GUI\\build\\gui\\gui.exe'

I tried disabling the antivirus but it's the same.

bwoodsend commented 2 years ago

Does bouncing back a PyInstaller version or two make any difference?

Edit: I see you tried with develop too so I imagine that the answer is no.

Aarekaz commented 2 years ago

@bwoodsend No I have not. Which version do you recommend?

Aarekaz commented 2 years ago

I created the exe after using 4.4 build of py installer, but when I run it, I get this error image

I used this command : image

rokm commented 2 years ago

I created the exe after using 4.4 build of py installer, but when I run it, I get this error

You'd get that error with any version of PyInstaller; what you need to do is ensure that data files of qtstylish package are collected, for example by adding --collect-data qtstylish to your pyinstaller command.

Back to the original topic, I suppose 4.4 working means that changes made in #6252 (and released as part of 4.6) might have made us less resilient to AV software's interference: previously, we created a copy of stock bootloader exe to a temporary file, then added icon, resources and manifest. Then we copied the modified exe from the temporary location into final destination and at the same time appended the pkg (without closing the file descriptor in between these two steps). So I suppose if AV kicks in after manifest is embedded, but locks the file only for modification (i.e., still allowing reading/copying), we might copy the prepared bootloader without issues in spite of the AV. And because both the copy and PKG appending is done without closing the file descriptor in between, AV may kick in again only after the whole procedure is complete.

In the simplified, post-#6252 pipeline, we make a copy of stock bootloader exe to the file destination, add icon, resources and manifest, and then append PKG to it. This means that if AV kicks in after manifest is embedded and locks the file for writing, we may fail to open it again in append mode to append the PKG to it.

So perhaps we should go back to using an intermediate temporary file to prepare the bootloader...

Aarekaz commented 2 years ago

Now I am getting this error :


PandasGUI INFO — pandasgui.gui — Opening PandasGUI
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/case-sensitive.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/case-sensitive.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/regex.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/regex.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/match-exactly.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/match-exactly.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/arrow-up.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/arrow-up.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/arrow-down.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/arrow-down.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/close.svg', because: The system cannot find the path specified.
qt.svg: Cannot open file 'C:/Users/anura/AppData/Local/Temp/_MEI328882/pandasgui/resources/images/close.svg', because: The system cannot find the path specified.
Traceback (most recent call last):
  File "gui.py", line 308, in <module>
  File "gui.py", line 266, in show_img_gui
  File "pandasgui\gui.py", line 468, in show
  File "pandasgui\gui.py", line 71, in __init__
  File "pandasgui\gui.py", line 161, in init_ui
  File "pandasgui\gui.py", line 251, in apply_settings
  File "qtstylish\qtstylish.py", line 27, in light
ImportError: cannot import name 'qtstylish_rc' from 'compiled' (unknown location)
[22980] Failed to execute script 'gui' due to unhandled exception!

[process exited with code 1 (0x00000001)] ```
bwoodsend commented 2 years ago

So perhaps we should go back to using an intermediate temporary file to prepare the bootloader...

I'd rather see if #6469 and #6470 solve our problems before we start chucking temporary files into the mix.

rokm commented 2 years ago

@Aarekaz

Now I am getting this error : ImportError: cannot import name 'qtstylish_rc' from 'compiled' (unknown location)

So you need to also add --hiddenimport qtstylish.compiled.qtstylish_rc to your pyinstaller command (or a --collect-submodules qtstylish, although I think qtstylish_rc is the only problematic import).

You should probably also add a --collect-data pandasgui to your PyInstaller command to avoid warnings about missing svg files.

At any rate, this has nothing to do with the permission denied problem, so if you have any further problems, open a new discussion.

rokm commented 2 years ago

So perhaps we should go back to using an intermediate temporary file to prepare the bootloader...

I'd rather see if #6469 and #6470 solve our problems before we start chucking temporary files into the mix.

Yeah, I agree. It could be that AV is forcing a scan due to the tainted hash of the original bootloader copy (and suppose that it fails the race when we are embedding icon, resources and manifest, so it scans it after that). I wish I could reproduce this, but no luck...

@fatihkabakk @Aarekaz Does the permission-denied problem also occur if you add --debug bootloader and/or --windowed to your PyInstaller command (i.e., when run_d.exe, runw.exe or runw_d.exe are used). EDIT: Or, if you rebuild the bootloader instead of using the pre-compiled ones?

Aarekaz commented 2 years ago

@Aarekaz

Now I am getting this error : ImportError: cannot import name 'qtstylish_rc' from 'compiled' (unknown location)

So you need to also add --hiddenimport qtstylish.compiled.qtstylish_rc to your pyinstaller command (or a --collect-submodules qtstylish, although I think qtstylish_rc is the only problematic import).

You should probably also add a --collect-data pandasgui to your PyInstaller command to avoid warnings about missing svg files.

At any rate, this has nothing to do with the permission denied problem, so if you have any further problems, open a new discussion.

I will try this. Do I add all those parameters in the same line?

Aarekaz commented 2 years ago

So perhaps we should go back to using an intermediate temporary file to prepare the bootloader...

I'd rather see if #6469 and #6470 solve our problems before we start chucking temporary files into the mix.

Yeah, I agree. It could be that AV is forcing a scan due to the tainted hash of the original bootloader copy (and suppose that it fails the race when we are embedding icon, resources and manifest, so it scans it after that). I wish I could reproduce this, but no luck...

@fatihkabakk @Aarekaz Does the permission-denied problem also occur if you add --debug bootloader and/or --windowed to your PyInstaller command (i.e., when run_d.exe, runw.exe or runw_d.exe are used). EDIT: Or, if you rebuild the bootloader instead of using the pre-compiled ones?

Haven't tried this. Will look into it.

ArtemConstantinov commented 2 years ago

The simple solution is to not give a exe extension to the file at the time when adding bytes to it for fast solving this issue on Windows 10 I replaced lines: 429 of ...\site-packages\PyInstaller\building\api.py from : 

self.name += '.exe'

to :

self.name += '.e' # can be whatever even an uuid4

After all is done, I just renamed the executable file.

p.s. In case if system scream as can not modify file name to .exe just write it to zip archive with relevant name/extension and after unzip it if required

DervishD commented 2 years ago

@ArtemConstantinov thanks for the fix. Effectively that is the problem, the .exe suffix, for some reason. I tried to disable the AV and it still failed until I applied your fix.

And of course, I can confirm this bug using version 4.9 on Windows 10 21H2.

DervishD commented 2 years ago

So perhaps we should go back to using an intermediate temporary file to prepare the bootloader...

I'd rather see if #6469 and #6470 solve our problems before we start chucking temporary files into the mix.

Yeah, I agree. It could be that AV is forcing a scan due to the tainted hash of the original bootloader copy (and suppose that it fails the race when we are embedding icon, resources and manifest, so it scans it after that). I wish I could reproduce this, but no luck...

@fatihkabakk @Aarekaz Does the permission-denied problem also occur if you add --debug bootloader and/or --windowed to your PyInstaller command (i.e., when run_d.exe, runw.exe or runw_d.exe are used). EDIT: Or, if you rebuild the bootloader instead of using the pre-compiled ones?

@rokm I just tried this and the same error happens. The only solution is to get rid of the .exe suffix, because for the life of me disabling the AV doesn't seem to have any effect. Probably Windows Defender does not get disabled, no matter if you tell it so.

DervishD commented 2 years ago

As you can see, the problem has been fixed (or rather, dealt with, as it's not a bug on Nuitka or PyInstaller, but a weird interaction) on Nuitka.

I've made a very crude modification to PyInstaller inspired by the fix in Nuitka, and it works after the first retry, waiting for one second. Of course it is VERY crude and a proper solution should be written because I'm not familiar with the code and I don't know the side effects of adding a retry loop in building/apy.py.

bwoodsend commented 2 years ago

I'd prefer the changing the suffix option to injecting pauses and retries. It sounds less likely spring back at us the next time an AV engine update occurs.

bwoodsend commented 2 years ago

One question to those saying that they've disabled their antivirus: Which did you disable? It's rare that Windows machines only have one. Microsoft always puts theirs on but most laptop providers are normally endorsed to install some other (invariably the most useless) one too. e.g. Dell and Lenovo come with MacAffee installed and I believe that HPs come with AVG.

DervishD commented 2 years ago

A change in suffix and a later rename sounds good, too, as long as it does not fires the race condition in Windows Defender. And yes, that race should not happen in the first place, and it's not PyInstaller fault but I think it is quite a common problem and it's better if PyInstaller handles it.

My only real concern with pausing and retrying is the duration of the pause. To me, one second sounds great, but timing a race condition can lead to problems later, you are right.

DervishD commented 2 years ago

One question to those saying that they've disabled their antivirus: Which did you disable? It's rare that Windows machines only have one. Microsoft always puts theirs on but most laptop providers are normally endorsed to install some other (invariably the most useless) one too. e.g. Dell and Lenovo come with MacAffee installed and I believe that HPs come with AVG.

I only have Windows Defender, which BTW is quite difficult to completely disable, and the problem still persists even when disabling it, adding the involved folders and binaries to the exceptions list, or even killing the service. And it's not a good solution IMHO, users should not have to disable their AV in order to run a program.

bwoodsend commented 2 years ago

And it's not a good solution IMHO, users should not have to disable their AV in order to run a program.

Yeah I agree. I just idealistically say that an AV issue should be fixed on the AV's side or the AV should be replaced instead of ducked and dodged around by any compiler or build tool that happens to want to copy an executable file and then modify it - particularly when the AV is provided by a company with as many resources as Microsoft and the build tool is maintained by two volunteers who don't even use Windows. Rant ranty rant rant...

I think that this magic pause explains things a bit though. Presumably, when PyInstaller first writes the executable an AV scan is triggered. The AV has to open the file in read mode whilst it scans it and, because on Windows attempting to write to a file that is already open by some other application leads to a permission error, PyInstaller hits this error when it tries to start modifying it again. The short delay is presumably just how long it takes for the scan to complete and the AV to close the file so that PyInstaller can start writing to it again. That would at least explain why #6468 did anything at all - after the 500th or so attempt to open the file, the AV scan had finished so the next attempt worked. So I guess, all we really have to do is either keep the file handle open from start to finish (although I don't think that the bit of Windows API that adds the icon allows this) or just accommodate this pause by merging #6468 (preferably with some time.sleep()s).

rokm commented 2 years ago

I don't think magic pause is a solution, because we (re)open the file for modification in several places, and the race can occur at any of them (e.g., this report is about appending PKG to exe, while #6506 is about a later stage, when we modify the build timestamp and set exe chekcsum). I.e., there's no reason to believe that re-scan does not occur after every modification.

I'd also prefer avoiding the delay loops at every point where we (re)open the file, because those involve different APIs and different abstraction levels (raw open, WinAPI, pefile).

I find the idea of changing/removing suffix during the assembly process and then renaming the file the least invasive one. That relies on the current behavior of the AV, though (i.e., picking on the file based on its extension instead of the PE header).

DervishD commented 2 years ago

And it's not a good solution IMHO, users should not have to disable their AV in order to run a program.

Yeah I agree. I just idealistically say that an AV issue should be fixed on the AV's side or the AV should be replaced instead of ducked and dodged around by any compiler or build tool that happens to want to copy an executable file and then modify it - particularly when the AV is provided by a company with as many resources as Microsoft and the build tool is maintained by two volunteers who don't even use Windows. Rant ranty rant rant...

I think that this magic pause explains things a bit though. Presumably, when PyInstaller first writes the executable an AV scan is triggered. The AV has to open the file in read mode whilst it scans it and, because on Windows attempting to write to a file that is already open by some other application leads to a permission error, PyInstaller hits this error when it tries to start modifying it again. The short delay is presumably just how long it takes for the scan to complete and the AV to close the file so that PyInstaller can start writing to it again. That would at least explain why #6468 did anything at all - after the 500th or so attempt to open the file, the AV scan had finished so the next attempt worked. So I guess, all we really have to do is either keep the file handle open from start to finish (although I don't think that the bit of Windows API that adds the icon allows this) or just accommodate this pause by merging #6468 (preferably with some time.sleep()s).

It's not a rant, I totally agree with you, Microsoft should do their laundry here and fix this issue because it's not normal at all you cannot modify an executable you just created. This said, we both know they are not going to fix it anytime soon and meanwhile it's PyInstaller the one paying the potential consequences by losing users who find this problem.

I agree with @rokm and as long as the AV does not look at the headers but the suffix, it's better to just avoid that suffix until the very last file write. I think it's quite pragmatic and it's easier to implement.

bwoodsend commented 2 years ago

Changing the suffix it is. I'll try to resist the urge to set the temporary suffix to something immature like .msdefenderisstupid.

DervishD commented 2 years ago

Changing the suffix it is. I'll try to resist the urge to set the temporary suffix to something immature like .msdefenderisstupid.

Can I +1 that temporary suffix? Because I utterly love it, maybe out of immaturity on my part but 🤣

Now seriously, thanks a lot for dealing with this.

bwoodsend commented 2 years ago

Can someone with this issue give the fix a test?

pip install https://github.com/bwoodsend/pyinstaller/archive/refs/heads/dodge-exe-suffix.zip

Then try building something as normal.

DervishD commented 2 years ago

@bwoodsend , your fix is quite similar to the one I was testing, and it works perfectly in a computer where I could reproduce the problem all the time. Thanks A LOT for the fix! That was fast!

If you need me to perform any other test, I've set up a separate venv for testing this in one of the machines where I can reproduce the problem.

DervishD commented 2 years ago

Can someone with this issue give the fix a test?

pip install https://github.com/bwoodsend/pyinstaller/archive/refs/heads/dodge-exe-suffix.zip

Then try building something as normal.

BTW, I just noticed a typo in building/api.py, and since you will be modifying related code:

            # Embed the manifest into the executable.
            if self.embed_manifest:
                logger.info("Emedding manifest in EXE")
                self.manifest.update_resources(build_name, [1])

Notice Emedding rather than Embedding. If you prefer I open a new PR for this issue so it is addressed separately, just let me know.

bwoodsend commented 2 years ago

I wouldn't bother chasing spelling errors. PyInstaller is littered with them. One day I'll get bored enough to run codespell on the whole repo and fix the lot.

bwoodsend commented 2 years ago

Fixed in #6629. Next release will probably be in a few days so you shouldn't have to wait long.