Closed LeonarddeR closed 5 years ago
CC @CToth, @TSpivey and others.
@leonardder commented on 8 Jun 2018, 08:16 CEST:
cx_Freeze
Compared to PyInstaller and py2exe, there are some drawbacks:
- There is nou built-in support for manifests and settings the UI Access privilege level. We will probably have to implement this manually, either by expanding setup.py or by building the base executables ourselves.
- By default, cx_freeze assumes that all python stuff is in a lib sub directory (i.e. in lib/library.zip) rather than in the root of the folder where the executables live. This is not easily customisable and seems to be hard-coded behaviour.
I guess all this can be solved by building our own base executables, adding the manifests to these. This integrates quite well within the scons environment and might even lead to less complex code as the current code in setup.py.
There is currently no Python 3.7 support (see https://github.com/anthony-tuininga/cx_Freeze/issues/399)
Explore the possibility of using Nuitka. It continues to improve, now has python 3.7 support, and the author is extremely extremely interested in working with people who have issues.
On Aug 6, 2018, at 12:01 PM, Leonard de Ruijter notifications@github.com wrote:
@leonardder commented on 8 Jun 2018, 08:16 CEST:
cx_Freeze
Compared to PyInstaller and py2exe, there are some drawbacks:
There is nou built-in support for manifests and settings the UI Access privilege level. We will probably have to implement this manually, either by expanding setup.py or by building the base executables ourselves. By default, cx_freeze assumes that all python stuff is in a lib sub directory (i.e. in lib/library.zip) rather than in the root of the folder where the executables live. This is not easily customisable and seems to be hard-coded behaviour. I guess all this can be solved by building our own base executables, adding the manifests to these. This integrates quite well within the scons environment and might even lead to less complex code as the current code in setup.py.
There is currently no Python 3.7 support (see anthony-tuininga/cx_Freeze#399)
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
@ctoth: Thanks for pointing this out. Here is the user manual.
@feerrenrut and @michaeldcurran, what are your thoughts regarding this? I assume that this issue has to be settled before Python 3 builds can be provided to end users.
Py2exe seems to have been foreked and is on pypy supporting Python 3.3 and newer: https://pypi.org/project/py2exe/
@michaelDCurran commented on 7 aug. 2018 07:09 CEST:
Py2exe seems to have been foreked and is on pypy supporting Python 3.3 and newer: https://pypi.org/project/py2exe/
I'm afraid that the most recent version is from 2014. Python 3.6 contained major byte code format changes and therefore won't work with this version of py2exe. This applies to the version on pypy as well as github repositories I've been able to find quickly. The py2exe sourceforge page doesn't provide more hope that this project will ever be developed further.
Ah, Fair enough.
I think we will have to try a few of these to really understand the trade-offs.
I assume that this issue has to be settled before Python 3 builds can be provided to end users.
We could try to stick with py2exe and upgrade to python 3.3. After the dust settles from this we can look to swap to a new tool. Then upgrade to a later version of python.
Hi, I advise against this route, as Python 3.3 is no longer supported by Python Software Foundation. See https://www.python.org/dev/peps/pep-0398/. Thanks.
Thanks for pointing that out @josephsl, I'll hide this section of the conversation, since its a bit of a red herring.
@ctoth commented on 6 Aug 2018, 20:41 CEST:
Explore the possibility of using Nuitka. It continues to improve, now has python 3.7 support, and the author is extremely extremely interested in working with people who have issues.
I've played a bit with Nuitka. It is a really impressive and interesting piece of software.
A major downside is that all python modules are converted to c++ code and than recompiled. Though a major performance boost is promised, it takes a significant time to run the distribution creation process. I also got errors like this:
fatal error C1002: compiler is out of heap space in pass 2
It might help when I start compiling again with less concurrent jobs. But the time increase is concerning, as it will take much more time to create try builds, etc.
Hmm, integrating the creation of a custom cx_freeze base in our scons workflow isn't as easy as I thought, it causes weird string conversion warnings I don't expect, probably because I'm missing some essential requirements to have scons build stuff that should be linked against the python library.
I filed https://github.com/anthony-tuininga/cx_Freeze/issues/413
I did some additional research into pyinstaller,, but at least its default configuration shows full paths in trace backs and raises disgusting PyInstallerImportErrors. Furthermore, it seems it is either a nvda.exe with everything included, or a nvda.exe with nothing included. Last but not least, the installation creation process doesn't feel very pythonic.
It seems it is either something that is ugly but has features (pyinstaller), or something that is much more elegant but has less features than py2exe and hasn't received official releases during last year (CX_freeze). The master branch of the latter is python 3.7 compatible, though.
If I may, I have been working on resurrecting py2exe and make it compatible with the latest versions of python. If you want, you could give a try at the prebuilt wheels that I just released here: https://github.com/albertosottile/py2exe/releases/tag/v0.9.3.0
Please, let me know if you find any errors when packaging NVDA. This is a big project and I am sure it will provide a good benchmark for these new wheels.
This is really interesting. We'll keep this in mind as soon as the Py3 transition begins.
For everyone who is interested in this discussion, I've updated https://github.com/nvaccess/nvda/issues/8375#issue-330532522 to contain an in depth overview of our options to replace Py2exe 0.6.9.
@albertosottile: You seem to be pretty interested in our possible efforts to freeze NVDA with your fork. It might also be the least effort on our end. However, it would help to know what your goals are with the project, and whether you're willing to maintain it in the near and far future. Would you be able to elaborate on this?
I think a missing requirement is to collect any dependent python extention dlls (.pyd files). I don't think I saw that in your list?
I think a missing requirement is to collect any dependent python extention dlls (.pyd files). I don't think I saw that in your list?
You're correct, I'll at that.
You seem to be pretty interested in our possible efforts to freeze NVDA with your fork. It might also be the least effort on our end. However, it would help to know what your goals are with the project, and whether you're willing to maintain it in the near and far future. Would you be able to elaborate on this?
Thanks for the tag. I read the first post carefully and it seems that there are several potential issues in using the forked py2exe
for your project, both technical and political. I will try to address them hereafter.
First of all, I never tried manifests and to be honest I was not even aware of this feature in py2exe. As far as I know, this could be the most serious technical hurdle you (we?) will probably have to face if you want to keep your existing script file. Alternatively, I am confident the same results could be achieved using resources, but that will need a rewrite of the script.
Second, I am not worried about the wxPython issue, since applying a workaround from your side is easy and, as I wrote here, it might not even be necessary in your case.
Thirdly, as you correctly noted, the fork is not listed on PyPI. This is also related to the small number of watchers/stars/forks. Honestly, I wanted the project to gain more traction before contacting the original maintainers and/or the PyPI team and get noticed. While it is true that the codebase seems to work now, I strongly believe that maintaining a project like py2exe
is not a job for one single person. (Side note: look at what is happening to PyInstaller now, they barely have the time to fix the most urgent issues, let alone implementing new features). Hence, I do agree with your worries about the future maintenance of py2exe
, not because I am not willing, but because I am totally sure that any effort from me alone will not be enough. Now, the way I see it, the more developers adopt this new py2exe
, the more traction the project will gain. So, it would be a great deal if NVDA could use it, thus my interest in this migration. Nevertheless, I will completely understand if you would decide to go for an alternative with wider adoption.
Wrapping things up, I think a good way to proceed for you will be to experiment with the forked py2exe
and see how far you can go using that, without applying excessive efforts on your existing codebase. In this way, if things work out, you will have a huge return for a quite small investment. Eventually, if it is possible to iron out all the issues without writing many fixes, you could even release a new version based on this py2exe
, helping me gaining some traction, hence increasing the chances of its survival. Otherwise, you will still have time to look into the alternatives without much to regret for having tried.
Just a quick update. I've been able to create a distribution using the py2exe fork that started properly. There are many things to fix and improve on our end, but what I've been able to accomplish in under two hours is really impressive.
I will give a more thorough update halfway next week.
@feerrenrut: I'm getting the following error from py2exe when trying to integrate images/nvda.ico into the executables:
I'm very unfamiliar with the ico format. Is it true that there could be multiple images embedded in one ico? Before I continue investigation of this, it might help to know whether the ico file is really valid.
Reply to https://github.com/nvaccess/nvda/issues/8375#issuecomment-492670031 from @albertosottile
First of all, I never tried manifests and to be honest I was not even aware of this feature in py2exe. As far as I know, this could be the most serious technical hurdle you (we?) will probably have to face if you want to keep your existing script file. Alternatively, I am confident the same results could be achieved using resources, but that will need a rewrite of the script.
There is some commented out code in py2exe regarding manifests. I think I'll first try to have a look at that code in order to see whether it can be reused. If so, I can file a pr against your fork to have it included.
Second, I am not worried about the wxPython issue, since applying a workaround from your side is easy and, as I wrote here, it might not even be necessary in your case.
First investigations indeed seem to prove that we aren't going to suffer from this issue.
Thirdly, as you correctly noted, the fork is not listed on PyPI. This is also related to the small number of watchers/stars/forks. Honestly, I wanted the project to gain more traction before contacting the original maintainers and/or the PyPI team and get noticed. While it is true that the codebase seems to work now, I strongly believe that maintaining a project like
py2exe
is not a job for one single person. (Side note: look at what is happening to PyInstaller now, they barely have the time to fix the most urgent issues, let alone implementing new features). Hence, I do agree with your worries about the future maintenance ofpy2exe
, not because I am not willing, but because I am totally sure that any effort from me alone will not be enough. Now, the way I see it, the more developers adopt this newpy2exe
, the more traction the project will gain. So, it would be a great deal if NVDA could use it, thus my interest in this migration. Nevertheless, I will completely understand if you would decide to go for an alternative with wider adoption.
I don't think that a lack of watchers and forks will hold us back from using your fork. IN fact, we've been using Py2exe 0.6.9 for over 10 years, and it is still pretty stable. Having said that, of course it would be good for us as well if your project gains more traction. May be NVDA can give this a boost somehow.
There is some commented out code in py2exe regarding manifests. I think I'll first try to have a look at that code in order to see whether it can be reused. If so, I can file a pr against your fork to have it included.
That's great news, and of course a PR would always be welcome.
Ico files contain multiple images suitable for various common display scenarios such as thumnails in File explorer, Desktop icon, Start menu icon, system tray icon etc. With newer Windows Operating Systems, the number of image sizes has gone up yet again.
However, I can't completely guarantee that there is no maximum amount of images aloud in an ico file. To my knowledge our ico file is valid.
I couldn't found such a maximum for images in an ico file.
I've filed https://github.com/albertosottile/py2exe/pull/9.
There are two major things I still have to look into:
I've filed albertosottile/py2exe#9.
PR accepted. Will be included in the next release.
There are two major things I still have to look into:
- The system dll routine has changed. System dll's like api-ms-win-crt-convert-l1-1-0.dll are ending up in the distribution
This occurs also with PyInstaller, so I assume the change was in the Python ABI, hence that is here to stay. The solution is to delete the api-*.dll
files from the distribution after packaging. This can be done efficiently in the setup.py
script, see e.g. here (L638).
@leonardder should this issue be closed now that the referenced pull request has benn merged?
Closed with #9605
Follow up of https://github.com/nvaccess/nvda/issues/7105#issuecomment-395424373
Background
For years, NVDA has used Py2exe to package Python code into something that is executable on a system that doesn't have Python installed. For Python 2.7, we have been at Py2exe 0.6.9 (introduced in 2008) for a long time. For Python 3, a new version has been introduced in 2014 which looks like a rewrite. The last official release (0.9.2.2) was introduced in 2014.
Starting with Python 3.6 (end 2016) however, Python contains a backwards incompatible change of the byte code format that is not supported by the official py2exe.
Note that many parts of the build process of NVDA are handled by SCons. This includes:
In summary, Scons deals with everything that is required to run NVDA from source, Py2exe is not involved in that process. Py2exe gets called as soon as a distribution of NVDA is created. The portable distribution that Py2exe creates is then packed into a launcher using NSiS.
Requirements
Before deciding on what packaging tool to use, this section lists the requirements for a packaging tool, based on the work that is now performed by Py2exe.
Byte compile every source file into a .pyc or .pyo file. Note that for Python 3, only the *pyc files remain.
Byte compile every Python dependency (comtypes, pyserial, etc.) required to run NVDA.
Collect Python extension modules (*.pyd files)
Bundle the several byte compiled files into a library.zip file in the main folder of the distribution.
Exclude several unneeded modules
Create executables from certain source files:
Bundle a manifest with the executables. Theoretically, this can also be done using SCons, though in contrast to what I thought earlier, there is no code within the current scons structure of NVDA that can be reused for this.
Make sure that the UIAccess flag is added to the manifest of nvda_uiAccess.exe and nvda_eoaProxy.exe. Strictly spoken, this is code that has been added to our subclass of build_exe.py2exe in order to inject the UIAccess flag into the manifest created by py2exe. However, droppign py2exe in favour of another tool might force us to completely rewrite this logic.
Embed the NVDA logo into the executables as icon.
Set several version info flags in the version info resource of the executables.
Collect several system dll's not available on every system. This includes visual C++ redistributables.
Collect data files created using SCons, such as several libraries, language files, documentation, etc.
Alternatives
According to research and ideas from others, there are four packaging/freezing options that can be used with Python 3. The following sections provide short descriptions along with pros and cons.
Py2exe fork by @albertosottile
See https://github.com/albertosottile/py2exe
Thankfully, @albertosottile contacted us in https://github.com/nvaccess/nvda/issues/8375#issuecomment-476612491, commenting that he picked up Py2exe where it has been abandoned in 2014. This resulted into a release of Py2exe 0.93.0. This version is said to work with Python 3.6 and 3.7.
Pros
Cons
Requirements scheme
PyInstaller
PyInstaller is an alternative tool to build executables. It also supports byte code encryption, though that seems not very useful in the case of NVDA.
Pros
Cons
More importantly, I've seen ridiculous tracebacks with one of my test applications referring to pyinstaller:
It looks like every import is somehow redirected to pyInstaller. I'm pretty sure this will cause issues with our use of comtypes com interface modules, globalPlugins, etc. which rely on appending paths to the module path variable. Showing raw paths to the python directory also looks a bit cheap, but I recall that with py2exe, we also had to work around this.
Requirements scheme
Here is the official features page from PyInstaller
cx_Freeze
This is an alternative that integrates well with distutils. Note that it is far from compatible with py2exe, so setup.py would still require an overhaul. Running cx_Freeze also results into a package structure that is very similar to the one py2exe creates, including a librar .zip file in the lib sub directory. The default settings do not create a zip file, but that can be customized easily.
cx_Freeze is hosted on GitHub. We can include it as a separate submodule. Though it requires some c code to be compiled, that shouldn't be that much of a problem and takes less than 20 seconds, so that doesn't justify an inclusion in misc deps in my opinion.
Pros
Cons
The last commit for the project dates from september 2018. There are 260 open issues, including two from myself:
Both issues haven't gotten any attention.
Requirements scheme
Nuitka
Suggestion by @ctoth in https://github.com/nvaccess/nvda/issues/8375#issuecomment-410810518
Nuitka chooses a completely different approach. Instead of compiling the Python code to byte code and bundling a loader with it, it converts all code to c and then compiles it using VC++ and links it against the Python library. It also claims to be faster than CPython.
PROS
Cons
Requirements scheme
HO to proceed?
Based on the above outline, I propose the following order of investigation of our options:
Note that option 1, 2 and 3 could be worked on in parallel. I'm tempted to consider Nuitka being out of scope for now.