Keep Python files in clear text

[ricardo-reis-1970]

Keep Python files in clear text

I have a medical device app in Python that I wish to deploy in other than:

• install python, git, visual studio compiler (hours and GB wasted)
• git clone
• poetry install

Enter cx-freeze. Now, being a novice to Python compiling / packaging, I'm slowly adjusting to stuff ending up in different locations and using different techniques to find my own location (the whole if getattr(sys, 'frozen', False): thing).

Config files

My app configuration depends on both JSON and Python dict files. The latter are very convenient because all they require is an import. The Python files contain things like text-to-speech messages and global parameters such as the clinic name (to appear in medical reports), shared folders' paths, etc.

On their current state (clear text deployment), these files can easily be edited by the end customer.

The issue

When I pack my app, all the Python files become pyc. Obviously these can no longer be edited. My options include:

• build a separate installer per client: slightly impractical;
• supress all configuration: need I comment on this?
• convert all Python config files to JSON: this is pretty deep in the code and would require some major refactor.

The (partial) solution

I managed to configure the MSI to preserve some files, with rules like:

build_exe_options = {
    'include_files': [
        ('my_app\\config\\main_config.py', 'my_app\\config\\main_config.py'),
        ('my_app\\config\\ReceptionParams.json', 'my_app\\config\\ReceptionParams.json'),

but these files, although being preserved, have mixed behaviour. While the JSON file is explicitly open and works fine, the main_config.py file is ignored, taking second chair to the compiled counterpart lib\my_app\main_config.pyc. I tried deleting this pyc file, hoping that the app would revert to the py version, but no luck there. They are in different folders.

The solution that actually works

This is what I would hope to learn here. I suspect that the solution could be something like including the my_app\config folder in the path, or doing something like:

    'replace_paths': [
        ('my_app\\config', 'C:\\shared\\workspaces\\my-app\\my_app\\config')
    ],

but this one did not work.

Desktop:

• Platform information: Windows 10
• OS architecture: amd64
• cx_Freeze version: 6.9
• Python version: 3.9

Additional context

We don't thank open source authors enough, but we complain about things like documentation, so please hear me out.

The documentation is very shallow. There are lots of entries that are not obvious and the examples are simply not enough. The ones in the samples folder of the repo are all very basic. I have an app with 94 dependencies, and it's not even big.

I'm struggling to have this operational. Some dependencies are pulled in in non-standard ways, so obviously they need to be declared in setup.py, but there are tens of features that are difficult to understand unless you actually know about packaging, making cx-freeze a non-entry-level tool and therefore more difficult to adopt.

Other than this issue with exposing some selected Python files, I would like for instance to be able to reduce all my own pyc files to pyd, so that they would be definitely and irreversibly compiled. pyc files are still possible to read.

I have long experience in producing technical documentation and training materials. My English is very high end and just like you, eu falo Português nativo (from Portugal, in my case). I can definitely help with the readthedocs part, but I need to understand it first.

[marcelotduarte]

I managed to configure the MSI to preserve some files, with rules like:

I think the solution for your case goes this way, with an addition: you have to exclude this file from the compilation. That is, include it as data and exclude it as a module, using the excludes option. In you example, excludes=["my_app.config.main_config"]

replace_paths

replace_paths is only used to not show the source of a module or package on your system. If there is an error, the traceback will show the entire path and can, for example, expose the username.

[marcelotduarte]

The documentation is very shallow

I agree. I've been using cx-freeze for a few years, I started contributing about 3.5 years ago when I realized that it wasn't keeping up with the evolution of python and I became a maintainer for 2.5 years ago. And for my needs it has a very good, very clear conception.

The ones in the samples folder of the repo are all very basic

The samples are simple today, because I and some contributors have been adding hooks to avoid using various options in the setup. I'll give you an example, when I started using bcrypt, cryprography and others, I had to use several includes and packages, today it's automatic.

I would like for instance to be able to reduce all my own pyc files to pyd

See

I have long experience in producing technical documentation and training materials.

I would love for you to contribute. I'm not good with documentation, but I like to see good documentation. A starting point, is that cx_Freeze is based on setuptools (previously on distutils), which is perhaps why the documentation is minimal, because cx_Freeze is based on python concepts for modules and packages and the packaging of setuptools, however at the level above as setuptools is used to package a library, cx_Freeze packages all libraries in use and also python shared libs for example.

eu falo Português nativo (from Portugal, in my case).

E ai beleza? (A common greeting in BR)

If you are going to contribute later we can arrange a way to clarify some questions in Portuguese, by email, for example.

[ricardo-reis-1970]

Oi cara!

I know "E aí beleza" because we have tons of Brasillian† novelas ever since "Gabriela cravo e canela" (1976).

Regarding documentation, I am good and I'm not afraid to say it out loud. It's a side effect of being old: we can say whatever we want. In this case, what I want to say is: vamos fazer isto, porque cx_Freeze é muito bom.

I managed to sort the clear Python thing today, but it was a fight. Instead of the excludes key (which did not work for me, as it caused a lot of ModuleNotFound errors), I did something of a preload and it worked for both live and frozen. Had this failed, I would have been forced to conver all my Python config files to JSON. I'm thinking that this could be a samples candidate.

Regarding the .pyc files — which indeed are "decompilable" — I once suceeded in creating a .pyd of my entire app, using Nuitka. Since it is all written in Python, I suppose it's something native to get these files, which are sourcecode-safe. If this could be imported into cx_Freeze it would be perfect. I also succeeded in generating .pyd files with cython, but these were small examples and none of them gave me the possibility of packaging an MSI.

Other things I'd like to do include adding other installers. For instance, my application requires Tesseract and wkhtml2pdf. I wish I could put all these in the same installer and have a one-stop-shop to my application.

My email is in my LinkedIn page, or it is ricardo.reis.1970@gmail.com. O que for mais fácil.

---

† Eu recuso-me terminantemente a escrever "Brasil" com "z", o mesmo com Açores e Moçambique. E não é por nenhum tipo de colonialismo, é mesmo porque para mim os Ingleses podem seguir o conselho do centrocampista Fábio Rochemback e ir "tomar no ..."

[marcelotduarte]

Had this failed, I would have been forced to conver all my Python config files to JSON.

There an another format to config files that I hope to use is TOML (toml or tomli packages) that is easy for the users.

using Nuitka

Some time ago I tried it and have no success with my project. Another developer reported the same. I use Cython in some modules, using build_ext. Slow down the build. To package it, you should declare all imports with packages or includes. Work!

MSI

To include files in a MSI buid, use:

1. Add 'skip_build': True option to the bdist_msi_options.
2. Run python setup.py build to generate the exe.
3. Add files.
4. Run python setup.py bdist_msi to build the windows installer from the previously built exe.

[ricardo-reis-1970]

Including files

To include files in a MSI buid, use:

1. Add 'skip_build': True option to the bdist_msi_options.
2. Run python setup.py build to generate the exe.
3. Add files.
4. Run python setup.py bdist_msi to build the windows installer from the previously built exe.

In the end, I managed to do it with build_exe_options['include_files'] and it seems to be working. So now I have my config files in Python, meaning that all they need is to be imported. TOML is also a good idea, no doubt, but like JSON it must be manually loaded and parsed, whereas Python files with dictionaries look just like JSON but are native.

In any case, this is a great tip!

Installing together

However, when I mentioned Tesseract and wkhtml2pdf, I did not mean just put those files in there. I meant that my installer could just install them. What I was hoping for was to have the MSI installing my app and then proceeding with the other installers, one by one.

I've seen this in countless installers but I don't know if this is possible here. Is it?

EDIT: you mentioned this is based on setuptools. Is there here any feature like the postinstallation script?

Path

On another topic, my installer adds vcruntime140.dll under its \lib\sklearn\.libs folder, but then it complains that this file is missing. When I copy this file to a folder in the system PATH it then works, but again I'm trying to have an automatic MSI that requires some manual tweaking.

On the manual, it is mentioned a build_exe_options['path'] entry, but the explanation is: just:

comma-separated list of paths to search; the default value is sys.path

How should we do in order to add folders to the PATH? I'm trying to add the folder where that .dll is, so that the installation won't require manual fixes.

EDIT: I just used include_msvcr and this is sorted.

[marcelotduarte]

cx_Freeze 6.11 has just been released. pip install --upgrade cx_Freeze Assuming this has been resolved. If you had issues please open a new issue.