How to properly bundle using pyinstaller?

[sudoLife]

Hi,

I have been going through the docs for the past week and still unable to figure it out properly.

My project structure is like this (simplified) : main.py my_api/

• server.py
• client.py
• some_other_stuff_server_depends_on.py

I (presumably correctly) add the server executable like so: --add-data ".\virt\Lib\site-packages\msl\loadlib\server32-windows.exe;." , but at runtime the server cannot start because the client can't find the server file in a bundle.

I don't think I can just use --add-data to simply add server.py because then I would also have to manually add all the dependencies etc, so I feel like there's a better way. Maybe I misunderstand the way the bundling works.

I have prepared a simple test project to illustrate my setup, which is in the echo_project.zip.

I would really appreciate some pointers. And great work with the library, it's very useful!

Platform: Windows 10 Python version: 3.11 Virtual env: yes

[jborbely]

Thank you for providing an example.

The concept to remember (I forget often, like I did when I started to debug your issue) is that server32-windows.exe starts a completely isolated Python interpreter and server32-windows.exe is itself a frozen application that was built using pyinstaller --onefile so when msl-loadlib starts the 32-bit server, all of the 32-bit stuff gets unzipped inside a C:\Users\username\AppData\Local\Temp\ subfolder. From the 32-bit server's point of view, my_api does not exist.

Typically, when you don't freeze your code you would tell the 32-bit server where your my_api package is located via something like

class EchoClient(Client64):

    def __init__(self):
        super(EchoClient, self).__init__(
            module32='my_api.echo_server',
            append_sys_path=os.path.join(os.path.dirname(__file__), '..'))

however, once you freeze main.py the my_api package is not available to an external interpreter (i.e., the interpreter inside server32-windows.exe) since the my_api package is now contained within main.exe.

When you run pyinstaller <your-options> main.py, pyinstaller creates the dist\main directory that contains a main.exe file and the _internal folder. The --add-data option adds server32-windows.exe inside _internal (so you probably did add the 32-bit server correctly). The problem occurs when msl-loadlib starts the 32-bit server and the 32-bit Python interpreter cannot find your my_api package because it is inside of main.exe.

There are a few options:

1. Do as you suggest, use --add-data to add the necessary files from my_api into _internal for the 32-bit Python interpreter to be able to import them.
2. Create a custom server32-windows.exe that embeds your my_api package inside the 32-bit server (see the docs for more details)
3. ?? (I'll think about it some more)

[sudoLife]

Thank you for a prompt reply @jborbely, I really appreciate it.

1. Do as you suggest, use --add-data to add the necessary files from my_api into _internal for the 32-bit Python interpreter to be able to import them.

I'll try it out, but it will probably not be the optimal solution for larger projects using msl-loadlib for this kind of application. I was thinking if we come up with an appropriate method here, we might add it to the docs.

Create a custom server32-windows.exe that embeds your my_api package inside the 32-bit server (see the [docs (https://msl-loadlib.readthedocs.io/en/stable/refreeze.html) for more details)

I read through that page quite a few times, but found it unclear because it has no mention of why I would want to refreeze the server and what benefits it would entail. It has no mention of embedding a custom module, though in the hindsight I suppose the hint about pip install numpy and the like is pointing in the right direction. Maybe it needs to be a bit more verbose on that topic for dummies like me 😄

So, continuing this train of thought, you're suggesting that I make my_api a proper python package and then "pip install" it from, let's say, github? I would then have to add setup.py and other features of a package for it to work, I guess? I might just do this, and it probably would also fit the purposes of larger software projects.

If I'm making a wrong assumption somewhere, please don't hesitate to correct me. And thank you very much!

[jborbely]

I think refreezing the 32-bit server with your custom package embedded will be the easiest, just not easy with the way msl-loadlib currently exposes the API for an end user to freeze the server. I will push some changes to help make this simpler for people -- hopefully in the next day or so.

Converting your my_api example into a pip-installable package is fairly easy. No need to publish your package on PyPI nor GitHub. In your root folder echo_project just create a file called pyproject.toml and paste the following in it (see here for other metadata you can include in this .toml file, I added the minimum metadata for your example)

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "my_api"
version = "0.1.0"

[tool.setuptools.packages.find]
include = ["my_api*"]
namespaces = false

[tool.setuptools.package-data]
my_api = ["cpp_lib32.dll"]

then, in a terminal, go to your echo_project directory and

pip install .

or, to make it an editable install

pip install -e .

[sudoLife]

Oh, I see. I'll probably shift the structure a bit so that the library itself contains this .toml file (it's pretty much like setup.py in that sense I guess), which will follow the same folder structure as for PyPI projects.

I will push some changes to help make this simpler for people -- hopefully in the next day or so.

Okay, that's great! I'll watch out for the mentions of this issue then :) Is there any way I can help?

[jborbely]

it's pretty much like setup.py in that sense I guess

Yes, pyproject.toml is considered the new standard and setup.py is legacy (see PEP 517 and PEP 621)

I have push changes to the main branch. You no longer need to be able to make your echo project be pip installable (the files in your original zip file are fine). Here's what to do to test the changes:

1) install 64-bit and 32-bit versions of Python 3 (download from here) 2) in the echo_project directory I created two virtual environments, one for each bitness of Python (I named the environments venv32 and venv64) 3) activate your 32-bit environment, install dependencies and create the 32-bit server with your my_api package embedded

    C:\...\echo_project>venv32\Scripts\activate
    (venv32) C:\...\echo_project>pip install pyinstaller git+https://github.com/mslnz/msl-loadlib.git
    (venv32) C:\...\echo_project>freeze32 --packages my_api --data my_api/cpp_lib32.dll:my_api

4) deactivate your 32-bit environment

    (venv32) C:\...\echo_project>deactivate

5) activate your 64-bit environment, install dependencies and freeze your main.py script

    C:\...\echo_project>venv64\Scripts\activate
    (venv64) C:\...\echo_project>pip install pyinstaller git+https://github.com/mslnz/msl-loadlib.git
    (venv64) C:\...\echo_project>pyinstaller --add-data server32-windows.exe:. main.py

6) deactivate your 64-bit environment, then run main.exe

    (venv64) C:\...\echo_project>deactivate
    C:\...\echo_project>cd dist\main
    C:\...\echo_project\dist\main>main.exe
    ((8, 5, 'test'), {'kwarg': 3})

Is there any way I can help?

Yes, thanks for offering. Could you please review the changes that I made to the docs and let me know if these instructions would have been helpful for you a week or so ago.

[sudoLife]

Hi @jborbely,

I have tested the setup you suggested and it seems to work! I will further test it on my actual project since it has more dependencies, but I don't expect there to be any problems. Great work!

I have also reviewed the changes you have made to the docs and yeah, they make much more sense now. Maybe a word on how a custom server module needs to be included and found by the server32-windows.exe is warranted as well, although the current docs might honestly be enough. I can't fully tell because things always seem obvious in hindsight.

I wonder if there's a place in the docs to just show how the test project we discussed here can be frozen and used as a standalone application. One concrete example together with the instructions could be useful. Maybe add it at the end of the doc page you mentioned? I could help out if you want.

[jborbely]

Thanks for confirming that the changes worked for your test project. Hopefully, it works for your actual project. If not, reach out. I plan to make a new release in the next week or so that contains a suitable fix for this issue. So another confirmation from you in regards to your actual project you would be useful.

Your suggestions to add a few more bits of the stuff we discussed to the docs is good. In particular, show an example on how to freeze msl-loadlib within a users application using something like (perhaps exactly like) your example _echoproject as a visual help instead of just instructions.

I appreciate that you raised this issue, I think the end result of the code is much better than what it was.

[sudoLife]

So another confirmation from you in regards to your actual project you would be useful.

Confirming that it works for my actual project as well!

I appreciate that you raised this issue, I think the end result of the code is much better than what it was.

I'm glad it was mutually useful. Should I write the instructions & tidy up the echo project for the docs?

[jborbely]

Should I write the instructions & tidy up the echo project for the docs?

No need for you to spend your time on this mundane task, thanks for the offer. I'll get things sorted shortly...

[sudoLife]

Hey @jborbely last but not least, the server must be present in the __init__.py of the my_api as an import for the freeze32 to include it. Could be obvious, but is probably worth mentioning anyway.