marcelotduarte / cx_Freeze

cx_Freeze creates standalone executables from Python scripts, with the same performance, is cross-platform and should work on any platform that Python itself works on.
https://marcelotduarte.github.io/cx_Freeze/
Other
1.28k stars 210 forks source link

Multiprocessing Manager Class Causes Infinite Loop on Mac OS (intel) #2435

Closed IsaiahCoroama closed 1 week ago

IsaiahCoroama commented 3 weeks ago

Bug Description When I try to create a multiprocessing.Manager() object, even with freeze_support, it causes an infinite loop.

Steps To Reproduce Create a main.py file.

is_darwin = True # This is determined by a module that does not affect the results.

def main():
    ...

if __name__ == "__main__":
    if is_darwin:
       import multiprocessing; multiprocessing.freeze_support()

       print("Creating Managers")
       manager_1 = multiprocessing.Manager()
       print("Finished Creating Manager #1")
       manager_2 = multiprocessing.Manager()
       print("Finished Creating Manager #2")
       manager_3 = multiprocessing.Manager()
       print("Finished Creating Manager #3")
       print("Finished Creating ALL Managers.")

    main()

Create a setup.py file.

from cx_Freeze import setup, Executable

build_exe_options = {
    ["zip_include_packages"]: []
}

setup(
    name="main-demo",
    version="0.0.0",
    options={"build_exe": build_exe_options},
    executables=[Executable("main.py", base="console")]
)

Run the setup file with the following configuration:

python setup.py bdist_mac

Run the generated application file:

./build/main-demo-1.0.0.app/Contents/MacOS/main

Expected behavior From the demo, it prints "Creating Managers" repeatedly indefinity.

System Information:

Additional context Within the project, I am using managers to share data between three separate processes. These processes run a loop attached to a manager.Event() object. In the loop, it checks if a boolean value within a manager.dict() object is True. If it is, it will call a tkinter function; these consist of filedialog.askdirectory(), messagebox.showerror(), and simpledialog.askstring(). Then, it sets the output to a value within the manager dictionary. The process is then repeated with a few other steps. If you would like, I can post the full source code, but currently, the issue only remains with creating the first manager object. I have gotten this to work with PyInstaller, but I need to create a universal2 application, and cx_Freeze has been the closest I have gotten to finally completing everything I need.

Although it states in the official documentation that multiprocessing.freeze_support() only works on Windows, it has fixed this same infinite loop within PyInstaller when I was trying to get that to work.

Disclaimer If there are any misspellings within the code above, that is an accident and not in my actual code. I wrote this issue from my Windows machine all by hand.

IsaiahCoroama commented 3 weeks ago

After looking into the source code of the hook function for the multiprocessing module, I read the comment about multiprocessing.spawn.freeze_support() working on all operating systems and tried running it as well. It fixed the issue. I'm sorry I didn't figure this out sooner and made an issue for it when there was already a fix. I at least hope that this issue helps others who may run into this problem.

is_darwin = True # This is determined by a module that does not affect the results.

def main():
    ...

if __name__ == "__main__":
    if is_darwin:
       import multiprocessing.spawn
       import multiprocessing

       multiprocessing.spawn.freeze_support() # This fixes the issue.

       print("Creating Managers")
       manager_1 = multiprocessing.Manager()
       print("Finished Creating Manager #1")
       manager_2 = multiprocessing.Manager()
       print("Finished Creating Manager #2")
       manager_3 = multiprocessing.Manager()
       print("Finished Creating Manager #3")
       print("Finished Creating ALL Managers.")

    main()
marcelotduarte commented 3 weeks ago

I use 3 tests on Windows, Linux, and macOS. Does this test work for you? https://github.com/marcelotduarte/cx_Freeze/blob/main/tests/test_multiprocessing.py#L53-L63 https://github.com/marcelotduarte/cx_Freeze/actions/runs/9274051105/job/25515543065#step:6:297

IsaiahCoroama commented 3 weeks ago

I reviewed all of your test cases with the following setup.

[sample1.py]

import multiprocessing, sys

def foo(q):
    q.put('hello')

if __name__ == '__main__':
    if sys.platform == 'win32':  # the conditional is unecessary
        multiprocessing.freeze_support()
    multiprocessing.set_start_method('spawn')
    q = multiprocessing.SimpleQueue()
    p = multiprocessing.Process(target=foo, args=(q,))
    p.start()
    print(q.get())
    p.join()

[sample2.py]

import multiprocessing, sys

def foo(q):
    q.put('hello')

if __name__ == '__main__':
    ctx = multiprocessing.get_context('spawn')
    if sys.platform == 'win32':  # the conditional is unecessary
        ctx.freeze_support()
    q = ctx.Queue()
    p = ctx.Process(target=foo, args=(q,))
    p.start()
    print(q.get())
    p.join()

[sample3.py]

if __name__ ==  "__main__":
    import multiprocessing, sys
    if sys.platform == 'win32':  # the conditional is unecessary
        multiprocessing.freeze_support()
    multiprocessing.set_start_method('spawn')
    mgr = multiprocessing.Manager()
    var = [1] * 10000000
    print("creating dict", end="...")
    mgr_dict = mgr.dict({'test': var})
    print("done!")

[setup.py]

from cx_Freeze import Executable, setup

setup(
    name="test_multiprocessing",
    version="0.1",
    description="Sample for test with cx_Freeze",
    executables=[
        Executable("sample1.py"),
        Executable("sample2.py"),
        Executable("sample3.py"),
    ],
    options={
        "build_exe": {
            "excludes": ["tkinter"],
            "silent": True,
        }
    }
)

[sample1.py Run Command]

./build/exe.macos-10.9-universal2-3.12/sample1
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample1

[sample2.py Run Command]

./build/exe.macos-10.9-universal2-3.12/sample2
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample2

[sample3.py Run Command]

./build/exe.macos-10.9-universal2-3.12/sample3
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3

[setup.py Run Command]

python setup.py bdist_mac

Output From the following tests above, all produced the expected output. For sample 3, I created an altered test case by removing the multiprocessing.set_start_method("spawn") and was able to reproduce the infinite spawn recursion as before.

Alternate Test Case For Sample 3

if __name__ ==  "__main__":
    import multiprocessing, sys
    if sys.platform == 'win32':  # the conditional is unecessary
        multiprocessing.freeze_support()
    # multiprocessing.set_start_method('spawn') Removed for testing purposes
    print("Creating Manager")
    mgr = multiprocessing.Manager()
    var = [1] * 10000000
    print("creating dict", end="...")
    mgr_dict = mgr.dict({'test': var})
    print("done!")

[Run Command]

./build/exe.macos-10.9-universal2-3.12/sample3
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3

[Output]

user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample3
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...

Conclusion From the following tests, not calling either mutliprocessing.spawn.freeze_support() or multiprocessing.set_start_method("spawn") will cause the multiprocessing.Manager() to run in an infinite loop.

marcelotduarte commented 3 weeks ago

Your conclusion is interesting, because according to the Python documentation, spawn is the default start method. Then there would be no need to configure it, and in the set_start_method documentation it states "that this should be called at most once". However based on the examples, I think that the alternative (correct?) for sample3 is:

if __name__ ==  "__main__":
    import multiprocessing
    mp_context = multiprocessing.get_context('spawn')
    print("Creating Manager")
    mgr = mp_context.Manager()
    var = [1] * 10000000
    print("creating dict", end="...")
    mgr_dict = mgr.dict({'test': var})
    print("done!")

Also, in the The spawn and forkserver start methods section has "Safe importing of main module" with recommendations to use set_start_method.

Can you check if the default start method is indeed spawn in python and in the executable?

ntindle commented 3 weeks ago

freeze_support from spawn worked for my full use case but freeze_support from multiprocessing directly did not. How odd. I didn't try the trimmed down example, but my work is with Queue and Process

IsaiahCoroama commented 3 weeks ago

[sample4.py]

if __name__ ==  "__main__":
    import multiprocessing
    mp_context = multiprocessing.get_context('spawn')
    print("Creating Manager")
    mgr = mp_context.Manager()
    var = [1] * 10000000
    print("creating dict", end="...")
    mgr_dict = mgr.dict({'test': var})
    print("done!")

Output

user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample3
Creating Manager
creating dict...done!
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3
Creating Manager
creating dict...done!

Alternate Test Case For Sample 4

if __name__ ==  "__main__":
    import multiprocessing
    #mp_context = multiprocessing.get_context('spawn') Removed for test purposes
    print("Creating Manager")
    mgr = multiprocessing.Manager()
    var = [1] * 10000000
    print("creating dict", end="...")
    mgr_dict = mgr.dict({'test': var})
    print("done!")

Output of Alternate Test Case

user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...

The Default Start Method

if __name__ ==  "__main__":
    import multiprocessing
    print(multiprocessing.get_start_method()) # The default start method
    print("Creating Manager")
    mgr = mp_context.Manager()
    var = [1] * 10000000
    print("creating dict", end="...")
    mgr_dict = mgr.dict({'test': var})
    print("done!")

Output

user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
spawn
Creating Manager
spawn
Creating Manager
spawn
Creating Manager
...
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
spawn
Creating Manager
spawn
Creating Manager
spawn
Creating Manager
...

Another Test Case with Getting the Default Spawn Method

if __name__ ==  "__main__":
    import multiprocessing.spawn
    import multiprocessing
    multiprocessing.spawn.freeze_support()
    print(multiprocessing.get_start_method()) # The default start method
    print("Creating Manager")
    mgr = mp_context.Manager()
    var = [1] * 10000000
    print("creating dict", end="...")
    mgr_dict = mgr.dict({'test': var})
    print("done!")

Output

user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
spawn
Creating Manager
creating dict...done!
user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
spawn
Creating Manager
creating dict...done!

Off Topic Question I've noticed that "universal2" is labeled in the name of the generated build_exe directory, but it is not a standalone file. Is there a way to create standalone universal2 executable files using cx_Freeze?

ntindle commented 3 weeks ago

Trimmed example for context:

from multiprocessing import Process, freeze_support

def f():
    print("Hello from cx_Freeze")

if __name__ == "__main__":
    freeze_support()
    Process(target=f).start()
ntindle commented 3 weeks ago

The default in python is spawn and it works if you set it to fork. However, the notice in the FAQ could probably just be updated to mention both freeze support functions without any additional changes

marcelotduarte commented 3 weeks ago

@IsaiahCoroama read this. For "Alternate Test Case For Sample 4" the mgr = mp_context.Manager() was changed to mgr = multiprocessing.Manager() to run?

@ntindle I should change the FAQ or redirect freeze_support in the hook to avoid future questions....

ntindle commented 3 weeks ago

My naive understanding is that you sometimes need one vs the other depending on if you want to spawn or fork which differs by platform

IsaiahCoroama commented 3 weeks ago

@IsaiahCoroama read this. For "Alternate Test Case For Sample 4" the mgr = mp_context.Manager() was changed to mgr = multiprocessing.Manager() to run?

@ntindle I should change the FAQ or redirect freeze_support in the hook to avoid future questions....

Apologies for the confusion. You're right, it should be multiprocessing.Manager() instead of mp_context.Manager(). I'll update the post above with the correct text. Additionally, I've rerun the test, and it yielded the same output as previously mentioned.

IsaiahCoroama commented 3 weeks ago

I've reviewed the information you provided, and I'm utilizing the universal2 version of Python. However, upon running the file command on the generated executables, I noticed they only show the x86_64 architecture. Are you suggesting that the universal2 part of the filename is merely for display purposes and doesn't reflect the actual build generation? Or am I misunderstanding the information you provided?

marcelotduarte commented 3 weeks ago

I don't use macOS, but I know that other people use universal2, and I recently stopped generating universal2 and started generating it again. But from what I understand, the python and wheel packages installed must be universal2 to work.

IsaiahCoroama commented 3 weeks ago

Believe it or not, I don't use Mac OS either; I'm only borrowing a Mac computer because I'm attempting to develop a multi-platform product. I ran the file command on a Python base library like _ctypes.cpython-312-darwin.so, and it displayed support for both x86_64 and arm64, indicating that the built-in libraries are indeed universal2. However, I created a simple script that walks through the exe.macosx-10.9-universal2-3.12 build directory and checks all files architecture. From it, I found that every single library file, files with the .so file extension, were universal2 but the main executable files generated by cx_Freeze were only the native architecture, in my case x86_64.

The File I Used to Check the Architectures of the Generated Binary Files

import subprocess
import os

def check_arch(filename):
    output = subprocess.check_output(["file", filename]).decode()

    if "x86_64" in output and "arm64" in output:
        arch = "universal2"
    elif "x86_64" in output:
        arch = "x86_64"
    elif "arm64" in output:
        arch = "arm64"
    elif "data" in output: # Typically files with the (.pyc) just output data by the file command
        arch = "data"
    else:
        arch = "unknown"

    arch = f"[{arch}]:" # Formatting

    print(f"{arch:<14}{filename}")

def main():
    path = os.path.abspath("multiprocessing/build/exe-macosx-10.9-universal2-3.12")

    for root, _, files in os.walk(path):
        for file in files:
            check_arch(os.path.join(root, file))

For the output, I cut down the path length to make it easier to read.

Output

Console Output
[x86_64]:     /exe.macosx-10.9-universal2-3.12/sample2
[x86_64]:     /exe.macosx-10.9-universal2-3.12/sample3
[x86_64]:     /exe.macosx-10.9-universal2-3.12/sample4
[unknown]:    /exe.macosx-10.9-universal2-3.12/frozen_application_license.txt
[x86_64]:     /exe.macosx-10.9-universal2-3.12/sample1
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_struct.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/array.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_queue.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_blake2.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha1.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_posixshmem.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_hashlib.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/grp.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_socket.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_decimal.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_csv.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_statistics.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha3.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/fcntl.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/Python
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_ssl.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/zlib.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/unicodedata.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_bisect.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_cn.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_multibytecodec.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/libssl.3.dylib
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_posixsubprocess.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/libcrypto.3.dylib
[unknown]:    /exe.macosx-10.9-universal2-3.12/lib/library.dat
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/select.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_lzma.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/pyexpat.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_pickle.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_jp.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_bz2.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_ctypes.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/mmap.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_datetime.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/math.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha2.cpython-312-darwin.so
[data]:       /exe.macosx-10.9-universal2-3.12/lib/library.zip
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_iso2022.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_tw.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_opcode.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_kr.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_heapq.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_md5.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_multiprocessing.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_random.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_contextvars.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_hk.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/binascii.cpython-312-darwin.so
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp861.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp875.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/quopri_codec.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp874.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp860.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_cyrillic.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1140.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp862.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp863.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_32.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_greek.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_8.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp720.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/oem.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp866.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_latin2.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_8.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1026.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp737.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp858.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp864.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp865.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_9.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1256.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp037.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_kr.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/shift_jis.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/big5.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/johab.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/zlib_codec.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1257.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1255.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_16_le.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/gb2312.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp949.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/koi8_t.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp424.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_32_le.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/koi8_u.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/hz.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1254.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1250.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/rot_13.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/kz1048.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/unicode_escape.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_ext.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1251.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/base64_codec.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1253.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/uu_codec.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1125.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/palmos.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/koi8_r.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp437.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1252.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/ascii.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/tis_620.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_32_be.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_roman.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/big5hkscs.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/undefined.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/idna.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_16_be.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/latin_1.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_2004.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/aliases.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_croatian.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/gbk.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/ptcp154.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/raw_unicode_escape.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp950.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_arabic.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1258.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_romanian.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp775.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_iceland.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/shift_jisx0213.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/hp_roman8.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/euc_jis_2004.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_4.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_7.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/charmap.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_turkish.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/hex_codec.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/bz2_codec.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_16.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp869.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp855.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/gb18030.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_5.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_7.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp857.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_14.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/euc_kr.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_15.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/punycode.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mbcs.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp856.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_8_sig.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_6.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_2.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp932.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_1.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_11.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp852.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/euc_jisx0213.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/shift_jis_2004.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_10.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/utf_16.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_3.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_1.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp500.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_2.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/euc_jp.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/mac_farsi.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp850.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso8859_13.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/iso2022_jp_3.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp1006.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/encodings/cp273.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/ctypes/_endian.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/ctypes/__init__.pyc
[unknown]:    /exe.macosx-10.9-universal2-3.12/lib/ctypes/macholib/fetch_macholib
[unknown]:    /exe.macosx-10.9-universal2-3.12/lib/ctypes/macholib/README.ctypes
[unknown]:    /exe.macosx-10.9-universal2-3.12/lib/ctypes/macholib/fetch_macholib.bat
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/sharedctypes.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/connection.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/resource_tracker.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/pool.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/popen_fork.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/heap.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/synchronize.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/context.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/reduction.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/resource_sharer.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/forkserver.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/queues.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/popen_forkserver.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/process.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/shared_memory.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/managers.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/util.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/spawn.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/popen_spawn_posix.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/popen_spawn_win32.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/dummy/connection.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/multiprocessing/dummy/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/urllib/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/urllib/parse.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/zipfile/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/zipfile/_path/glob.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/zipfile/_path/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/xml/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/xml/parsers/expat.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/xml/parsers/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/http/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/http/client.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/_bootstrap.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/abc.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/_bootstrap_external.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/simple.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/_abc.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/readers.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/machinery.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/util.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/_adapters.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/abc.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/simple.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/readers.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/_legacy.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/_common.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/resources/_itertools.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_adapters.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_meta.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_collections.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_text.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_functools.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/importlib/metadata/_itertools.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/xmlrpc/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/xmlrpc/client.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/collections/abc.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/collections/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/logging/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/encoders.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/parser.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/policy.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/_policybase.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/charset.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/_parseaddr.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/errors.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/iterators.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/contentmanager.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/base64mime.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/headerregistry.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/_header_value_parser.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/header.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/quoprimime.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/feedparser.pyc
[unknown]:    /exe.macosx-10.9-universal2-3.12/lib/email/architecture.rst
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/message.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/generator.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/utils.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/email/_encoded_words.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/re/_parser.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/re/_compiler.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/re/_casefix.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/re/__init__.pyc
[data]:       /exe.macosx-10.9-universal2-3.12/lib/re/_constants.pyc

Same Test Code with .so Filter

import subprocess
import os

def check_arch(filename):
    output = subprocess.check_output(["file", filename]).decode()

    if "x86_64" in output and "arm64" in output:
        arch = "universal2"
    elif "x86_64" in output:
        arch = "x86_64"
    elif "arm64" in output:
        arch = "arm64"
    elif "data" in output: # Typically files with the (.pyc) just output data by the file command
        arch = "data"
    else:
        arch = "unknown"

    arch = f"[{arch}]:" # Formatting

    print(f"{arch:<14}{filename}")

def main():
    path = os.path.abspath("multiprocessing/build/exe-macosx-10.9-universal2-3.12")

    for root, _, files in os.walk(path):
        for file in files:
            if not file.endswith(".so"): # Skip non-library files
                continue
            check_arch(os.path.join(root, file))

Output

Console Output
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_struct.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/array.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_queue.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_blake2.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha1.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_posixshmem.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_hashlib.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/grp.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_socket.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_decimal.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_csv.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_statistics.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha3.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/fcntl.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_ssl.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/zlib.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/unicodedata.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_bisect.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_cn.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_multibytecodec.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_posixsubprocess.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/select.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_lzma.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/pyexpat.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_pickle.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_jp.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_bz2.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_ctypes.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/mmap.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_datetime.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/math.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_sha2.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_iso2022.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_tw.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_opcode.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_kr.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_heapq.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_md5.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_multiprocessing.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_random.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_contextvars.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/_codecs_hk.cpython-312-darwin.so
[universal2]: /exe.macosx-10.9-universal2-3.12/lib/binascii.cpython-312-darwin.so

Executable File Architectures

[x86_64]:     /exe.macosx-10.9-universal2-3.12/sample2
[x86_64]:     /exe.macosx-10.9-universal2-3.12/sample3
[x86_64]:     /exe.macosx-10.9-universal2-3.12/sample4
[x86_64]:     /exe.macosx-10.9-universal2-3.12/sample1

Conclusion Based on the data provided, it appears that even when utilizing universal2 binary files obtained from installing the universal2 version of Python, cx_Freeze does not generate corresponding universal2 binary files. This suggests that there might be limitations or configurations within cx_Freeze's executable generation that prevents the creation of universal2 binaries alongside it.

IsaiahCoroama commented 3 weeks ago

The conclusion above along with the data for it, is more related to https://github.com/marcelotduarte/cx_Freeze/issues/2329, that it is to the current issue. (https://github.com/marcelotduarte/cx_Freeze/issues/2435)

marcelotduarte commented 3 weeks ago

Uninstall cx_Freeze and reinstall to check the wheel installed, or download https://files.pythonhosted.org/packages/e8/12/70285b18f9dea354e69ee1b472f46e79f9ae9aa6749d23a3109bb73aa9fc/cx_Freeze-7.1.0.post0-cp312-cp312-macosx_10_9_universal2.whl and install it, so you should get a universal2 executable. image

marcelotduarte commented 2 weeks ago

Based on information from you and others, I improved the hook for multiprocessing. You can test the patch in the latest development build: pip install --force --no-cache --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze The provisional documentation: https://cx-freeze--2443.org.readthedocs.build/en/2443/faq.html#multiprocessing-support

marcelotduarte commented 1 week ago

Release 7.1.1 is out! Documentation