No available dynamic library for windows.x86_64 with features ['25'] or ['11'] / OSError: exception: access violation reading 0x0000000000000000 / SystemError: unknown opcode

[sla-te]

On on Windows 10 2004

With super plus vm protection; python 3.9 on obfuscation: ERROR No available dynamic library for windows.x86_64 with features ['25']

With advanced mode plus vm protection; python 3.9 on attempting to start app:

XXX lineno: 6, opcode: 228
XXX lineno: 1378, opcode: 88
Traceback (most recent call last):
  File "<frozen app>", line 6, in <module>
SystemError: unknown opcode

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<C:\Users\user\PycharmProjects\myapp\dist\app.py>", line 3, in <module>
  File "<frozen app>", line 1378, in <module>
SystemError: unknown opcode

With super mode; python 3.9 on obfuscation: ERROR No available dynamic library for windows.x86_64.11.py39 with features ['11']

With advanced mode; python 3.9 on attempting to start app:

Traceback (most recent call last):
  File "C:\Users\user\PycharmProjects\mrt\dist\app.py", line 2, in <module>
    pyarmor_runtime(suffix='_vax_000139')
  File "C:\Users\user\PycharmProjects\mrt\dist\pytransform_vax_000139\__init__.py", line 344, in pyarmor_runtime
    pyarmor_init(path, is_runtime=1, suffix=suffix)
  File "C:\Users\user\PycharmProjects\mrt\dist\pytransform_vax_000139\__init__.py", line 340, in pyarmor_init
    return init_pytransform()
  File "C:\Users\user\PycharmProjects\mrt\dist\pytransform_vax_000139\__init__.py", line 57, in wrap
    return func(*args, **kwargs)
  File "C:\Users\user\PycharmProjects\mrt\dist\pytransform_vax_000139\__init__.py", line 75, in init_pytransform
    ret = init_module(major, minor, pythonapi._handle)
OSError: exception: access violation reading 0x0000000000000000

[jondy]

There is still no super mode binary library for Python3.9, I'll build it later. For advanced mode, please upgrade pyarmor to latest version v6.4.4, it fixes python3.9 issues.

[sla-te]

This time I updated before testing 😊 So I have version 6.4.4

[jondy]

There are still some problems for Python3.9, I'm fixing it, it may be released next week.

[jondy]

Here is pre-release v6.4.5, download it then pip install pyarmor-6.4.5.zip

It supports super mode for Python3.9, but advanced mode both --advanced 1 and --advanced 3 will not be supported for Python3.9

[sla-te]

I used super plus vm mode and it seems to work with one of my apps but haven't tested everything yet but for the other app its preventing any feature from starting.

I create a new process in the app on feature start as in process = Process(target=self.my_func, args=(AClass, a_dictionary, a_string, b_string), name=self.name_of_myself, daemon=True); process.start(). It will go through that but the function that is being targeted with the process is never being called. It cant really be related to multiprocessing at this point because my other app does the same, it does not use daemon or name args though. Another difference from this app to the other is that this one makes excessive use of inheritance.

Tried a lot of things to reproduce this issue in a code snippet but failed I think for this one I have to send you the code again.

Only thing I was able to figure out that if I use the default multiprocessing implementation I will get an exception in my code RuntimeError: Queue objects should only be shared between processes through inheritance. This problem can be solved by using https://github.com/uqfoundation/multiprocess but after obfuscating it will then just not spawn the process at all.

[jondy]

If the scripts are obfuscated by restrict mode > 2, try to set restrict mode to 0 or 1.

And does it work without obfuscation in Python 3.9 and default multiprocessing?

And could you add some print statements around this line to check why set_spawning_popen failed? https://github.com/uqfoundation/multiprocess/blob/master/py3.9/multiprocess/popen_spawn_win32.py#L90

[sla-te]

And does it work without obfuscation in Python 3.9 and default multiprocessing?

No, this is one of the reasons I am using multiprocess, because with this one it can pickle anything in python. If I use default multiprocessing also not obfuscated, I get this error: RuntimeError: Queue objects should only be shared between processes through inheritance, but with non-default one from uqfoundation it works.

If the scripts are obfuscated by restrict mode > 2, try to set restrict mode to 0 or 1.

Currently its set to restrict mode 1, will try with 0.

And could you add some print statements around this line to check why set_spawning_popen failed?

        print("Attempting to spawn")
            # send information to child
            print(prep_data, to_child, process_obj)
            set_spawning_popen(self)
            try:
                try:
                    reduction.dump(prep_data, to_child)
                except Exception as e:
                    print(type(e), e)
                    raise e
                try:
                    reduction.dump(process_obj, to_child)
                except Exception as e_2:
                    print(type(e_2), e_2)
                    raise e_2
            finally:
                set_spawning_popen(None)
            print("End of spawn process")

gives:

Attempting to spawn
{'log_to_stderr': False, 'authkey': b'a\x15{\xd5\xcd!_\x0e\xd4F\x87\x8c\xc3]\xb7\xf9\xf3\xddK$\xb5g\x00?\xb5\xef\x9c\xe9o0\t\x89', 'name': 'My Process', 'sys_path': ['C:\\Users\\user\\PycharmProjects\\my_project\\dist\\python\\python39.zip', 'C:\\Users\\user\\Pych
armProjects\\my_project\\dist\\python', 'C:\\Users\\user\\PycharmProjects\\my_project\\dist', 'C:\\Users\\user\\PycharmProjects\\my_project\\dist\\python\\win32', 'C:\\Users\\user\\PycharmProjects\\my_project\\dist\\python\\win32\\lib', 'C:\\Users\\user\\PycharmProjects\\massd
m\\dist\\python\\Pythonwin'], 'sys_argv': ['app.py'], 'orig_dir': 'C:\\Users\\user\\PycharmProjects\\my_project\\dist', 'dir': 'C:\\Users\\user\\PycharmProjects\\my_project\\dist', 'start_method': 'spawn', 'init_main_from_path': 'C:\\Users\\user\\PycharmProjects\\my_project
\\dist\\app.py'} <_io.BufferedWriter name=3> <Process name='My Process' parent=4664 initial daemon>
End of spawn process

[jondy]

It seems python system libraries are obfuscated either. There are many directories in the path my_project/dist/python, right? If that, how about excluding this path --exclude python as obfuscating the scripts?

[sla-te]

my_project/dist/python is the directory of the embedded python for my app, it is not considered during obfuscation.

my_project is root, so on obfusaction it will put the obfuscated scripts to default output folder set by pyarmor my_project/dist/ after i put the python folder in the dist folder manually.

maybe i can show you via anydesk if you have 30 minutes time?

[jondy]

If they're not obfuscated, it should not be problem.

Since process.start() doesn't work, how about inserting some print statements in this function? https://github.com/uqfoundation/multiprocess/blob/master/py3.9/multiprocess/process.py#L110

[sla-te]

        '''
        Start child process
        '''
        print("in_start")
        self._check_closed()
        assert self._popen is None, 'cannot start a process twice'
        print("NOT duplicate process")
        assert self._parent_pid == os.getpid(), \
               'can only start a process object created by current process'
        print("NOT can only start a process object created by current process")
        assert not _current_process._config.get('daemon'), \
               'daemonic processes are not allowed to have children'
        print("NOT a daemon")
        _cleanup()
        print("cleand up")
        self._popen = self._Popen(self)
        print("assigned popen")
        self._sentinel = self._popen.sentinel
        print("assigned sentine,")
        # Avoid a refcycle if the target function holds an indirect
        # reference to the process object (see bpo-30775)
        del self._target, self._args, self._kwargs
        print("deleted target")
        _children.add(self)
        print("children added")

gives full print from top to bottom :\

[sla-te]

Ok I think I was able to reproduce it:

# -*- coding: utf-8 -*-
from multiprocess import Queue, Process
from time import sleep

class XBaseClass:
    def __init__(self):
        self.queue_z = Queue()

class XBaseGUI:
    def __init__(self):
        self.queue_1 = Queue()
        self.queue_2 = Queue()
        self.process = None
        self.base = None

    def set_base(self, x_base):
        self.base = x_base

class XFeature(XBaseGUI):
    def __init__(self):
        super().__init__()
        self.process = None

    def do_work(self, obj):
        print("work")
        sleep(1)
        print("work_2")

    def create_proc(self):
        self.process = Process(target=self.do_work, args=(self.queue_1,), name="a_proc")
        self.process.start()

def main():
    x_base_class = XBaseClass()
    feat = XFeature()
    feat.set_base(x_base_class)
    feat.create_proc()

if __name__ == "__main__":
    print("starting")
    main()

obfuscate this one and try to run it, it should not print work and work_2 if you run it obfuscated but it will print it if you run it not ofbuscated.

And there is more weirdness. If I run this script obfuscated with my system python or inside a venv it starts without error but if i use embedded python, it will drop this error:

Traceback (most recent call last):
  File "<C:\Users\chwba\PycharmProjects\massdm\dist\test_file_obf\obf_x.py>", line 2, in <module>
  File "<frozen obf_x>", line 87, in <module>
  File "<frozen obf_x>", line 82, in protect_pytransform
  File "<frozen obf_x>", line 75, in check_lib_pytransform
RuntimeError: unexpected C:\Users\chwba\PycharmProjects\massdm\dist\pytransform_vax_000139.pyd

You can reproduce it by downloading https://www.python.org/ftp/python/3.9.0/python-3.9.0-embed-amd64.zip and extracting in a folder "python". Then change the content of the file python39._pth to:

python39.zip
.
..

# Uncomment to run site.main() automatically
import site

And from your system python (must be exactly the same version, 3.9.0) you do python -m pip install -U multiprocess --target python

Then you can run it with embedded python inside my_test folder with folder structure: my_test -> python ->my_test_script.py -> dist\my_test_script.py (obfuscated) python\python.exe dist\mytest_script.py

[jondy]

RuntimeError: unexpected C:\Users\chwba\PycharmProjects\massdm\dist\pytransform_vax_000139.pyd

with option --no-cross-protection to disable this error.

[sla-te]

RuntimeError: unexpected C:\Users\chwba\PycharmProjects\massdm\dist\pytransform_vax_000139.pyd

with option --no-cross-protection to disable this error.

I use the gui, can you give me the full command for these settings:

[jondy]

Turn off the second line `Cross Protection' in the first figure

[sla-te]

Turn off the second line `Cross Protection' in the first figure

Yes, I know its just hard to configure it with the GUI for debugging, anyway I did that now, the error is gone, it starts the script (still weird though because why does cross protection trigger in an embedded python but not with system python or venv) and i get the same result, it prints "starting" but does not print work and work_2. - Using this very same embedded python running the script not obfuscated will print work and work_2.

[jondy]

@chwba I just have a glance multiprocess, please first open log in the test script:

if __name__ == "__main__":
    print("starting")
        from multiprocess.util import log_to_stderr, SUBDEBUG
        log_to_stderr(SUBDEBUG)
    main()

Then check the log, if possible, compare the log before obfuscated with the log after obfuscated.

If still could not fix it, please upload log here. Tomorrow I'll build the test environment, and debug it.

[sla-te]

obfuscated

starting
[DEBUG/MainProcess] created semlock with handle 696
[DEBUG/MainProcess] created semlock with handle 324
[DEBUG/MainProcess] created semlock with handle 660
[DEBUG/MainProcess] created semlock with handle 692
[DEBUG/MainProcess] created semlock with handle 672
[DEBUG/MainProcess] created semlock with handle 680
in_start
NOT duplicate process
NOT can only start a process object created by current process
NOT a daemon
cleaned up
C:\Users\user\PycharmProjects\massdm\dist\python\python.exe
no venv
Attempting to spawn
{'log_to_stderr': True, 'authkey': b'\xb3\x0f\xe5\xc4CjV\x13\xf9\xd5\r\x0b\x1dC\xef\x16\xc3\xfa\x81\xcf\x011\x8e(\xf0`)a\xa3\xf0R\x07', 'log_level': 5, 'name': 'a_proc', 'sys_path': ['C:\\Users\\user\\Pych
armProjects\\my_proj\\dist\\python\\python39.zip', 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist\\python', 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist', 'C:\\Users\\user\\PycharmProjects\\my_proj\\di
st\\python\\win32', 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist\\python\\win32\\lib', 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist\\python\\Pythonwin'], 'sys_argv': ['test_file_obf\\test_file.py'],
 'orig_dir': 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist', 'dir': 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist', 'start_method': 'spawn', 'init_main_from_path': 'C:\\Users\\user\\PycharmProjects\\
my_proj\\dist\\test_file_obf\\test_file.py'} <_io.BufferedWriter name=3> <Process name='a_proc' parent=5088 initial>
set_spawning_popen
<class 'multiprocess.popen_spawn_win32.Popen'>
set_spawning_popen
<class 'NoneType'>
End of spawn process
assigned popen
assigned sentinel
deleted target
children added
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[INFO/MainProcess] calling join() for process a_proc
[DEBUG/MainProcess] recreated blocker with handle 140
[DEBUG/MainProcess] recreated blocker with handle 136
[DEBUG/MainProcess] recreated blocker with handle 156

[DEBUG/MainProcess] recreated blocker with handle 132
[DEBUG/MainProcess] recreated blocker with handle 160
[INFO/MainProcess] child process calling self.run()
[DEBUG/MainProcess] running the remaining "atexit" finalizers
[Level 5/MainProcess] finalizer no longer registered

not obfuscated

starting
[DEBUG/MainProcess] created semlock with handle 328
[DEBUG/MainProcess] created semlock with handle 320
[DEBUG/MainProcess] created semlock with handle 596
[DEBUG/MainProcess] created semlock with handle 600
[DEBUG/MainProcess] created semlock with handle 616
[DEBUG/MainProcess] created semlock with handle 620
in_start
NOT duplicate process
NOT can only start a process object created by current process
NOT a daemon
cleaned up
C:\Users\user\PycharmProjects\my_proj\dist\python\python.exe
no venv
Attempting to spawn
{'log_to_stderr': True, 'authkey': b'\xe3z\xbbTG[\xee\x10\x7f\xca\xd1\x8f\xc0Sv\xca\x8eq\x93^\x8d\x15i\xa4\xbf\xe0\x85.e\xf7\xa2\xca', 'log_level': 5, 'name': 'a_proc', 'sys_path': ['C:\\Users\\user\\PycharmProjects\\my_proj\\dist\\py
thon\\python39.zip', 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist\\python', 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist', 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist\\python\\win32', 'C:\\Users\\user\\PycharmProjects\\m
assdm\\dist\\python\\win32\\lib', 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist\\python\\Pythonwin'], 'sys_argv': ['test_file.py'], 'orig_dir': 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist', 'dir': 'C:\\Users\\user\\PycharmPro
jects\\my_proj\\dist', 'start_method': 'spawn', 'init_main_from_path': 'C:\\Users\\user\\PycharmProjects\\my_proj\\dist\\test_file.py'} <_io.BufferedWriter name=3> <Process name='a_proc' parent=24140 initial>
set_spawning_popen
<class 'multiprocess.popen_spawn_win32.Popen'>
set_spawning_popen
<class 'NoneType'>
End of spawn process
assigned popen
assigned sentinel
deleted target
children added
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[INFO/MainProcess] calling join() for process a_proc
[DEBUG/MainProcess] recreated blocker with handle 140
[DEBUG/MainProcess] recreated blocker with handle 144
[DEBUG/MainProcess] recreated blocker with handle 152
[DEBUG/MainProcess] recreated blocker with handle 156
[DEBUG/MainProcess] recreated blocker with handle 172
[DEBUG/MainProcess] recreated blocker with handle 176
[INFO/MainProcess] child process calling self.run()
work
work_2
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
[INFO/MainProcess] process exiting with exitcode 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
[Level 5/MainProcess] finalizer no longer registered

[jondy]

I find the problem, some obfuscated information of the target function do_work are missed when multiprocess passes this code object across the process in Windows. One solution is to wrap this target function with another un-obfuscated function lambda_do_work, here is the changed version and it works in my test machine

class XFeature(XBaseGUI):

        ...

        # function name starts with `lambda_` will not be obfuscated
    def lambda_do_work(self, obj):
        self.do_work(obj)

    def do_work(self, obj):
        print("work")
        sleep(1)
        print("work_2")

    def create_proc(self):
                # change target to `lambda_do_work
        self.process = Process(target=self.lambda_do_work, args=(self.queue_1,), name="a_proc")
        self.process.start()

[sla-te]

More of an interesting fix 👍 It really works, good job.