SystemError: module filename missing in super mode

[kaoh]

When using the super mode I get:

Traceback (most recent call last): File "dist/obf/foo.py", line 2, in SystemError: module filename missing [2216871] Failed to execute script foo

The default mode works.

I use the command line:

pyarmor pack --clean --debug --with-license outer  --name foo -e " --add-data data/file1.txt:data --add-data data/file2.txt:data --hidden-import _cffi_backend" -x " --advanced 2 --exclude venv --exclude tests --exclude venv_tools --exclude foobar/foobar/tests --exclude deepspeech --exclude foobar/venv " foo.py

My Python version is running under Ubuntu 20.04:

Python 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0] on linux

[jondy]

Please first follow this guide to find the problem https://pyarmor.readthedocs.io/en/latest/questions.html#the-final-bundle-does-not-work

If it doesn't work, try to use repack script https://pyarmor.readthedocs.io/en/latest/advanced.html#repack-pyinstaller-bundle-with-obfuscated-scripts

[kaoh]

No success. With the latter I get:

713 INFO: Repacking EXE "foo_obf" --- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.8/logging/init.py", line 1081, in emit msg = self.format(record) File "/usr/lib/python3.8/logging/init.py", line 925, in format return fmt.format(record) File "/usr/lib/python3.8/logging/init.py", line 664, in format record.message = record.getMessage() File "/usr/lib/python3.8/logging/init.py", line 369, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/home/bar/foo/venv/lib/python3.8/site-packages/pyarmor/helper/repack.py", line 301, in excepthook logging.error(exc.args[0], *exc.args[1:]) Message: 'utf-8' Arguments: (b'\x01\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa5<=\x00\x00\x00\x00\x00\x0b\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 24, 25, 'invalid start byte')

[jondy]

Do you only obfuscate the scripts by super mode with all the same options in the -x to an empty path? And run the obfuscated scripts to check it work or not?

[kaoh]

Oh, sorry, I forgot to give the command lines I was using:

pyinstaller --clean --add-data deepspeech/kenlm.scorer:deepspeech --add-data deepspeech/output_graph.tflite:deepspeech --hidden-import _cffi_backend bar.py
pyarmor obfuscate --src="." --advanced 2 --exclude venv --exclude tests --exclude venv_tools --exclude speech_processing/speech_recognition/tests --exclude deepspeech --exclude speech_processing/venv -r -O obfdist --with-license outer bar.py
python -m pyarmor.helper.repack -p obfdist dist/bar/bar

But I found that already the obfuscated script before the packaging is not working:

python bar.py Traceback (most recent call last): File "bar.py", line 2, in from pytransform import pyarmor SystemError: module filename missing

[jondy]

Hi guy, do you make sure the command obfuscate run successfully?

[kaoh]

I think it succeeded:

pyarmor obfuscate --src="." --advanced 2 --exclude venv --exclude tests --exclude venv_tools --exclude speech_processing/speech_recognition/tests --exclude deepspeech --exclude speech_processing/venv -r -O obfdist --with-license outer foo.py INFO PyArmor Trial Version 6.7.2 INFO Python 3.8.5 INFO Target platforms: Native WARNING The trial version could not download the latest core libraries, tag r41.15a is always used WARNING The core library excepted version is r43.17, but got r41.15a from platform list file /home/foobar/.pyarmor/platforms/index.json INFO Update target platforms to: ['linux.x86_64.11.py38'] INFO Source path is "/home/foobar/Projekte/foo" INFO Entry scripts are ['./foo.py'] INFO Use cached capsule /home/foobar/.pyarmor/.pyarmor_capsule.zip INFO Search scripts mode: Recursive INFO Exclude path "venv" INFO Exclude path "tests" INFO Exclude path "venv_tools" INFO Exclude path "speech_processing/speech_recognition/tests" INFO Exclude path "deepspeech" INFO Exclude path "speech_processing/venv" INFO Auto exclude output path "obfdist" no previously-included directories found matching 'tests' no previously-included directories found matching 'obfdist' INFO Save obfuscated scripts to "obfdist" INFO Read product key from capsule INFO Obfuscate module mode is 2 INFO Obfuscate code mode is 1 INFO Wrap mode is 1 INFO Restrict mode is 1 INFO Advanced value is 2 INFO Super mode is True INFO Generating super runtime library to "obfdist" WARNING The trial version could not download the latest core libraries, tag r41.15a is always used WARNING The core library excepted version is r43.17, but got r41.15a from platform list file /home/foobar/.pyarmor/platforms/index.json INFO Extract pytransform.key INFO Use outer license file INFO Copying /home/foobar/.pyarmor/platforms/linux/x86_64/11/py38/pytransform.cpython-38-x86_64-linux-gnu.so INFO Patch extension obfdist/pytransform.so INFO Patch library file OK INFO Generate runtime files OK INFO Use protection template: venv/lib/python3.8/site-packages/pyarmor/protect_code2.pt INFO Start obfuscating the scripts... INFO foo.py -> obfdist/foo.py INFO Patch this entry script with protection code INFO speech_processing/speech_recognition/init.py -> obfdist/speech_processing/speech_recognition/init.py INFO speech_processing/speech_recognition/speech_recognition.py -> obfdist/speech_processing/speech_recognition/speech_recognition.py INFO Obfuscate 3 scripts OK.

But like reported before the obfuscated script is not working:

python bar.py Traceback (most recent call last): File "bar.py", line 2, in from pytransform import pyarmor SystemError: module filename missing

[jondy]

How about rename obfdist/pytransform.so to obfdist/pytransform.cpython-38-x86_64-linux-gnu.so?

[kaoh]

Unfortunately this did not help..

(venv) bar@bar-ThinkPad-X250:~/Projekte/foo/obfdist$ mv pytransform.so pytransform.cpython-38-x86_64-linux-gnu.so (venv) bar@bar-ThinkPad-X250:~/Projekte/foo/obfdist$ python3 foo.py Traceback (most recent call last): File "foo.py", line 2, in from pytransform import pyarmor SystemError: module filename missing (venv) bar@bar-ThinkPad-X250:~/Projekte/foo/obfdist$ ls -la total 1136 drwxrwxr-x 3 bar bar 4096 Jun 30 17:02 . drwxrwxr-x 14 bar bar 4096 Jun 30 15:21 .. -rw-rw-r-- 1 bar bar 12058 Jun 30 15:21 foo.py -rw-rw-r-- 1 bar bar 1136672 Jun 30 15:21 pytransform.cpython-38-x86_64-linux-gnu.so drwxrwxr-x 3 bar bar 4096 Jun 30 15:21 speech_processing

Are there any steps how I can debug it? I already tried this, because because this is a system exception, it did no reveal much.

When it is complaining about line 2 in the file foo.py:

!/usr/bin/env python3

from pytransform import pyarmor pyarmor(name, file, b

I tried to enter `from pytransform import pyarmor' directly in python, but got also:

(venv) bar@bar-ThinkPad-X250:~/Projekte/foo/obfdist$ python Python 3.8.5 (default, May 27 2021, 13:30:53) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

from pytransform import pyarmor Traceback (most recent call last): File "", line 1, in SystemError: module filename missing

[kaoh]

Can this be related? There are not a lot of hits when searching for "SystemError: module filename missing": https://stackoverflow.com/questions/11024089/python-c-extension-with-data-file

[jondy]

It seems the extension is problem. So what's output of this command?

python3.8-config --extension-suffix

If it's different from .cpython-38-x86_64-linux-gnu.so, rename obfdist/pytransform.so as it is.

And also make sure this python is built with --enable-shared

[kaoh]

python3.8-config --extension-suffix .cpython-38-x86_64-linux-gnu.so

I started python3 with -v:

# extension module 'apt_pkg' loaded from '/usr/lib/python3/dist-packages/apt_pkg.cpython-38-x86_64-linux-gnu.so'
# extension module 'apt_pkg' executed from '/usr/lib/python3/dist-packages/apt_pkg.cpython-38-x86_64-linux-gnu.so'
# extension module '_bz2' loaded from '/usr/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so'
# extension module '_bz2' executed from '/usr/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so'
# extension module '_lzma' loaded from '/usr/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so'
# extension module '_lzma' executed from '/usr/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so'
# extension module '_hashlib' loaded from '/usr/lib/python3.8/lib-dynload/_hashlib.cpython-38-x86_64-linux-gnu.so'
# extension module '_hashlib' executed from '/usr/lib/python3.8/lib-dynload/_hashlib.cpython-38-x86_64-linux-gnu.so'
# extension module '_ssl' loaded from '/usr/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so'
# extension module '_ssl' executed from '/usr/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so'
# extension module '_queue' loaded from '/usr/lib/python3.8/lib-dynload/_queue.cpython-38-x86_64-linux-gnu.so'
# extension module '_queue' executed from '/usr/lib/python3.8/lib-dynload/_queue.cpython-38-x86_64-linux-gnu.so'
# extension module '_brotli' loaded from '/usr/lib/python3/dist-packages/_brotli.cpython-38-x86_64-linux-gnu.so'
# extension module '_brotli' executed from '/usr/lib/python3/dist-packages/_brotli.cpython-38-x86_64-linux-gnu.so'
# extension module '_opcode' loaded from '/usr/lib/python3.8/lib-dynload/_opcode.cpython-38-x86_64-linux-gnu.so'
# extension module '_opcode' executed from '/usr/lib/python3.8/lib-dynload/_opcode.cpython-38-x86_64-linux-gnu.so'
# /usr/local/lib/python3.8/dist-packages/cryptography-3.3.2-py3.8-linux-x86_64.egg/cryptography/x509/__pycache__/extensions.cpython-38.pyc matches /usr/local/lib/python3.8/dist-packages/cryptography-3.3.2-py3.8-linux-x86_64.egg/cryptography/x509/extensions.py
# code object from '/usr/local/lib/python3.8/dist-packages/cryptography-3.3.2-py3.8-linux-x86_64.egg/cryptography/x509/__pycache__/extensions.cpython-38.pyc'
# extension module '_cffi_backend' loaded from '/usr/local/lib/python3.8/dist-packages/cffi-1.14.5-py3.8-linux-x86_64.egg/_cffi_backend.cpython-38-x86_64-linux-gnu.so'
# extension module '_cffi_backend' executed from '/usr/local/lib/python3.8/dist-packages/cffi-1.14.5-py3.8-linux-x86_64.egg/_cffi_backend.cpython-38-x86_64-linux-gnu.so'
# extension module 'bcrypt._bcrypt' loaded from '/usr/lib/python3/dist-packages/bcrypt/_bcrypt.abi3.so'
# extension module 'bcrypt._bcrypt' executed from '/usr/lib/python3/dist-packages/bcrypt/_bcrypt.abi3.so'
import 'cryptography.x509.extensions' # <_frozen_importlib_external.SourceFileLoader object at 0x7f75eb428190>
# extension module 'cryptography.hazmat.bindings._openssl' loaded from '/usr/local/lib/python3.8/dist-packages/cryptography-3.3.2-py3.8-linux-x86_64.egg/cryptography/hazmat/bindings/_openssl.abi3.so'
# extension module 'cryptography.hazmat.bindings._openssl' executed from '/usr/local/lib/python3.8/dist-packages/cryptography-3.3.2-py3.8-linux-x86_64.egg/cryptography/hazmat/bindings/_openssl.abi3.so'
# extension module '_decimal' loaded from '/usr/lib/python3.8/lib-dynload/_decimal.cpython-38-x86_64-linux-gnu.so'
# extension module '_decimal' executed from '/usr/lib/python3.8/lib-dynload/_decimal.cpython-38-x86_64-linux-gnu.so'
# extension module 'simplejson._speedups' loaded from '/usr/lib/python3/dist-packages/simplejson/_speedups.cpython-38-x86_64-linux-gnu.so'
# extension module 'simplejson._speedups' executed from '/usr/lib/python3/dist-packages/simplejson/_speedups.cpython-38-x86_64-linux-gnu.so'
# extension module '_json' loaded from '/usr/lib/python3.8/lib-dynload/_json.cpython-38-x86_64-linux-gnu.so'
# extension module '_json' executed from '/usr/lib/python3.8/lib-dynload/_json.cpython-38-x86_64-linux-gnu.so'
# cleanup[2] removing cryptography.x509.extensions
# destroy cryptography.x509.extensions

The extension is not loaded.

I tried to set export PYTHONPATH=.. This also has no effect.

[jondy]

It works in my linux machine with both suffix .so and .cpython-38-x86_64-linux-gnu.so.

It may need trace the package importlib in this machine to find the problem.

Just some ideas,

chmod 777 obfdist/pytransform.so

cp obfdist/pytransform.so /usr/lib/python3.8/lib-dynload/pytransform.cpython-38-x86_64-linux-gnu.so

[jondy]

Check the dependencies by ldd, compare the differences with system extension

ldd obfdist/pytransform.so
ldd /usr/lib/python3.8/lib-dynload/_decimal.cpython-38-x86_64-linux-gnu.so

[kaoh]

ldd pytransform.so 
    linux-vdso.so.1 (0x00007ffcc7b7d000)
    libpython3.8.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f2f6de71000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2f6dc7f000)
    libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f2f6dc51000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2f6dc35000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2f6dc12000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2f6dc0c000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f2f6dc05000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2f6dab6000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f2f6e71e000)
bar@bar-ThinkPad-X250:~/projects/foo/obfdist$ ldd /usr/lib/python3.8/lib-dynload/_decimal.cpython-38-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007ffca71fa000)
    libmpdec.so.2 => /usr/lib/x86_64-linux-gnu/libmpdec.so.2 (0x00007f773bc02000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f773bbdf000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f773b9ed000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f773b89e000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f773bca4000)
bar@bar-ThinkPad-X250:~/projects/foo/obfdist$

[kaoh]

chmod 777 obfdist/pytransform.so cp obfdist/pytransform.so /usr/lib/python3.8/lib-dynload/pytransform.cpython-38-x86_64-linux-gnu.so

did not help.

[kaoh]

I just wanted to search for the source code of pytransform.so, but could not find it. I would have assumed it is a CPython extension, isn't it?

[kaoh]

Nevermind, found it in https://github.com/dashingsoft/pyarmor-core

[kaoh]

Well, I thought I found it. Is the platform.so library also Open Source?

[jondy]

No, it's not open source.

Does it work to import pytransform only?

And if there are many python versions installed in this machine, make sure python3 is same as the python3.8

[kaoh]

Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytransform
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pytransform'
>>> 

I have only Python 3.8 in my virtual env.

We can continue to find the issue, but if pytransform.so is not open source, my project policy unfortunately does not allow to use closed source tools. All parts must be auditable.

[jondy]

Sorry, this part will not be opened for the security.

[jondy]

I think I may find the problem. Try to copy license.lic of the obfuscated scripts to the current path, or set environment variable PYARMOR_LICENSE to this license file, then run final bundle or the obfuscated scripts. For example

PYARMOR_LICENSE=/path/to/license.lic python obfdist/foo.py

[kaoh]

Unfortunately this also does not resolve it.

[kaoh]

Is there any chance that the pytransform library will be opened sourced? Security by obscurity is proven to be the wrong approach. There are white box cryptography schemes which are not perfect but are adding a certain level of security by a smart scheme of weaving in the key (which is my assumption why the core library is not open sourced).

[jondy]

White box cryptography scheme is highest security level, it's the final goal for pyarmor. In the future if pyarmor could achieve this level, it may be open. But not now.