search

borgbackup / borg

Deduplicating archiver with compression and authenticated encryption.
https://www.borgbackup.org/
Other
11.18k stars 742 forks source link

check nuitka (again) #3227

Open ThomasWaldmann opened 7 years ago

ThomasWaldmann commented 7 years ago

https://nuitka.net/doc/user-manual.html#

can it produce a single-file "exe" now?

advantages / disadvantages compared to pyinstaller?

Bounty: https://www.bountysource.com/issues/50879646-check-nuitka-again

ThomasWaldmann commented 7 years ago

note: when trying it, pip install nuitka (into the virtual env, do not use the debian/ubuntu package).

ThomasWaldmann commented 4 years ago

nuitka 0.6.7 (installed via pip):

(borg-env) user@development:~/w/borg/src$ python -m nuitka --standalone --include-package=borg --show-progress --full-compat borg/__main__.py
...
Nuitka:INFO:Total memory usage before generating C code: 258.21 MB (270757888 bytes):
Nuitka:WARNING:Unresolved '__import__' call at '/home/user/w/borg-env/lib/python3.5/site-packages/apipkg/__init__.py:72' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '/home/user/w/borg-env/lib/python3.5/site-packages/py/_vendored_packages/apipkg.py:69' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:INFO:Total memory usage before running scons: 359.33 MB (376786944 bytes):
Error, needs 'chrpath' on your system, due to 'RPATH' settings in used shared
libraries that need to be removed.

After a while, there is a executable called __main__.dist/__main__ and when running it, it outputs:

user@development:~/w/borg/src/__main__.dist$ ./__main__ 
Traceback (most recent call last):
  File "/home/user/w/borg/src/__main__.dist/__main__.py", line 14, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/home/user/w/borg/src/__main__.dist/borg/__init__.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/home/user/w/borg/src/__main__.dist/distutils/__init__.py", line 11, in <module>
AttributeError: module 'opcode' has no attribute '__file__'

@KayHayen any idea?

kayhayen commented 4 years ago

Note: Can can leave out the __main__.py as you can with Python, and it will produce a borg binary.

kayhayen commented 4 years ago

Regarding the __file__ attribute, there have been improvements in this area, I am almost sure, that opcode will have it on the develop branch, this has been added a couple of releases ago. I was unware that it affects distutils.

kayhayen commented 4 years ago

Regarding your question, single-file is for acceleration only.

kayhayen commented 4 years ago

And don't use --full-compat as it makes worse binaries (e.g. compatibly wrong line numbers), uses more compile time memory, and all the while solves no real problems. I am making the wording change from need not be used to should not be used.

The reason we have this, is so we can compare CPython output with Nuitka program output and expect identical output in all cases.

ThomasWaldmann commented 4 years ago

@kayhayen thanks for the hints! I'll try again with a future release.

ThomasWaldmann commented 4 years ago

A new nuitka release 0.6.8 is out since a while.

ThomasWaldmann commented 2 years ago

Just gave it a quick try, using the current code from 1.2-maint branch:

$ python -m nuitka --standalone --include-package=borg --show-progress --full-compat src/borg/__main__.py
... (translates and compiles lots of stuff) ...
Nuitka:INFO: Created binary that runs on macOS 12.0 (arm64) or higher.
Nuitka:INFO: Successfully created '__main__.dist/__main__'.

(borg-env) tw@mba2020 borg % ls -l __main__.dist/   
total 91664
-rw-r--r--  1 tw  staff   3576976 Apr 27 16:51 Python
-rwxr-xr-x  1 tw  staff  32530336 Apr 27 16:51 __main__
-rw-r--r--  1 tw  staff    101648 Apr 27 16:51 _asyncio.so
-rw-r--r--  1 tw  staff     70272 Apr 27 16:51 _bisect.so
-rw-r--r--  1 tw  staff     89440 Apr 27 16:51 _blake2.so
-rw-r--r--  1 tw  staff     72656 Apr 27 16:51 _bz2.so
-rw-r--r--  1 tw  staff    185920 Apr 27 16:51 _codecs_cn.so
-rw-r--r--  1 tw  staff    186128 Apr 27 16:51 _codecs_hk.so
-rw-r--r--  1 tw  staff     72592 Apr 27 16:51 _codecs_iso2022.so
-rw-r--r--  1 tw  staff    302144 Apr 27 16:51 _codecs_jp.so
-rw-r--r--  1 tw  staff    169536 Apr 27 16:51 _codecs_kr.so
-rw-r--r--  1 tw  staff    169024 Apr 27 16:51 _codecs_tw.so
-rw-r--r--  1 tw  staff     68752 Apr 27 16:51 _contextvars.so
-rw-r--r--  1 tw  staff     68832 Apr 27 16:51 _crypt.so
-rw-r--r--  1 tw  staff     90656 Apr 27 16:51 _csv.so
-rw-r--r--  1 tw  staff    159104 Apr 27 16:51 _ctypes.so
-rw-r--r--  1 tw  staff    133472 Apr 27 16:51 _datetime.so
-rw-r--r--  1 tw  staff     71840 Apr 27 16:51 _dbm.so
-rw-r--r--  1 tw  staff    194752 Apr 27 16:51 _decimal.so
-rw-r--r--  1 tw  staff    117568 Apr 27 16:51 _elementtree.so
-rw-r--r--  1 tw  staff     97328 Apr 27 16:51 _hashlib.so
-rw-r--r--  1 tw  staff     69888 Apr 27 16:51 _heapq.so
-rw-r--r--  1 tw  staff     91360 Apr 27 16:51 _json.so
-rw-r--r--  1 tw  staff     72064 Apr 27 16:51 _lsprof.so
-rw-r--r--  1 tw  staff     92480 Apr 27 16:51 _lzma.so
-rw-r--r--  1 tw  staff     95952 Apr 27 16:51 _multibytecodec.so
-rw-r--r--  1 tw  staff     69376 Apr 27 16:51 _opcode.so
-rw-r--r--  1 tw  staff    170144 Apr 27 16:51 _pickle.so
-rw-r--r--  1 tw  staff     72656 Apr 27 16:51 _posixsubprocess.so
-rw-r--r--  1 tw  staff     71776 Apr 27 16:51 _queue.so
-rw-r--r--  1 tw  staff     72160 Apr 27 16:51 _random.so
-rw-r--r--  1 tw  staff     70992 Apr 27 16:51 _scproxy.so
-rw-r--r--  1 tw  staff     70656 Apr 27 16:51 _sha512.so
-rw-r--r--  1 tw  staff    131696 Apr 27 16:51 _socket.so
-rw-r--r--  1 tw  staff    199936 Apr 27 16:51 _ssl.so
-rw-r--r--  1 tw  staff     68880 Apr 27 16:51 _statistics.so
-rw-r--r--  1 tw  staff     94704 Apr 27 16:51 _struct.so
-rw-r--r--  1 tw  staff     68624 Apr 27 16:51 _uuid.so
-rw-r--r--  1 tw  staff     90240 Apr 27 16:51 _zoneinfo.so
-rw-r--r--  1 tw  staff     96608 Apr 27 16:51 array.so
-rw-r--r--  1 tw  staff     89520 Apr 27 16:51 audioop.so
-rw-r--r--  1 tw  staff     89520 Apr 27 16:51 binascii.so
drwxr-xr-x  9 tw  staff       288 Apr 27 16:51 borg
-rw-r--r--  1 tw  staff     70672 Apr 27 16:51 fcntl.so
-rw-r--r--  1 tw  staff     71328 Apr 27 16:51 grp.so
-rw-r--r--  1 tw  staff   2192928 Apr 27 16:51 libcrypto.1.1.dylib
-rw-r--r--  1 tw  staff    110112 Apr 27 16:51 libgdbm.6.dylib
-rw-r--r--  1 tw  staff     53584 Apr 27 16:51 libgdbm_compat.4.dylib
-rw-r--r--  1 tw  staff    158832 Apr 27 16:51 liblz4.1.dylib
-rw-r--r--  1 tw  staff    182464 Apr 27 16:51 liblzma.5.dylib
-rw-r--r--  1 tw  staff    205584 Apr 27 16:51 libmpdec.3.dylib
-rw-r--r--  1 tw  staff    293744 Apr 27 16:51 libreadline.8.dylib
-rw-r--r--  1 tw  staff    506208 Apr 27 16:51 libssl.1.1.dylib
-rw-r--r--  1 tw  staff     55264 Apr 27 16:51 libxxhash.0.dylib
-rw-r--r--  1 tw  staff    622000 Apr 27 16:51 libzstd.1.dylib
drwxr-xr-x  3 tw  staff        96 Apr 27 16:51 markupsafe
-rw-r--r--  1 tw  staff    111184 Apr 27 16:51 math.so
-rw-r--r--  1 tw  staff     73232 Apr 27 16:51 mmap.so
drwxr-xr-x  3 tw  staff        96 Apr 27 16:51 msgpack
drwxr-xr-x  4 tw  staff       128 Apr 27 16:51 platform
-rw-r--r--  1 tw  staff    114224 Apr 27 16:51 pyexpat.so
-rw-r--r--  1 tw  staff     94512 Apr 27 16:51 readline.so
-rw-r--r--  1 tw  staff     91808 Apr 27 16:51 select.so
-rw-r--r--  1 tw  staff     70912 Apr 27 16:51 termios.so
-rw-r--r--  1 tw  staff   1163840 Apr 27 16:51 unicodedata.so
-rw-r--r--  1 tw  staff     91632 Apr 27 16:51 zlib.so

(borg-env) tw@mba2020 borg % file __main__.dist/__main__
__main__.dist/__main__: Mach-O 64-bit executable arm64

(borg-env) tw@mba2020 borg % __main__.dist/__main__ 
Traceback (most recent call last):
  File "/Users/tw/w/borg/__main__.dist/borg/archiver.py", line 38, in <module>
  File "/Users/tw/w/borg/__main__.dist/borg/helpers/__init__.py", line 14, in <module>
  File "/Users/tw/w/borg/__main__.dist/borg/helpers/misc.py", line 5, in <module>
  File "/Users/tw/w/borg/__main__.dist/platform/__init__.py", line 7, in <module>
ImportError: attempted relative import beyond top-level package
kayhayen commented 2 years ago

Do not use --full-compat, it's not for end users really. Also you might have a case where --python-flag=-m is intended for.

ThomasWaldmann commented 2 years ago

Hi @kayhayen !

Oh, sorry, you already said that, but I did not re-read all the comments above.

next try

(borg-env) tw@mba2020 borg % python -m nuitka --standalone --include-package=borg --show-progress src/borg
...                                                                            
Nuitka:INFO: Completed Python level compilation and optimization.                                                                                                                                         
Nuitka:INFO: Generating source code for C backend compiler.
Nuitka:INFO: Total memory usage before generating C code: 308.70 MB (323698688 bytes):
Nuitka:INFO: Total memory usage before running scons: 402.38 MB (421920768 bytes):                                                                                                                        
Nuitka:INFO: Running data composer tool for optimal constant value handling.
Nuitka:INFO: Running C compilation via Scons.
Nuitka-Scons:INFO: Backend C compiler: clang (clang).
TypeError: Directory /Users/tw/w/borg/borg.dist/borg found where file expected.:
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/Backend.scons", line 936:
    target = env.Program(result_exe, source_files)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Environment.py", line 255:
    return MethodWrapper.__call__(self, target, source, *args, **kw)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Environment.py", line 219:
    return self.method(*nargs, **kwargs)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Builder.py", line 651:
    return self._execute(env, target, source, OverrideWarner(kw), ekw)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Builder.py", line 563:
    tlist, slist = self._create_nodes(env, target, source)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Builder.py", line 508:
    tlist = env.arg2nodes(target, target_factory, target=target, source=source)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Environment.py", line 478:
    v = node_factory(self.subst(v, **kw))
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Node/FS.py", line 1382:
    return self._lookup(name, directory, File, create)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Node/FS.py", line 1361:
    return root._lookup_abs(p, fsclass, create)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Node/FS.py", line 2402:
    result.diskcheck_match()
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Node/FS.py", line 2634:
    diskcheck_match(self, self.isdir,
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Node/FS.py", line 393:
    return self.func(*args, **kw)
  File "/Users/tw/w/borg-env/lib/python3.9/site-packages/nuitka/build/inline_copy/lib/scons-3.1.2/SCons/Node/FS.py", line 414:
    raise TypeError(errorfmt % node.get_abspath())
(borg-env) tw@mba2020 borg % ls -l borg.dist     
total 12544
-rw-r--r--  1 tw  staff   101616 Apr 27 18:27 _asyncio.so
...
drwxr-xr-x  9 tw  staff      288 Apr 27 18:27 borg
-rw-r--r--  1 tw  staff    70656 Apr 27 18:27 fcntl.so
...

(borg-env) tw@mba2020 borg % ls -l borg.dist/borg
total 1944
drwxr-xr-x  3 tw  staff      96 Apr 27 18:27 algorithms
-rw-r--r--  1 tw  staff  184009 Apr 27 18:27 chunker.so
-rw-r--r--  1 tw  staff  264314 Apr 27 18:27 compress.so
drwxr-xr-x  3 tw  staff      96 Apr 27 18:27 crypto
-rw-r--r--  1 tw  staff  153371 Apr 27 18:27 hashindex.so
-rw-r--r--  1 tw  staff  388470 Apr 27 18:27 item.so
drwxr-xr-x  4 tw  staff     128 Apr 27 18:27 platform

OK, so it looks like it produced a filename collision within paths / filenames it created itself.

next try

(borg-env) tw@mba2020 borg % python -m nuitka --standalone --include-package=borg \
                                              --show-progress --onefile -o borg.exe src/borg

(borg-env) tw@mba2020 borg % ./borg.exe -V
borg.exe 1.2.1.dev118+gbcac974d.d20220427

(borg-env) tw@mba2020 borg %  file borg.exe
borg.exe: Mach-O 64-bit executable arm64

Yay!

Did not quite understand why / when I would need --python-flag=-m

ThomasWaldmann commented 2 years ago 
(borg-env) tw@mba2020 borg % ls -lh borg.exe
-rwxr-xr-x  1 tw  staff    46M Apr 27 18:39 borg.exe

(borg-env) tw@mba2020 borg % ./borg.exe init -e none repo-nuitka
(borg-env) tw@mba2020 borg % ./borg.exe create repo-nuitka::arch src
(borg-env) tw@mba2020 borg % ./borg.exe info repo-nuitka      
Repository ID: b4cab9c6bca332a5d7be6d60e53c3d2caa905f6f275adcdde1a2576e45d01c54
Location: /Users/tw/w/borg/repo-nuitka
Encrypted: No
Cache: /Users/tw/.cache/borg/b4cab9c6bca332a5d7be6d60e53c3d2caa905f6f275adcdde1a2576e45d01c54
Security dir: /Users/tw/.config/borg/security/b4cab9c6bca332a5d7be6d60e53c3d2caa905f6f275adcdde1a2576e45d01c54
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:               15.71 MB              5.29 MB              5.32 MB

                       Unique chunks         Total chunks
Chunk index:                     334                  338
(borg-env) tw@mba2020 borg % ./borg.exe check --verify-data repo-nuitka
(borg-env) tw@mba2020 borg % ./borg.exe check --verify-data -v repo-nuitka
Starting repository check
finished segment check at segment 5
Starting repository index check
Index object count match.
Finished full repository check, no problems found.
Starting archive consistency check...
Starting cryptographic data integrity verification...
Finished cryptographic data integrity verification, verified 335 chunks with 0 integrity errors.
Analyzing archive arch (1/1)
Archive consistency check complete, no problems found.

Looks good so far. Will run all the tests with that binary later.

kayhayen commented 2 years ago

It's s rather strange difference, you can execute a program like python -m package_name and like python package_name, in one case, the __main__ will have a package and relative imports will work, in the other it will not have that, and relative imports will give the error you had there. I think there is indeed an issue with a borg binary and directory, but it is not clear to me what you had there and how you solved it. It sounds familiar, it might be one those for which I never had a reproducer. I am OK if you raise one and point to the repo in Nuitka issue tracker.