Closed mgeeky closed 3 months ago
Can you compile with --python-flag=-v and show the imports right before _ctypes
So it looks like colorama
might be my problematic import. One I cannot really do away, as my programs heavily use colored console outputs:
PS C:\Program\Path> .\program.exe -h
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
# installing zipimport hook
import 'time' # <class '_frozen_importlib.BuiltinImporter'>
import 'zipimport' # <class '_frozen_importlib.FrozenImporter'>
# installed zipimport hook
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <class '_frozen_importlib.FrozenImporter'>
import 'encodings.aliases' # <class '_frozen_importlib.FrozenImporter'>
import 'encodings' # <class '_frozen_importlib.FrozenImporter'>
import 'encodings.utf_8' # <class '_frozen_importlib.FrozenImporter'>
import 'encodings.cp1250' # <class '_frozen_importlib.FrozenImporter'>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <class '_frozen_importlib.FrozenImporter'>
import 'io' # <class '_frozen_importlib.FrozenImporter'>
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <class '_frozen_importlib.FrozenImporter'>
import '_collections_abc' # <class '_frozen_importlib.FrozenImporter'>
import 'genericpath' # <class '_frozen_importlib.FrozenImporter'>
import '_winapi' # <class '_frozen_importlib.BuiltinImporter'>
import 'ntpath' # <class '_frozen_importlib.FrozenImporter'>
import 'os' # <class '_frozen_importlib.FrozenImporter'>
import 'warnings' # <class '_frozen_importlib.FrozenImporter'>
import 'importlib' # <class '_frozen_importlib.FrozenImporter'>
Setup nuitka compiled module/bytecode/extension importer.
import inspect # considering responsibility (find_spec)
import inspect # claimed responsibility (bytecode)
import inspect # created module
import inspect # execute module
import ast # considering responsibility (find_spec)
import ast # claimed responsibility (bytecode)
import ast # created module
import ast # execute module
import '_ast' # <class '_frozen_importlib.BuiltinImporter'>
import contextlib # considering responsibility (find_spec)
import contextlib # claimed responsibility (bytecode)
import contextlib # created module
import contextlib # execute module
import collections # considering responsibility (find_spec)
import collections # claimed responsibility (bytecode)
import collections # created module
import collections # execute module
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
import keyword # considering responsibility (find_spec)
import keyword # claimed responsibility (bytecode)
import keyword # created module
import keyword # execute module
import 'keyword' # <class 'nuitka_module_loader'>
import operator # considering responsibility (find_spec)
import operator # claimed responsibility (bytecode)
import operator # created module
import operator # execute module
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <class 'nuitka_module_loader'>
import reprlib # considering responsibility (find_spec)
import reprlib # claimed responsibility (bytecode)
import reprlib # created module
import reprlib # execute module
import 'reprlib' # <class 'nuitka_module_loader'>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <class 'nuitka_module_loader'>
import functools # considering responsibility (find_spec)
import functools # claimed responsibility (bytecode)
import functools # created module
import functools # execute module
import types # considering responsibility (find_spec)
import types # claimed responsibility (bytecode)
import types # created module
import types # execute module
import 'types' # <class 'nuitka_module_loader'>
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
import 'functools' # <class 'nuitka_module_loader'>
import 'contextlib' # <class 'nuitka_module_loader'>
import enum # considering responsibility (find_spec)
import enum # claimed responsibility (bytecode)
import enum # created module
import enum # execute module
import 'enum' # <class 'nuitka_module_loader'>
import 'ast' # <class 'nuitka_module_loader'>
import dis # considering responsibility (find_spec)
import dis # claimed responsibility (bytecode)
import dis # created module
import dis # execute module
import opcode # considering responsibility (find_spec)
import opcode # claimed responsibility (bytecode)
import opcode # created module
import opcode # execute module
import '_opcode' # <class '_frozen_importlib.BuiltinImporter'>
import 'opcode' # <class 'nuitka_module_loader'>
import 'dis' # <class 'nuitka_module_loader'>
import collections.abc # considering responsibility (find_spec)
import collections.abc # claimed responsibility (bytecode)
import collections.abc # created module
import collections.abc # execute module
import 'collections.abc' # <class 'nuitka_module_loader'>
import importlib.machinery # considering responsibility (find_spec)
import importlib.machinery # claimed responsibility (bytecode)
import importlib.machinery # created module
import importlib.machinery # execute module
import 'importlib.machinery' # <class 'nuitka_module_loader'>
import linecache # considering responsibility (find_spec)
import linecache # claimed responsibility (bytecode)
import linecache # created module
import linecache # execute module
import tokenize # considering responsibility (find_spec)
import tokenize # claimed responsibility (bytecode)
import tokenize # created module
import tokenize # execute module
import re # considering responsibility (find_spec)
import re # claimed responsibility (bytecode)
import re # created module
import re # execute module
import re._compiler # considering responsibility (find_spec)
import re._compiler # claimed responsibility (bytecode)
import re._compiler # created module
import re._compiler # execute module
import '_sre' # <class '_frozen_importlib.BuiltinImporter'>
import re._parser # considering responsibility (find_spec)
import re._parser # claimed responsibility (bytecode)
import re._parser # created module
import re._parser # execute module
import re._constants # considering responsibility (find_spec)
import re._constants # claimed responsibility (bytecode)
import re._constants # created module
import re._constants # execute module
import 're._constants' # <class 'nuitka_module_loader'>
import 're._parser' # <class 'nuitka_module_loader'>
import re._casefix # considering responsibility (find_spec)
import re._casefix # claimed responsibility (bytecode)
import re._casefix # created module
import re._casefix # execute module
import 're._casefix' # <class 'nuitka_module_loader'>
import 're._compiler' # <class 'nuitka_module_loader'>
import copyreg # considering responsibility (find_spec)
import copyreg # claimed responsibility (bytecode)
import copyreg # created module
import copyreg # execute module
import 'copyreg' # <class 'nuitka_module_loader'>
import 're' # <class 'nuitka_module_loader'>
import token # considering responsibility (find_spec)
import token # claimed responsibility (bytecode)
import token # created module
import token # execute module
import 'token' # <class 'nuitka_module_loader'>
import 'tokenize' # <class 'nuitka_module_loader'>
import 'linecache' # <class 'nuitka_module_loader'>
import 'inspect' # <class 'nuitka_module_loader'>
# destroy __main__
import argparse # considering responsibility (find_spec)
import argparse # claimed responsibility (bytecode)
import argparse # created module
import argparse # execute module
import gettext # considering responsibility (find_spec)
import gettext # claimed responsibility (bytecode)
import gettext # created module
import gettext # execute module
import 'gettext' # <class 'nuitka_module_loader'>
import 'argparse' # <class 'nuitka_module_loader'>
import textwrap # considering responsibility (find_spec)
import textwrap # claimed responsibility (bytecode)
import textwrap # created module
import textwrap # execute module
import 'textwrap' # <class 'nuitka_module_loader'>
import lib # considering responsibility (find_spec)
import lib # claimed responsibility (compiled)
import lib # created module
import lib # execute module
Loaded lib
import 'lib' # <class 'nuitka_module_loader'>
import lib.logger # considering responsibility (find_spec)
import lib.logger # claimed responsibility (compiled)
import lib.logger # created module
import lib.logger # execute module
import colorama # considering responsibility (find_spec)
import colorama # claimed responsibility (compiled)
import colorama # created module
import colorama # execute module
import colorama.initialise # considering responsibility (find_spec)
import colorama.initialise # claimed responsibility (compiled)
import colorama.initialise # created module
import colorama.initialise # execute module
import 'atexit' # <class '_frozen_importlib.BuiltinImporter'>
import colorama.ansitowin32 # considering responsibility (find_spec)
import colorama.ansitowin32 # claimed responsibility (compiled)
import colorama.ansitowin32 # created module
import colorama.ansitowin32 # execute module
import colorama.ansi # considering responsibility (find_spec)
import colorama.ansi # claimed responsibility (compiled)
import colorama.ansi # created module
import colorama.ansi # execute module
Loaded colorama.ansi
import 'colorama.ansi' # <class 'nuitka_module_loader'>
import colorama.winterm # considering responsibility (find_spec)
import colorama.winterm # claimed responsibility (compiled)
import colorama.winterm # created module
import colorama.winterm # execute module
import 'msvcrt' # <class '_frozen_importlib.BuiltinImporter'>
import colorama.win32 # considering responsibility (find_spec)
import colorama.win32 # claimed responsibility (compiled)
import colorama.win32 # created module
import colorama.win32 # execute module
import ctypes # considering responsibility (find_spec)
import ctypes # claimed responsibility (bytecode)
import ctypes # created module
import ctypes # execute module
import _ctypes # considering responsibility (find_spec)
import _ctypes # claimed responsibility (extension)
import _ctypes # created module
import _ctypes # execute module
import _ctypes # LoadLibraryExW("C:\Program\Path\_ctypes.pyd");
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "C:\Program\Path\ctypes\__init__.py", line 8, in <module>
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
ImportError: LoadLibraryExW 'C:\Program\Path\_ctypes.pyd' failed: The specified module could not be found.
# destroy _ctypes
# destroy ctypes
IMPORT_HARD_CTYPES : "failed hard import of 'ctypes'"
Assertion failed: false, file C:\Users\Mariusz\AppData\Roaming\Python\Python311\site-packages\nuitka\build\static_src\HelpersImportHard.c, line 120
Nobody is asking you to change anything, although I can see how you want a workaround, I usually want to fix the underlying issue. I wonder, if this is because of a dirt effect, with Nuitka's inline copy of colorama, because I don't see how. I do of course not reproduce this myself, the import of ctypes is pretty clearly visible. Can we first eliminate the --onefile usage and go to mere standalone and see if it happens for you then? I have a suspect, that it would have to be on that level somehow, and then of course the question becomes what might be missing for it to load. Also if you compile only "import colorama.win32" and run that, does it happen too?
Hi @kayhayen - thank you so much for a swift response!
So after merely changing from --onefile
into --standalone
program worked as expected! Looks like indeed there's problem with --onefile
merge of implicit dependencies.
I did two additional tests:
colorama
only hello world as --onefile
executable, worked without issues.Here's second program's imports list I've used (again, this has built without any issues):
import colorama
import atexit
import re
import os
import csv
import glob
import requests
import ssl
import json
import hashlib
import random
import shutil
import tempfile
import string
import pefile
import socket
import datetime
import binascii
import threading
import time
import urllib3
import yaml
import win32com
import win32com.client
import win32api
import win32con
import pythoncom
from tqdm import tqdm
from pathlib import Path
from OpenSSL import crypto
from OpenSSL.crypto import _lib, _ffi, X509
from urllib.parse import urlparse
from dateutil.parser import *
from dateutil import tz
from asn1crypto import pkcs12
from collections.abc import Iterable
from PyPDF2 import PdfReader, PdfWriter
from PyPDF2.generic import DecodedStreamObject, NameObject, DictionaryObject, createStringObject, ArrayObject
import xml.etree.ElementTree as ET
import urllib.request
import shutil
from pathlib import Path
from urllib.parse import urlparse,urlunparse,urljoin
from lxml import etree
from tqdm import tqdm
from urllib.parse import urlparse, parse_qs
from time import strftime, localtime
from datetime import timedelta
from typing import *
So at this point, I'm really clueless how to further isolate my problem or how to come up with a minimal test case script.
It definitely looks like PyImport_ImportModule("ctypes")
fails to execute for --onefile
whereas it doesn't for --standalone
. A close look into files and directories comprising standlone build indicates that there's _ctypes.pyd
file present locally. Maybe during --onefile
build that module doesn't get copied into local build directory, which could be causing import to fail?
Any fresh ideas for a workaround or how to proceed from here?
Best, Mariusz
Can you remove your onefile tempdir spec and try that way. It looks good to me, but it might somehow not be used properly, if you can, maybe make the program not terminate, and check unpacked files in the temp directory.
I see the option that shell quoting is your enemy there, maybe. Nuitka reports your command line options back to you, and a compilation report would have them too. If the \
was misinterpreted by a shell, say git bash, that could cause funny things. Your value doesn't look too different from the default, "{TEMP}" + os.path.sep + "onefile_{PID}_{TIME}"
but the {TEMP}
may not give a slash, and then you would be writing to a place where you may not even be allowed to write, or put files, not sure now. It may also be that the slash I push there is not even needed. With --experimental=debug-onefile-handling
it would output the payload path that results.
I think it would have to be that, since onefile is used a lot and hasn't changed in a long time for those things.
Hi there!
Coming back with an update, looks like it's all sorted now! After going through all of the imported modules dropped to the standalone build.dist directory I found that for some reason numpy
+ matplotlib
+ cv2
were implicitely imported, even though my program shouldn't be using those.
So in a first step I've explicitely denied importing them. Then I've also removed --onedir-tempdir-spec
as suggested and added ctypes
import explicitely, just in case.
As a result, these flags were added on top of previously used ones:
--include-module=ctypes --noinclude-custom-mode=numpy:nofollow --noinclude-custom-mode=numpy.libs:nofollow --noinclude-custom-mode=matplotlib:nofollow --noinclude-custom-mode=cv2:nofollow
Seems they did the trick as I'm now up and running again!
Thank you so much @kayhayen for quickly finding a workaround.
This probably leaves us with some outstanding issues in how the --onefile-tempdir-spec
parameter and its inner slashes/backslashes might be handled. Possibly we could add a statement to the parameter description that user is expected to use slashes instead of backslashes.
Best regards, Mariusz.
I'm sorry to report that the problem still remains, don't know how in the world it compiled these seven hours ago but I cannot seem to compile it back on again so that it would work as previously.
I mean, I'm compiling without --onefile-tempdir-spec
but the problem seems to remain. --onefile
compiled program keeps crashing on being unable to import ctypes
.
Now I'm really clueless how to troubleshoot this :(
Your report appears to make no sense to me. What could be going on is that some module pulls in a conflicting runtime DLL. Since no reproducer has been provided, there is not much I can do without that.
Addressing my doubts about onefile spec, indeed it can be used in the wrong way.
Nuitka: Launching 'HelloWorldTest.bin'.
Error, failed to open '/tmp_/HelloWorldTest.bin' for writing.
These system directories need to require being followed by a slash. If your quoting is off, the slash may create a newline if e.g. used like \n
and interpreted by the shell, which is what I was suspecting there. Unfortunately you never made it clear if you use the correct command line or not as echoed back by Nuitka. Checking for a slash will make it harder to have escaping issues directly following that slash. Your \t
might be meaning you create an illegal path. I recently added a function isLegalPath
and it seems Windows rejects those kinds of characters.
FATAL: Cannot use general system folder {TEMP}, without a path separator '--onefile-tempdir-spec={TEMP}_', just appending to these is not allowed, needs to be below them.
So this is a new checked I added for 2.4, which makes it more like to catch it.
Seems I am wrong about \t
being an illegal path on Windows, works just fine in my test just now. Probably not a good idea, but I imagine people will like using non-printable characters to hide it or whatever, so I will not interfere with it and just make it a warning.
Ah, now I fooled myself, was using absolute path \t\t
and that works of course.
Ok, but we now have errors like those:
FATAL: Cannot use illegal paths '--onefile-tempdir-spec={TEMP}\something.', due to contains illegal suffix '.'.
FATAL: Cannot use illegal paths '--onefile-tempdir-spec={TEMP}\ \something.', due to contains illegal suffix ' ' in path part ' '.
But what ought to happen is be unable to write the main executable already, so it probably also is never your issue, it might be AV preventing the load of files it still checks or something like that, but that's only a guess.
Alright, so here's what I found:
--output-dir=C:\build
path.C:\Build
C:\Build\program.build
C:\Build\program.dist
C:\Build\program.onefile-build
C:\Build\program.exe
C:\Build\program.pdb
C:\Build
directory and then run program.exe
- it runs just fineC:\Build
- it breaks due to being unable to find ctypes
Any ideas how to proceed next?
It makes no sense to me, except if you have a spurious corruption issue. I recently fixed one for lazy module values, maybe retry with current factory branch.
Which branch you refer to? Also, are we speaking Nuitka commercial (I'm a subscriber) or the public repo one?
For commercial, the staging branch has the fix, and public the factory branch has it.
I think this might be my culprit:
Unexpected output from this command:
<clcache> /Fostatic_src\OnefileBootstrap.obj /c "static_src\OnefileBootstrap.c" /nologo /GL /std:c11 /wd5105 -O2 /Z7 /FS /EHsc /J /Gd /bigobj /W4 /wd4505 /wd4127 /wd4100 /wd4702 /wd4189 /wd4211 /WX /wd4512 /wd4510 /wd4610 /wd4996 /MT /D_NUITKA_NO_DEPLOYMENT_SELF_EXECUTION /D_NUITKA_ONEFILE_MODE /D_NUITKA_EXPERIMENTAL_INIT_PROGRAM=1 /D_NUITKA_EXPERIMENTAL_WRITEABLE_CONSTANTS=1 /D_NUITKA_PLUGIN_MULTIPROCESSING_ENABLED=1 /IC:\PYTHON~2\Lib\SITE-P~1\nuitka\build\inline_copy\zlib /I. /IC:\PYTHON~2\Lib\SITE-P~1\nuitka\build\include /IC:\PYTHON~2\Lib\SITE-P~1\nuitka\build\static_src /IC:\PYTHON~2\Lib\SITE-P~1\nuitka\build\inline_copy\zstd
static_src\OnefileBootstrap.c(827): error C2220: the following warning is treated as an error
static_src\OnefileBootstrap.c(827): warning C4267: '+=': conversion from 'size_t' to 'int', possible loss of data
scons: *** [static_src\OnefileBootstrap.obj] Error 2
FATAL: Error, onefile bootstrap binary build failed.
Here's the OnefileBootstrap.c(827)
for context:
#if _WIN32
static wchar_t *getCommandLineForChildProcess(void) {
int argc;
LPWSTR *argv = CommandLineToArgvW(GetCommandLineW(), &argc);
assert(argc > 0);
argv[0] = getBinaryPath();
// Free memory allocated for CommandLineToArgvW arguments.
LocalFree(argv);
int size = 0;
for (int i = 0; i < argc; i++) {
size += wcslen(argv[i]); // <=== 827 line
}
size += argc;
wchar_t *result = malloc((size + 1) * sizeof(wchar_t));
result[0] = 0;
for (int i = 0; i < argc; i++) {
appendWStringSafeW(result, argv[i], size);
if (i != argc - 1) {
appendCharSafeW(result, ' ', size);
}
}
return result;
}
#endif
Obviously to satsify MSVC compiler, size
variable should be of type size_t
instead of current int
.
Moreover, this compilation crash happens only when I plug in --debug
flag.
Debug mode turns on MSVC's /W4
compilation flag, which causes all warnings to fail as errors.
The flag specifications is included in nuitka/build/Onefile.scons
script.
When I turn off --debug
, program runs fine.
I wasn't aware of this one yet for some reason. You can make it ignore warnings with --experimental=allow-c-warnings which unfortunately is needed for Python generated code sometimes too.
However, this code may indeed to be to blame for your issues. I need to think of what could be wrong there and how your report ties into it. This seems really bad, but it totally seems reasonable that it could cause it.
The warning itself is not a bug, it's not relevant, but if that code malfunctions, it would explain things maybe.
This code messes with sys.argv
and tries to make it absolute, such that if invoked non-absolute, os.path.dirname(sys.argv)
will be a reliable path to telling where the program is launched from. This wasn't yet in the code that you observed the issue first with.
I updated staging, so --debug won't give you the issue with onefile anymore.
Thank you @kayhayen for your time working this out with me! Happy I could help. Since I'm not having this particular issue anymore and I know you've got a lot on your plate already, I guess we can close this report.
Cheers, Mariusz.
Hi,
I'm having issues compiling my programs that used to compile just fine a few nuitka versions ago (can't really tell what was the last working Nuitka version for me).
Whenever I run the executable, there's simply no output whatsever, even though I'm building console programs. My programs don't import ctypes directly, but I have some dependencies that might import it indirectly.
I took the liberty to compile program with
--debug --python-flag=-v
and below is what I got (only meaningful excerpt).Now during compilation, I'm getting this:
Here's how I compile my program:
Here's Nuitka version:
My program's direct dependencies (aka
requirements.txt
):Any help would be really appreciated!
Best regards, Mariusz