Nuitka / Nuitka

Nuitka is a Python compiler written in Python. It's fully compatible with Python 2.6, 2.7, 3.4-3.12. You feed it your Python app, it does a lot of clever things, and spits out an executable or extension module.
http://nuitka.net
Apache License 2.0
11.94k stars 642 forks source link

IMPORT_HARD_CTYPES : failed hard import of ctypes #2973

Closed mgeeky closed 3 months ago

mgeeky commented 3 months ago

Hi,

I'm having issues compiling my programs that used to compile just fine a few nuitka versions ago (can't really tell what was the last working Nuitka version for me).

Whenever I run the executable, there's simply no output whatsever, even though I'm building console programs. My programs don't import ctypes directly, but I have some dependencies that might import it indirectly.

I took the liberty to compile program with --debug --python-flag=-v and below is what I got (only meaningful excerpt).

Now during compilation, I'm getting this:

import _ctypes # LoadLibraryExW("C:\Program\Path\_ctypes.pyd");
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\Program\Path\ctypes\__init__.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
ImportError: LoadLibraryExW 'C:\Program\Path\_ctypes.pyd' failed: The specified module could not be found.
# destroy _ctypes
# destroy ctypes
IMPORT_HARD_CTYPES : "failed hard import of 'ctypes'"
Assertion failed: false, file C:\Users\user\AppData\Roaming\Python\Python311\site-packages\nuitka\build\static_src\HelpersImportHard.c, line 120

Here's how I compile my program:

py -3.11 -m nuitka --msvc=latest --include-data-dir="C:\Program\Path\assets\data=assets\data" --follow-imports --include-module=win32api --debug --python-flag=-v --unstriped --show-modules --include-module=win32api --noinclude-custom-mode=PIL:nofollow --noinclude-custom-mode=scipy:nofollow --noinclude-custom-mode=PyQt2:nofollow --noinclude-custom-mode=PyQt3:nofollow --noinclude-custom-mode=PyQt4:nofollow --noinclude-custom-mode=PyQt5:nofollow --noinclude-custom-mode=PyQt6:nofollow --noinclude-custom-mode=PyQt7:nofollow --noinclude-custom-mode=PySide2:nofollow --noinclude-custom-mode=PySide3:nofollow --noinclude-custom-mode=PySide4:nofollow --noinclude-custom-mode=PySide5:nofollow --noinclude-custom-mode=PySide6:nofollow --noinclude-custom-mode=PySide7:nofollow --python-flag=no_site --python-flag=no_docstrings --windows-icon-from-ico="C:\Program\Path\assets\icons\icon.png" --onefile-tempdir-spec="{TEMP}\tmp{PID}{TIME}" --enable-plugin=anti-bloat --enable-plugin=pylint-warnings --enable-plugin=data-hiding --enable-plugin=anti-debugger --warn-unusual-code --assume-yes-for-downloads --windows-console-mode=force -o "program.exe" --onefile --output-dir="C:\Program\Path\release\_temp" "program.py"

Here's Nuitka version:

2.3.9
Commercial: 2.6.1
Python: 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
Flavor: CPython Official
Executable: C:\Python311\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 11
Version C compiler: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\Hostx64\x64\cl.exe (cl 14.3).

My program's direct dependencies (aka requirements.txt):

pywin32
python-dateutil
olefile
pefile
pyyaml
lxml
jsmin
pycdlib
pypdf2
wmi
pycryptodome
requests
pyjwt
pyjwt[crypto]
prettytable
colorama
tqdm
pylnk3
psutil
pyreadline3
olefile
pyOpenSSL
prompt_toolkit
python-minifier
pyautogui
paramiko
asn1crypto
icoextract

# Nuitka and Docs requirements
mkdocs
mkdocs-material
ordered-set 
zstandard 
imageio 
bcj-cffi

Any help would be really appreciated!

Best regards, Mariusz

kayhayen commented 3 months ago

Can you compile with --python-flag=-v and show the imports right before _ctypes

mgeeky commented 3 months ago

So it looks like colorama might be my problematic import. One I cannot really do away, as my programs heavily use colored console outputs:

PS C:\Program\Path> .\program.exe -h
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
# installing zipimport hook
import 'time' # <class '_frozen_importlib.BuiltinImporter'>
import 'zipimport' # <class '_frozen_importlib.FrozenImporter'>
# installed zipimport hook
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <class '_frozen_importlib.FrozenImporter'>
import 'encodings.aliases' # <class '_frozen_importlib.FrozenImporter'>
import 'encodings' # <class '_frozen_importlib.FrozenImporter'>
import 'encodings.utf_8' # <class '_frozen_importlib.FrozenImporter'>
import 'encodings.cp1250' # <class '_frozen_importlib.FrozenImporter'>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <class '_frozen_importlib.FrozenImporter'>
import 'io' # <class '_frozen_importlib.FrozenImporter'>
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <class '_frozen_importlib.FrozenImporter'>
import '_collections_abc' # <class '_frozen_importlib.FrozenImporter'>
import 'genericpath' # <class '_frozen_importlib.FrozenImporter'>
import '_winapi' # <class '_frozen_importlib.BuiltinImporter'>
import 'ntpath' # <class '_frozen_importlib.FrozenImporter'>
import 'os' # <class '_frozen_importlib.FrozenImporter'>
import 'warnings' # <class '_frozen_importlib.FrozenImporter'>
import 'importlib' # <class '_frozen_importlib.FrozenImporter'>
Setup nuitka compiled module/bytecode/extension importer.
import inspect # considering responsibility (find_spec)
import inspect # claimed responsibility (bytecode)
import inspect # created module
import inspect # execute module
import ast # considering responsibility (find_spec)
import ast # claimed responsibility (bytecode)
import ast # created module
import ast # execute module
import '_ast' # <class '_frozen_importlib.BuiltinImporter'>
import contextlib # considering responsibility (find_spec)
import contextlib # claimed responsibility (bytecode)
import contextlib # created module
import contextlib # execute module
import collections # considering responsibility (find_spec)
import collections # claimed responsibility (bytecode)
import collections # created module
import collections # execute module
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
import keyword # considering responsibility (find_spec)
import keyword # claimed responsibility (bytecode)
import keyword # created module
import keyword # execute module
import 'keyword' # <class 'nuitka_module_loader'>
import operator # considering responsibility (find_spec)
import operator # claimed responsibility (bytecode)
import operator # created module
import operator # execute module
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <class 'nuitka_module_loader'>
import reprlib # considering responsibility (find_spec)
import reprlib # claimed responsibility (bytecode)
import reprlib # created module
import reprlib # execute module
import 'reprlib' # <class 'nuitka_module_loader'>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <class 'nuitka_module_loader'>
import functools # considering responsibility (find_spec)
import functools # claimed responsibility (bytecode)
import functools # created module
import functools # execute module
import types # considering responsibility (find_spec)
import types # claimed responsibility (bytecode)
import types # created module
import types # execute module
import 'types' # <class 'nuitka_module_loader'>
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
import 'functools' # <class 'nuitka_module_loader'>
import 'contextlib' # <class 'nuitka_module_loader'>
import enum # considering responsibility (find_spec)
import enum # claimed responsibility (bytecode)
import enum # created module
import enum # execute module
import 'enum' # <class 'nuitka_module_loader'>
import 'ast' # <class 'nuitka_module_loader'>
import dis # considering responsibility (find_spec)
import dis # claimed responsibility (bytecode)
import dis # created module
import dis # execute module
import opcode # considering responsibility (find_spec)
import opcode # claimed responsibility (bytecode)
import opcode # created module
import opcode # execute module
import '_opcode' # <class '_frozen_importlib.BuiltinImporter'>
import 'opcode' # <class 'nuitka_module_loader'>
import 'dis' # <class 'nuitka_module_loader'>
import collections.abc # considering responsibility (find_spec)
import collections.abc # claimed responsibility (bytecode)
import collections.abc # created module
import collections.abc # execute module
import 'collections.abc' # <class 'nuitka_module_loader'>
import importlib.machinery # considering responsibility (find_spec)
import importlib.machinery # claimed responsibility (bytecode)
import importlib.machinery # created module
import importlib.machinery # execute module
import 'importlib.machinery' # <class 'nuitka_module_loader'>
import linecache # considering responsibility (find_spec)
import linecache # claimed responsibility (bytecode)
import linecache # created module
import linecache # execute module
import tokenize # considering responsibility (find_spec)
import tokenize # claimed responsibility (bytecode)
import tokenize # created module
import tokenize # execute module
import re # considering responsibility (find_spec)
import re # claimed responsibility (bytecode)
import re # created module
import re # execute module
import re._compiler # considering responsibility (find_spec)
import re._compiler # claimed responsibility (bytecode)
import re._compiler # created module
import re._compiler # execute module
import '_sre' # <class '_frozen_importlib.BuiltinImporter'>
import re._parser # considering responsibility (find_spec)
import re._parser # claimed responsibility (bytecode)
import re._parser # created module
import re._parser # execute module
import re._constants # considering responsibility (find_spec)
import re._constants # claimed responsibility (bytecode)
import re._constants # created module
import re._constants # execute module
import 're._constants' # <class 'nuitka_module_loader'>
import 're._parser' # <class 'nuitka_module_loader'>
import re._casefix # considering responsibility (find_spec)
import re._casefix # claimed responsibility (bytecode)
import re._casefix # created module
import re._casefix # execute module
import 're._casefix' # <class 'nuitka_module_loader'>
import 're._compiler' # <class 'nuitka_module_loader'>
import copyreg # considering responsibility (find_spec)
import copyreg # claimed responsibility (bytecode)
import copyreg # created module
import copyreg # execute module
import 'copyreg' # <class 'nuitka_module_loader'>
import 're' # <class 'nuitka_module_loader'>
import token # considering responsibility (find_spec)
import token # claimed responsibility (bytecode)
import token # created module
import token # execute module
import 'token' # <class 'nuitka_module_loader'>
import 'tokenize' # <class 'nuitka_module_loader'>
import 'linecache' # <class 'nuitka_module_loader'>
import 'inspect' # <class 'nuitka_module_loader'>
# destroy __main__
import argparse # considering responsibility (find_spec)
import argparse # claimed responsibility (bytecode)
import argparse # created module
import argparse # execute module
import gettext # considering responsibility (find_spec)
import gettext # claimed responsibility (bytecode)
import gettext # created module
import gettext # execute module
import 'gettext' # <class 'nuitka_module_loader'>
import 'argparse' # <class 'nuitka_module_loader'>
import textwrap # considering responsibility (find_spec)
import textwrap # claimed responsibility (bytecode)
import textwrap # created module
import textwrap # execute module
import 'textwrap' # <class 'nuitka_module_loader'>
import lib # considering responsibility (find_spec)
import lib # claimed responsibility (compiled)
import lib # created module
import lib # execute module
Loaded lib
import 'lib' # <class 'nuitka_module_loader'>
import lib.logger # considering responsibility (find_spec)
import lib.logger # claimed responsibility (compiled)
import lib.logger # created module
import lib.logger # execute module
import colorama # considering responsibility (find_spec)
import colorama # claimed responsibility (compiled)
import colorama # created module
import colorama # execute module
import colorama.initialise # considering responsibility (find_spec)
import colorama.initialise # claimed responsibility (compiled)
import colorama.initialise # created module
import colorama.initialise # execute module
import 'atexit' # <class '_frozen_importlib.BuiltinImporter'>
import colorama.ansitowin32 # considering responsibility (find_spec)
import colorama.ansitowin32 # claimed responsibility (compiled)
import colorama.ansitowin32 # created module
import colorama.ansitowin32 # execute module
import colorama.ansi # considering responsibility (find_spec)
import colorama.ansi # claimed responsibility (compiled)
import colorama.ansi # created module
import colorama.ansi # execute module
Loaded colorama.ansi
import 'colorama.ansi' # <class 'nuitka_module_loader'>
import colorama.winterm # considering responsibility (find_spec)
import colorama.winterm # claimed responsibility (compiled)
import colorama.winterm # created module
import colorama.winterm # execute module
import 'msvcrt' # <class '_frozen_importlib.BuiltinImporter'>
import colorama.win32 # considering responsibility (find_spec)
import colorama.win32 # claimed responsibility (compiled)
import colorama.win32 # created module
import colorama.win32 # execute module
import ctypes # considering responsibility (find_spec)
import ctypes # claimed responsibility (bytecode)
import ctypes # created module
import ctypes # execute module
import _ctypes # considering responsibility (find_spec)
import _ctypes # claimed responsibility (extension)
import _ctypes # created module
import _ctypes # execute module
import _ctypes # LoadLibraryExW("C:\Program\Path\_ctypes.pyd");
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "C:\Program\Path\ctypes\__init__.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
ImportError: LoadLibraryExW 'C:\Program\Path\_ctypes.pyd' failed: The specified module could not be found.
# destroy _ctypes
# destroy ctypes
IMPORT_HARD_CTYPES : "failed hard import of 'ctypes'"
Assertion failed: false, file C:\Users\Mariusz\AppData\Roaming\Python\Python311\site-packages\nuitka\build\static_src\HelpersImportHard.c, line 120
kayhayen commented 3 months ago

Nobody is asking you to change anything, although I can see how you want a workaround, I usually want to fix the underlying issue. I wonder, if this is because of a dirt effect, with Nuitka's inline copy of colorama, because I don't see how. I do of course not reproduce this myself, the import of ctypes is pretty clearly visible. Can we first eliminate the --onefile usage and go to mere standalone and see if it happens for you then? I have a suspect, that it would have to be on that level somehow, and then of course the question becomes what might be missing for it to load. Also if you compile only "import colorama.win32" and run that, does it happen too?

mgeeky commented 3 months ago

Hi @kayhayen - thank you so much for a swift response!

So after merely changing from --onefile into --standalone program worked as expected! Looks like indeed there's problem with --onefile merge of implicit dependencies.

I did two additional tests:

  1. Built a colorama only hello world as --onefile executable, worked without issues.
  2. Built a second test program including all the dependencies I've seen fitting - also worked as expected.

Here's second program's imports list I've used (again, this has built without any issues):

import colorama
import atexit
import re
import os
import csv
import glob
import requests
import ssl
import json
import hashlib
import random
import shutil
import tempfile
import string
import pefile
import socket
import datetime
import binascii
import threading
import time
import urllib3
import yaml

import win32com
import win32com.client
import win32api
import win32con
import pythoncom

from tqdm import tqdm
from pathlib import Path
from OpenSSL import crypto
from OpenSSL.crypto import _lib, _ffi, X509
from urllib.parse import urlparse
from dateutil.parser import *
from dateutil import tz
from asn1crypto import pkcs12
from collections.abc import Iterable

from PyPDF2 import PdfReader, PdfWriter
from PyPDF2.generic import DecodedStreamObject, NameObject, DictionaryObject, createStringObject, ArrayObject

import xml.etree.ElementTree as ET
import urllib.request
import shutil

from pathlib import Path
from urllib.parse import urlparse,urlunparse,urljoin
from lxml import etree
from tqdm import tqdm
from urllib.parse import urlparse, parse_qs

from time import strftime, localtime
from datetime import timedelta
from typing import *

So at this point, I'm really clueless how to further isolate my problem or how to come up with a minimal test case script.

It definitely looks like PyImport_ImportModule("ctypes") fails to execute for --onefile whereas it doesn't for --standalone. A close look into files and directories comprising standlone build indicates that there's _ctypes.pyd file present locally. Maybe during --onefile build that module doesn't get copied into local build directory, which could be causing import to fail?

Any fresh ideas for a workaround or how to proceed from here?

Best, Mariusz

kayhayen commented 3 months ago

Can you remove your onefile tempdir spec and try that way. It looks good to me, but it might somehow not be used properly, if you can, maybe make the program not terminate, and check unpacked files in the temp directory.

kayhayen commented 3 months ago

I see the option that shell quoting is your enemy there, maybe. Nuitka reports your command line options back to you, and a compilation report would have them too. If the \ was misinterpreted by a shell, say git bash, that could cause funny things. Your value doesn't look too different from the default, "{TEMP}" + os.path.sep + "onefile_{PID}_{TIME}" but the {TEMP} may not give a slash, and then you would be writing to a place where you may not even be allowed to write, or put files, not sure now. It may also be that the slash I push there is not even needed. With --experimental=debug-onefile-handling it would output the payload path that results.

I think it would have to be that, since onefile is used a lot and hasn't changed in a long time for those things.

mgeeky commented 3 months ago

Hi there!

Coming back with an update, looks like it's all sorted now! After going through all of the imported modules dropped to the standalone build.dist directory I found that for some reason numpy + matplotlib + cv2 were implicitely imported, even though my program shouldn't be using those.

So in a first step I've explicitely denied importing them. Then I've also removed --onedir-tempdir-spec as suggested and added ctypes import explicitely, just in case.

As a result, these flags were added on top of previously used ones:

--include-module=ctypes --noinclude-custom-mode=numpy:nofollow --noinclude-custom-mode=numpy.libs:nofollow --noinclude-custom-mode=matplotlib:nofollow --noinclude-custom-mode=cv2:nofollow

Seems they did the trick as I'm now up and running again!

Thank you so much @kayhayen for quickly finding a workaround.

This probably leaves us with some outstanding issues in how the --onefile-tempdir-spec parameter and its inner slashes/backslashes might be handled. Possibly we could add a statement to the parameter description that user is expected to use slashes instead of backslashes.

Best regards, Mariusz.

mgeeky commented 3 months ago

I'm sorry to report that the problem still remains, don't know how in the world it compiled these seven hours ago but I cannot seem to compile it back on again so that it would work as previously.

I mean, I'm compiling without --onefile-tempdir-spec but the problem seems to remain. --onefile compiled program keeps crashing on being unable to import ctypes.

Now I'm really clueless how to troubleshoot this :(

kayhayen commented 3 months ago

Your report appears to make no sense to me. What could be going on is that some module pulls in a conflicting runtime DLL. Since no reproducer has been provided, there is not much I can do without that.

kayhayen commented 3 months ago

Addressing my doubts about onefile spec, indeed it can be used in the wrong way.

Nuitka: Launching 'HelloWorldTest.bin'.
Error, failed to open '/tmp_/HelloWorldTest.bin' for writing.

These system directories need to require being followed by a slash. If your quoting is off, the slash may create a newline if e.g. used like \n and interpreted by the shell, which is what I was suspecting there. Unfortunately you never made it clear if you use the correct command line or not as echoed back by Nuitka. Checking for a slash will make it harder to have escaping issues directly following that slash. Your \t might be meaning you create an illegal path. I recently added a function isLegalPath and it seems Windows rejects those kinds of characters.

kayhayen commented 3 months ago

FATAL: Cannot use general system folder {TEMP}, without a path separator '--onefile-tempdir-spec={TEMP}_', just appending to these is not allowed, needs to be below them.

So this is a new checked I added for 2.4, which makes it more like to catch it.

kayhayen commented 3 months ago

Seems I am wrong about \t being an illegal path on Windows, works just fine in my test just now. Probably not a good idea, but I imagine people will like using non-printable characters to hide it or whatever, so I will not interfere with it and just make it a warning.

kayhayen commented 3 months ago

Ah, now I fooled myself, was using absolute path \t\t and that works of course.

kayhayen commented 3 months ago

Ok, but we now have errors like those:

FATAL: Cannot use illegal paths '--onefile-tempdir-spec={TEMP}\something.', due to contains illegal suffix '.'.

FATAL: Cannot use illegal paths '--onefile-tempdir-spec={TEMP}\ \something.', due to contains illegal suffix ' ' in path part ' '.

But what ought to happen is be unable to write the main executable already, so it probably also is never your issue, it might be AV preventing the load of files it still checks or something like that, but that's only a guess.

mgeeky commented 3 months ago

Alright, so here's what I found:

  1. Artifact after successful compilation gets saved to --output-dir=C:\build path.
  2. In this directory there are few more dirs and files created:
C:\Build
C:\Build\program.build
C:\Build\program.dist
C:\Build\program.onefile-build
C:\Build\program.exe
C:\Build\program.pdb
  1. Now if I enter C:\Build directory and then run program.exe - it runs just fine
  2. If I move that executable file outside of this C:\Build - it breaks due to being unable to find ctypes

Any ideas how to proceed next?

kayhayen commented 3 months ago

It makes no sense to me, except if you have a spurious corruption issue. I recently fixed one for lazy module values, maybe retry with current factory branch.

mgeeky commented 3 months ago

Which branch you refer to? Also, are we speaking Nuitka commercial (I'm a subscriber) or the public repo one?

kayhayen commented 3 months ago

For commercial, the staging branch has the fix, and public the factory branch has it.

mgeeky commented 3 months ago

I think this might be my culprit:

Unexpected output from this command:
<clcache> /Fostatic_src\OnefileBootstrap.obj /c "static_src\OnefileBootstrap.c" /nologo /GL /std:c11 /wd5105 -O2 /Z7 /FS /EHsc /J /Gd /bigobj /W4 /wd4505 /wd4127 /wd4100 /wd4702 /wd4189 /wd4211 /WX /wd4512 /wd4510 /wd4610 /wd4996 /MT /D_NUITKA_NO_DEPLOYMENT_SELF_EXECUTION /D_NUITKA_ONEFILE_MODE /D_NUITKA_EXPERIMENTAL_INIT_PROGRAM=1 /D_NUITKA_EXPERIMENTAL_WRITEABLE_CONSTANTS=1 /D_NUITKA_PLUGIN_MULTIPROCESSING_ENABLED=1 /IC:\PYTHON~2\Lib\SITE-P~1\nuitka\build\inline_copy\zlib /I. /IC:\PYTHON~2\Lib\SITE-P~1\nuitka\build\include /IC:\PYTHON~2\Lib\SITE-P~1\nuitka\build\static_src /IC:\PYTHON~2\Lib\SITE-P~1\nuitka\build\inline_copy\zstd
static_src\OnefileBootstrap.c(827): error C2220: the following warning is treated as an error
static_src\OnefileBootstrap.c(827): warning C4267: '+=': conversion from 'size_t' to 'int', possible loss of data
scons: *** [static_src\OnefileBootstrap.obj] Error 2
FATAL: Error, onefile bootstrap binary build failed.

Here's the OnefileBootstrap.c(827) for context:

#if _WIN32
static wchar_t *getCommandLineForChildProcess(void) {
    int argc;
    LPWSTR *argv = CommandLineToArgvW(GetCommandLineW(), &argc);

    assert(argc > 0);
    argv[0] = getBinaryPath();

    // Free memory allocated for CommandLineToArgvW arguments.
    LocalFree(argv);

    int size = 0;

    for (int i = 0; i < argc; i++) {
        size += wcslen(argv[i]);              // <=== 827 line
    }

    size += argc;

    wchar_t *result = malloc((size + 1) * sizeof(wchar_t));
    result[0] = 0;

    for (int i = 0; i < argc; i++) {
        appendWStringSafeW(result, argv[i], size);

        if (i != argc - 1) {
            appendCharSafeW(result, ' ', size);
        }
    }

    return result;
}
#endif

Obviously to satsify MSVC compiler, size variable should be of type size_t instead of current int.

Moreover, this compilation crash happens only when I plug in --debug flag.

Debug mode turns on MSVC's /W4 compilation flag, which causes all warnings to fail as errors. The flag specifications is included in nuitka/build/Onefile.scons script.

When I turn off --debug, program runs fine.

kayhayen commented 3 months ago

I wasn't aware of this one yet for some reason. You can make it ignore warnings with --experimental=allow-c-warnings which unfortunately is needed for Python generated code sometimes too.

However, this code may indeed to be to blame for your issues. I need to think of what could be wrong there and how your report ties into it. This seems really bad, but it totally seems reasonable that it could cause it.

kayhayen commented 3 months ago

The warning itself is not a bug, it's not relevant, but if that code malfunctions, it would explain things maybe.

kayhayen commented 3 months ago

This code messes with sys.argv and tries to make it absolute, such that if invoked non-absolute, os.path.dirname(sys.argv) will be a reliable path to telling where the program is launched from. This wasn't yet in the code that you observed the issue first with.

I updated staging, so --debug won't give you the issue with onefile anymore.

mgeeky commented 3 months ago

Thank you @kayhayen for your time working this out with me! Happy I could help. Since I'm not having this particular issue anymore and I know you've got a lot on your plate already, I guess we can close this report.

Cheers, Mariusz.