Python runtime is configured incorrectly on Windows.

[IVLIVS-III]

@IVLIVS-III The example project still does not work after the fix. Here is the log:

Launching lib\main.dart on Windows in debug mode...
lib\main.dart:1
√  Built build\windows\runner\Debug\example.exe.
Connecting to VM Service at ws://127.0.0.1:50354/PgO4aU0b_qk=/ws
Could not find platform independent libraries <prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = 'C:\Users\Anton\Documents\Flutter\dart_python_ffi\packages\python_ffi\example\Lib'
  sys._base_executable = 'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example\\build\\windows\\runner\\Debug\\example.exe'
  sys.base_prefix = 'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example'
  sys.base_exec_prefix = 'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example'
  sys.platlibdir = 'DLLs'
  sys.executable = 'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example\\build\\windows\\runner\\Debug\\example.exe'
  sys.prefix = 'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example'
  sys.exec_prefix = 'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example'
  sys.path = [
    'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example\\build\\windows\\runner\\Debug\\python311.zip',
    'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example',
    'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example\\Lib',
    'C:\\Users\\Anton\\Documents\\Flutter\\dart_python_ffi\\packages\\python_ffi\\example\\build\\windows\\runner\\Debug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00000bfc (most recent call first):
  <no Python frame>
Lost connection to device.
Exited (sigterm)

_Originally posted by @pushkinman in https://github.com/IVLIVS-III/dart_python_ffi/issues/5#issuecomment-1600920808_

[IVLIVS-III]

Hi @pushkinman, can you provide further steps to reproduce this issue?

On my system, the example project works fine. I did the following steps:

1. Check out the dart_python_ffi repo.
2. Install melos globally on your system via dart pub global activate melos
3. Run melos bs
4. Revert changes melos bs made to packages/python_ffi_cpython_dart/lib/src/ffi/generated_bindings.dart (no longer necessary after https://github.com/IVLIVS-III/dart_python_ffi/commit/ec33eda38949be553ca0332dce44ac2f31b15b50)
5. Run the packages/python_ffi/example project
6. No issues, the flutter app launches fine

[pushkinman]

@IVLIVS-III after running melos bs i get the following errors

PS C:\Users\Anton\Documents\Flutter\dart_python_ffi> melos bs
melos bootstrap
  └> C:\Users\Anton\Documents\Flutter\dart_python_ffi

Running "flutter pub get" in workspace packages...
  ✓ lark
    └> packages/examples/lark
  ✓ python_ffi_lint
    └> packages/python_ffi_lint
  ✓ python_ffi_lint_dart
    └> packages/python_ffi_lint_dart
  ✓ basic_cli_adder
    └> packages/examples/basic_cli_adder
  ✓ scripts
    └> packages/python_ffi_cpython/scripts
  ✓ python_ffi_dart
    └> packages/python_ffi_dart
  ✓ dartpip_example
    └> packages/dartpip/example
  ✓ pytimeparse_dart
    └> packages/examples/pytimeparse_dart
  ✓ python_ffi_interface
    └> packages/python_ffi_interface
  ✓ python_ffi_cpython_dart
    └> packages/python_ffi_cpython_dart
  ✓ dartpip
    └> packages/dartpip
  ✓ python_ffi_dart_example
    └> packages/python_ffi_dart/example
  ✓ basic_dataclass
    └> packages/examples/basic_dataclass
  ✓ fj_playground
    └> packages/examples/fj_playground
  ✓ python_ffi_cpython
    └> packages/python_ffi_cpython
  ✓ python_ffi
    └> packages/python_ffi
  ✓ python_ffi_example
    └> packages/python_ffi/example
  > SUCCESS

Generating IntelliJ IDE files...
  > SUCCESS

 -> 17 packages bootstrapped

melos bootstrap [post]
  └> melos run ffigen

melos run ffigen
  └> melos exec -- "dart run ffigen
     melos run post_ffigen
     "
     └> RUNNING

$ melos exec
  └> dart run ffigen & C:\development\flutter/bin/internal/exit_with_errorlevel.bat
     └> RUNNING (in 1 packages)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------python_ffi_cpython_dart:
Running in Directory: 'c:\Users\Anton\Documents\Flutter\dart_python_ffi\packages\python_ffi_cpython_dart'
[SEVERE] : Couldn't find dynamic library in default locations.
[SEVERE] : Please supply one or more path/to/llvm in ffigen's config under the key 'llvm-path'.
Unhandled exception:
Exception: Couldn't find dynamic library in default locations.
#0      findDylibAtDefaultLocations (package:ffigen/src/config_provider/spec_utils.dart:631:3)
#1      Config._getSpecs.<anonymous closure> (package:ffigen/src/config_provider/config.dart:261:29)
#2      Config._extract (package:ffigen/src/config_provider/config.dart:247:50)
#3      new Config.fromYaml (package:ffigen/src/config_provider/config.dart:194:17)
#4      getConfigFromPubspec (package:ffigen/src/executables/ffigen.dart:113:17)
#5      getConfig (package:ffigen/src/executables/ffigen.dart:80:14)
#6      main (package:ffigen/src/executables/ffigen.dart:48:14)
<asynchronous suspension>

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
$ melos exec
  └> dart run ffigen & C:\development\flutter/bin/internal/exit_with_errorlevel.bat
     └> FAILED (in 1 packages)
        └> python_ffi_cpython_dart (with exit code 255)

melos run ffigen
  └> melos exec -- "dart run ffigen
     melos run post_ffigen
     "
     └> FAILED
ScriptException: The script ffigen failed to execute.

ScriptException: The script command/bootstrap/hooks/post failed to execute.

and then the exact same error as before when trying to start debuging on windows

[IVLIVS-III]

@pushkinman yep, I forgot that invoking package:ffigen requires llvm to be installed on your system.

However, we discard the output of ffigen anyways, see step 4:

4. Revert changes melos bs made to packages/python_ffi_cpython_dart/lib/src/ffi/generated_bindings.dart

This means, failing to run ffigen is not an issue… (I could decouple ffigen from bootstrap actually)

Anyways, I would have expected the app to run.

---

Which versions of Python did you install on your system and where are the install locations?

[pushkinman]

Python 3.11.4 C:\Users\Anton\AppData\Local\Microsoft\WindowsApps\python.exe

[IVLIVS-III]

Can you run the packages/examples/fj_playground example?

Sadly I cannot reproduce your issue with packages/python_ffi/example locally…

[pushkinman]

You are using a windows machine, right? (to test)

[IVLIVS-III]

You are using a windows machine, right? (to test)

Yes, to debug this issue I use a native Windows OS (no virtualization) on mac Intel hardware, i.e. natively booting directly into Windows.

[pushkinman]

I'm using bootcamp on Intel MacBook as well.

[pushkinman]

Can you run the packages/examples/fj_playground example?

Sadly I cannot reproduce your issue with packages/python_ffi/example locally…

It reproduces a similar error. Let me double check everything on my side, if you are saying that it works fine on your side, it should work for me as well.

[IVLIVS-III]

Yes, both work fine on my side…

[pushkinman]

Hello @IVLIVS-III, we've tested your example project as well as our own one, but we still get the same error. (We've tested on different Windows machines)

C:\Users\Pavel\dns-advisor-poc\ui>flutter run
Connected devices:
Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.19045.3086]
Chrome (web)      • chrome  • web-javascript • Google Chrome 114.0.5735.134
Edge (web)        • edge    • web-javascript • Microsoft Edge 114.0.1823.58
[1]: Windows (windows)
[2]: Chrome (chrome)
[3]: Edge (edge)
Please choose one (or "q" to quit): 1
Launching lib\main.dart on Windows in debug mode...
Building Windows application...                                   208.1s
√  Built build\windows\runner\Debug\dns_advisor_poc.exe.
Syncing files to device Windows...                               2,262ms

Flutter run key commands.
r Hot reload.
R Hot restart.
h List all available interactive commands.
d Detach (terminate "flutter run" but leave application running).
c Clear the screen
q Quit (terminate the application on the device).

A Dart VM Service on Windows is available at: http://127.0.0.1:49767/6rKNrlNPnx8=/
Could not find platform independent libraries <prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = 'C:\Users\Pavel\dns-advisor-poc\ui\Lib'
  sys._base_executable = 'C:\\Users\\Pavel\\dns-advisor-poc\\ui\\build\\windows\\runner\\Debug\\dns_advisor_poc.exe'
  sys.base_prefix = 'C:\\Users\\Pavel\\dns-advisor-poc\\ui'
  sys.base_exec_prefix = 'C:\\Users\\Pavel\\dns-advisor-poc\\ui'
  sys.platlibdir = 'DLLs'
  sys.executable = 'C:\\Users\\Pavel\\dns-advisor-poc\\ui\\build\\windows\\runner\\Debug\\dns_advisor_poc.exe'
  sys.prefix = 'C:\\Users\\Pavel\\dns-advisor-poc\\ui'
  sys.exec_prefix = 'C:\\Users\\Pavel\\dns-advisor-poc\\ui'
  sys.path = [
   'C:\\Users\\Pavel\\dns-advisor-poc\\ui\\build\\windows\\runner\\Debug\\python311.zip',
   'C:\\Users\\Pavel\\dns-advisor-poc\\ui',
   'C:\\Users\\Pavel\\dns-advisor-poc\\ui\\Lib',
   'C:\\Users\\Pavel\\dns-advisor-poc\\ui\\build\\windows\\runner\\Debug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000120c (most recent call first):
<no Python frame>
The Flutter DevTools debugger and profiler on Windows is available at:
[http://127.0.0.1:9100?uri=http://127.0.0.1:49767/6rKNrlNPnx8=/](http://127.0.0.1:9100/?uri=http://127.0.0.1:49767/6rKNrlNPnx8=/)
Lost connection to device.

[maxim-saplin]

@IVLIVS-III I've tried running fj_playground example from master channel (as of yesterday) on Windows 11, when Python is installed, the app starts. When Python is removed the app doesn't work and fails with same error log as above:

Could not find platform independent libraries <prefix>
Python path configuration:
...

Python runtime and all dependencies are supposed to be bundled with the app and not depend on local installation of Python - is that right?

[IVLIVS-III]

Python runtime and all dependencies are supposed to be bundled with the app and not depend on local installation of Python - is that right?

@maxim-saplin yes, correct, everything should be bundled when running the Flutter specific plugin python_ffi as is the case in the fj_playground example.

Thanks for this valuable insight, I will try and get a better testing setup for Windows shortly.

[maxim-saplin]

@IVLIVS-III , when do you think you'll be able to check bundling for Windows? Looking forward to testing it)

[IVLIVS-III]

when do you think you'll be able to check bundling for Windows?

@maxim-saplin not before 5th July 2023.

Edit: Since the act of solely bundling the Python runtime for Windows would not interfere with my research / bachelors thesis, I am open to review and accept a PR that addresses this issue in case you have more resources than myself, and in case you need a more timely fix than I could deliver. It looks like you have better testing infrastructure for Windows at the moment.

[maxim-saplin]

I could try submitting the PR. Though would appreciate your guidance, what should I start with)

[IVLIVS-III]

for macOS I settled on compiling the Python runtime myself (including code signing for Apple Distribution).

See here for the patch as well as some scripts I created for simplifying the build process.

I also started to port the patch to Linux and Windows, but without any luck since.

To successfully bundle the runtime, one would need to sync the compiled dynamic Python library with the distributed Python stdlib (zip-file). That's what I did for macOS with the patch and custom build.

I would prefer to distribute / bundle only a single stdlib-zip-file across all platforms and have custom compiled dynamic libraries that work together with this version of the stdlib.

For now you could either start to

1. use the official Python DLL from the embeddable Windows package on python.org, but tweak the bundled stdlib-zip-file or
2. compile a custom DLL that works with the currently distributed stdlib-zip-file (as I did for macOS).

In the long term I would prefer the second option, but in the short term it might be easier to go after option 1.

As long as it works, I'll be happy to accept a solution for now.

[IVLIVS-III]

The root cause for the above error is that the currently used (main) DLL (python311.dll) expects auxiliary dlls to be available in the PYTHON_PATH but those are not bundled with the plugin (and I would like to keep it that way). That's why for macOS, I statically included those auxiliary dlls in the libpython311.dylib when building from source. After that it worked on macOS.

See also lines 14-328 in this patch file

[maxim-saplin]

OK, thanks for the direction. Will pick it up from here.

[maxim-saplin]

Tried running dart run scripts/bin/scripts.dart cpython all and the patch process breaks with configure files:

Applying patch './patches/windows3.11.3.patch'...

C:/Users/user/Documents/GitHub/dart_python_ffi/packages/python_ffi_cpython/patches/windows3.11.3.patch:7: trailing whitespace.    
# -*- makefile -*-
C:/Users/user/Documents/GitHub/dart_python_ffi/packages/python_ffi_cpython/patches/windows3.11.3.patch:8: trailing whitespace.    
# The file Setup is used by the makesetup script to construct the files
C:/Users/user/Documents/GitHub/dart_python_ffi/packages/python_ffi_cpython/patches/windows3.11.3.patch:9: trailing whitespace.    
# Makefile and config.c, from Makefile.pre and config.c.in,
C:/Users/user/Documents/GitHub/dart_python_ffi/packages/python_ffi_cpython/patches/windows3.11.3.patch:10: trailing whitespace.   
# respectively.  Note that Makefile.pre is created from Makefile.pre.in
C:/Users/user/Documents/GitHub/dart_python_ffi/packages/python_ffi_cpython/patches/windows3.11.3.patch:11: trailing whitespace.   
# by the toplevel configure script.
warning: configure has type 100644, expected 100755
error: patch failed: configure:23615
error: configure: patch does not apply
error: patch failed: configure.ac:6887
error: configure.ac: patch does not apply

❌ error (code 1): git failed applying patch

While I'm investigating the build process, could you please explain what are those auxiliary DLLs that need to be included statically? Besides, if I get it right, the Dart scripts clone CPython and use it to build Python runtime as standalone dynamic library?

[IVLIVS-III]

@maxim-saplin Looks like the issue is line endings in the patch file. At least that's what I would guess reading the error message… you could manually apply the patch, i.e. copy the lines over manually.

Auxiliary DLLs are separate DLLs for each module of the Python standard library which are crated during the cpython build process. They can be statically linked into the main DLL with the Setup.local file.

Yes, once complete you should have a single standalone Python dynamic library, which you could place into the windows folder (replacing python311.dll).

[maxim-saplin]

1. use the official Python DLL from the embeddable Windows package on python.org, but tweak the bundled stdlib-zip-file or

This options sound like an easier one. Going into CPython compilation might be a dead end for me. Do you mean that: 1) I need to download embedded Python package (e.g. https://www.python.org/ftp/python/3.11.4/python-3.11.4-embed-amd64.zip), take python311.dll from there and put it to windows folder? 2) Then under/python_ffi_cpython/assets/ there should be a separate version of python3.11.zip which is tailored (somehow) to Windows 3) Tailor Python init in the rest of the packages assuming that for Windows there's a separate stdlib zip file

[IVLIVS-III]

@maxim-saplin yes, I meant exactly these steps.

And yes, I too feel that compiling CPython on Windows is the more difficult option.

[maxim-saplin]

Do you have an advice/direction for python3.11.zip? Where can I start? Sorry for lot's of questions, my Dart side is stronger than Python/C)

[IVLIVS-III]

I assume all modules listed here should be included. All of those modules combined are called the Python standard library. Most of them have C implementations for speed benefits, which are compiled into said auxiliary DLLs by default. For macOS I recompiled CPython and statically included all C implementations of those Python modules directly in libpython3.11.dylib. If we don't want to recompile python311.dll for Windows, we should include a full (pure) Python inplementation of those modules inside the python3.11.zip and somehow tell the Python runtime not to try and load the C implementations of those modules.

[maxim-saplin]

Quick status updated (https://github.com/maxim-saplin/dart_python_ffi/commit/3bcb75a6fe2e5abb001c5657acf8bad71df074ac):

• Added x64 version of Python DLL
• Updated the ZIP file with Lib contents from CPython
• Started playing with _initializeRuntimeDefault() method to make the app working

Few times I even managed to get the UI started (fj example), yet is isInitialized returned false due to 0 in Py_IsInitialized():

bool get isInitialized =>
      areBindingsInitialized && bindings.Py_IsInitialized() != 0;

Iterated more on runtime init, sporadically at different parts of _initializeRuntimeDefault() the app crashed with no error message. The current version from above commit doesn't start (forget what I did differently when the UI was showing up).

Windows logs show the following error:

Faulting application name: fj_playground.exe, version: 0.1.0.0, time stamp: 0x649d76f3
Faulting module name: ntdll.dll, version: 10.0.22000.2057, time stamp: 0x8dc95719
Exception code: 0xc0000374
Fault offset: 0x000000000010c429
Faulting process id: 0x17dc
Faulting application start time: 0x01d9aa8408baf20b
Faulting application path: C:\Users\user\Documents\GitHub\dart_python_ffi\packages\examples\fj_playground\build\windows\runner\Debug\fj_playground.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: 981f2c3e-1cb0-4e6d-b761-d07468e68a63
Faulting package full name: 
Faulting package-relative application ID: 

Note the exception code (374) - there're plenty of search results for python 0xc0000374.

[IVLIVS-III]

great, looks kinda promising… thanks a lot for the status update. How did you fetch the error logs on Windows?

[maxim-saplin]

Checked system logs via Windows Event Viewer. Will iterate over the weekend, let's see how it goes

[maxim-saplin]

@IVLIVS-III , did another try. Unfortunately blindly iterating over code with little understanding of behind the scenes didn't provide any clues. Currently I don't have the capacity to proceed with further investigation. I'm happy to do testing on my Windows environment.

We'll be looking forward to your inputs. Please let us know if you won't be able to do the Windows runtime next week. We might need to go with a backup plan and use PyInstaller and rely on external app and communication via files.

[IVLIVS-III]

@maxim-saplin I cannot guarantee that I will be able to work on the Windows runtime next week. At most I will be able to allocate 3-5 hours on Wednesday 5th July 2023.

Even with your valuable advances, I don't think this time slot will be enough to solve this issue. Besides the above mentioned time slot, I expect to continue no earlier than Monday 10th July 2023.

I am committed to eventually supporting Windows, macOS, and Linux equally. But please keep in mind that my current priority is in other areas of this project. If you need short-term support/availability guarantees, I am afraid, I cannot provide those at the moment. After all Windows support is currently declared to be in alpha.

Still, I'd like to continue debugging this issue together, or at least getting additional testing from your side once I make substantial advances.

[maxim-saplin]

Thanks for the update and clarifications! Let me know when you need my help with testing