Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

[sl5net]

> from 1 | grep .git
Python path configuration:
PYTHONHOME = '/tmp/.mount_Extrat1GMhA4/usr/'
PYTHONPATH = '/tmp/.mount_Extrat1GMhA4/usr/share/pyshared/:'
program name = '/usr/bin/python3'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = '/usr/bin/python3'
sys.base_prefix = '/tmp/.mount_Extrat1GMhA4/usr'
sys.base_exec_prefix = '/tmp/.mount_Extrat1GMhA4/usr'
sys.platlibdir = 'lib'
sys.executable = '/usr/bin/python3'
sys.prefix = '/tmp/.mount_Extrat1GMhA4/usr'
sys.exec_prefix = '/tmp/.mount_Extrat1GMhA4/usr'
sys.path = [
'/tmp/.mount_Extrat1GMhA4/usr/share/pyshared/',
'',
'/tmp/.mount_Extrat1GMhA4/usr/lib/python310.zip',
'/tmp/.mount_Extrat1GMhA4/usr/lib/python3.10',
'/tmp/.mount_Extrat1GMhA4/usr/lib/python3.10/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fa996320740 (most recent call first):
<no Python frame>
 ~/.l/s/0/m/autocivp (master) [127]> from 1 | grep .git     

i useing

ExtratermQt-0.73.0.glibc2.29-x86_64.AppImage

[darinpp]

I have the same issue when I run the AppImage

[sedwards2009]

The dependency on Python for this is becoming a real weakness. I'm thinking of rewriting those small commands into C and using https://justine.lol/cosmopolitan/index.html to make it run (most) places out of the box.

[mtalexan]

Since the shell code is running in the context of the AppImage environment, it seems the AppImage is missing the python module encodings that's used by the from.py.

Python is notorious for not being consistently distributable, and heavily reliant on the host environment. As it is, the python-based extras will only ever work if the host system python is a supported version (which puts the effort on your development to make sure your script supports a huge range of python versions), and all the python modules you need are already installed on the host, something you can usually assume will never be true if you use any non-core modules.

Using a language that compiles to a single static binary is probably your best choice since you can't assume any external dependencies will exist on the host. Notably, that includes most C/C++ libraries outside of the stdlibs. You would need to be sure to statically compile your application, which means finding static libraries for any libraries you need and then abiding by the licensing restrictions of them (fingers crossed they don't conflict with each other!).
Commonly recommended solutions for this exact issue are Golang and Rust, which are designed for this use case and both have large libraries of public module/crates. Those languages compile everything into a single static binary that can be run in-place.

[mtalexan]

It should also be pointed out here, that Terminals are generally not something that works as an AppImage. When running a terminal you usually don't want to be in the AppImage separated environment, you instead want direct access to the host.

The error output above is a good example of how it wouldn't be possible to run python scripts on the native host from the AppImage version of Extraterm-Qt. The "system" python is the one from the AppImage.

[sedwards2009]

You don't have to convince me that Python apps are one of the hardest things to distribute to end users. I've seen the problems at work (for a desktop PyQt app).

I do have some good news though. I've been working on this problem lately and I've got C implementations of the from and show commands. (Using small, often single file libraries for the support I need.) Right now I'm trying to figure out how to compile them to ELF binaries which support Linux and the BSDs in one file. I'm not sure if AMD64 and ARM can be supporting in one file though, but it would be nice.

so, almost there.

[mtalexan]

The main way I've seen to do this is with Cosmopolitan (https://github.com/jart/cosmopolitan) that steals an idea from malware and adds a bunch of binaries for different architectures to the end of a shell-like script that acts like an architecture detection and jump table. It supports BSD and Linux, as well as x86_64, ARM, etc.

The host system has to have binfmt_misc (on Linux), which most distros do by default, and for even more performance they should have ape. If they don't have the latter, it automatically gets used from the output binary being run.

[sedwards2009]

yeah, I've been trying out Cosmopolitan. The whole APE and binfmt_misc isn't quite as out-of-the-box as I would like, but I don't need Windows support and I'm more interested in Linux and similar unix systems. For those it should be possible to get a "normal" ELF binary out of Cosmopolitan which side-steps the whole APE thing.