[macOS] ModuleNotFoundError: No module named 'encodings' when compiling with meson

ben-xD commented 1 day ago

I'm trying to build gstreamer locally with python 3.11 from uv. I've managed to build gstreamer locally with the homebrew python. I use python 3.11 because the gstreamer version im using has a dependency that relies on distutils, removed in python3.12.

I would like to use python by uv. I modified the steps by @samypr100 in https://github.com/astral-sh/uv/issues/8966#issuecomment-2466513707

Reproducible steps

# Some dependencies of packages
brew install pkg-config ninja

git clone --branch 1.24 https://gitlab.freedesktop.org/gstreamer/gstreamer.git
pushd gstreamer

uv python install 3.11
uv venv -p 3.11
uv tool install 'git+https://github.com/bluss/sysconfigpatcher'
sysconfigpatcher $HOME/.local/share/uv/python/cpython-3.11*
uv pip install meson docutils

# Export PYTHON so the gstreamer build will use it
export PYTHON="$HOME:$(pwd)/.venv/bin/python"

# Build and install gstreamer
PKG_CONFIG_LIBDIR=/dev/null uv run meson setup build -Dwrap_mode=forcefallback -Dintrospection=enabled -Dpython=enabled -Dgst-plugins-good:lame=disabled -Dglib:tests=false -Dtests=disabled --prefix=$HOME/gstreamer

## FAILS HERE
uv run meson compile -C build

# Install to $HOME/gstreamer
uv run meson install -C build

Error

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Would you have any advice on this error?

Full error

Click to expand

``` uv run meson compile -C build INFO: autodetecting backend as ninja INFO: calculating backend command to run: /opt/homebrew/bin/ninja -C /Users/user/repos/gstreamer/build ninja: Entering directory `/Users/user/repos/gstreamer/build' [2/3] Generating subprojects/gst-editing-services/ges/GES-1.0.gir with a custom command (wrapped by meson to set env) FAILED: subprojects/gst-editing-services/ges/GES-1.0.gir env PKG_CONFIG_PATH=/Users/user/repos/gstreamer/build/meson-uninstalled PKG_CONFIG=/opt/homebrew/bin/pkg-config CC=cc /Users/user/repos/gstreamer/build/subprojects/gobject-introspection-1.74.0/tools/g-ir-scanner --quiet --no-libtool --namespace=GES --nsversion=1.0 --warn-all --output subprojects/gst-editing-services/ges/GES-1.0.gir '--add-init-section=extern void gst_init(gint*,gchar**);extern void ges_init(void);g_setenv("GST_REGISTRY_1.0", "/no/way/this/exists.reg", TRUE);g_setenv("GST_PLUGIN_PATH_1_0", "", TRUE);g_setenv("GST_PLUGIN_SYSTEM_PATH_1_0", "", TRUE);g_setenv("GST_DEBUG", "0", TRUE);gst_init(NULL,NULL);ges_init();' --quiet --c-include=ges/ges.h --cflags-begin -I/Users/user/repos/gstreamer/subprojects/gst-editing-services/ges/.. -I/Users/user/repos/gstreamer/build/subprojects/gst-editing-services/ges/.. --cflags-end -I/Users/user/repos/gstreamer/subprojects/gst-editing-services/ges -I/Users/user/repos/gstreamer/build/subprojects/gst-editing-services/ges -I/Users/user/repos/gstreamer/subprojects/gst-editing-services/. -I/Users/user/repos/gstreamer/build/subprojects/gst-editing-services/. -I/Users/user/repos/gstreamer/subprojects/gstreamer/. -I/Users/user/repos/gstreamer/build/subprojects/gstreamer/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/. -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/glib -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/glib -I/Users/user/repos/gstreamer/subprojects/proxy-libintl/. -I/Users/user/repos/gstreamer/build/subprojects/proxy-libintl/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/gobject -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gobject -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/gmodule -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gmodule -I/Users/user/repos/gstreamer/subprojects/gstreamer/libs -I/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs -I/Users/user/repos/gstreamer/subprojects/gst-plugins-base/gst-libs -I/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs -I/Users/user/repos/gstreamer/subprojects/orc/. -I/Users/user/repos/gstreamer/build/subprojects/orc/. -I/Users/user/repos/gstreamer/subprojects/zlib-1.2.13/. -I/Users/user/repos/gstreamer/build/subprojects/zlib-1.2.13/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/gio -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gio -I/Users/user/repos/gstreamer/subprojects/libxml2-2.10.3/include -I/Users/user/repos/gstreamer/build/subprojects/libxml2-2.10.3/include -I/Users/user/repos/gstreamer/subprojects/gst-devtools/validate/. -I/Users/user/repos/gstreamer/build/subprojects/gst-devtools/validate/. -I/Users/user/repos/gstreamer/subprojects/json-glib-1.6.6/. -I/Users/user/repos/gstreamer/build/subprojects/json-glib-1.6.6/. --filelist=/Users/user/repos/gstreamer/build/subprojects/gst-editing-services/ges/libges-1.0.0.dylib.p/GES_1.0_gir_filelist --include=Gst-1.0 --include=GstPbutils-1.0 --include=GstVideo-1.0 --include=Gio-2.0 --include=GObject-2.0 --symbol-prefix=ges --identifier-prefix=GES --pkg-export=gst-editing-services-1.0 --cflags-begin -I/Users/user/repos/gstreamer/subprojects/gst-editing-services/. -I/Users/user/repos/gstreamer/build/subprojects/gst-editing-services/. -I/Users/user/repos/gstreamer/subprojects/gstreamer/. -I/Users/user/repos/gstreamer/build/subprojects/gstreamer/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/. -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/glib -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/glib -I/Users/user/repos/gstreamer/subprojects/proxy-libintl/. -I/Users/user/repos/gstreamer/build/subprojects/proxy-libintl/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/gobject -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gobject -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/gmodule -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gmodule -I/Users/user/repos/gstreamer/subprojects/gstreamer/libs -I/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs -I/Users/user/repos/gstreamer/subprojects/gst-plugins-base/gst-libs -I/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs -I/Users/user/repos/gstreamer/subprojects/orc/. -I/Users/user/repos/gstreamer/build/subprojects/orc/. -I/Users/user/repos/gstreamer/subprojects/zlib-1.2.13/. -I/Users/user/repos/gstreamer/build/subprojects/zlib-1.2.13/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/gio -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gio -I/Users/user/repos/gstreamer/subprojects/libxml2-2.10.3/include -I/Users/user/repos/gstreamer/build/subprojects/libxml2-2.10.3/include -I/Users/user/repos/gstreamer/subprojects/gst-devtools/validate/. -I/Users/user/repos/gstreamer/build/subprojects/gst-devtools/validate/. -I/Users/user/repos/gstreamer/subprojects/json-glib-1.6.6/. -I/Users/user/repos/gstreamer/build/subprojects/json-glib-1.6.6/. -I/Users/user/repos/gstreamer/subprojects/gstreamer/gst/parse -I/Users/user/repos/gstreamer/build/subprojects/gstreamer/gst/parse -I/Users/user/repos/gstreamer/subprojects/pcre2-10.42/. -I/Users/user/repos/gstreamer/build/subprojects/pcre2-10.42/. -I/Users/user/repos/gstreamer/subprojects/pcre2-10.42/src -I/Users/user/repos/gstreamer/build/subprojects/pcre2-10.42/src -I/Users/user/repos/gstreamer/subprojects/libffi/. -I/Users/user/repos/gstreamer/build/subprojects/libffi/. -I/Users/user/repos/gstreamer/subprojects/libffi/include -I/Users/user/repos/gstreamer/build/subprojects/libffi/include -I/Users/user/repos/gstreamer/subprojects/gst-plugins-base/. -I/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/. -I/Users/user/repos/gstreamer/subprojects/glib-2.78.3/subprojects/gvdb/. -I/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/subprojects/gvdb/. -I/opt/homebrew/include -DPCRE2_STATIC -I/Users/user/repos/gstreamer/subprojects/gobject-introspection-1.74.0/girepository/. -I/Users/user/repos/gstreamer/build/subprojects/gobject-introspection-1.74.0/girepository/. -I/opt/homebrew/opt/libffi/include --cflags-end --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gstreamer/gst --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/base --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/video --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/tag --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/audio --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/pbutils --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/controller --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/check --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/app --add-include-path=/Users/user/repos/gstreamer/build/subprojects/json-glib-1.6.6/json-glib --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gst-devtools/validate/gst/validate --add-include-path=/Users/user/repos/gstreamer/build/subprojects/gobject-introspection-1.74.0/gir -L/Users/user/repos/gstreamer/build/subprojects/gstreamer/gst -L/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/glib -L/Users/user/repos/gstreamer/build/subprojects/proxy-libintl -L/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gobject -L/Users/user/repos/gstreamer/build/subprojects/libffi/src -L/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gmodule --extra-library=gstreamer-1.0 --extra-library=glib-2.0 --extra-library=intl --extra-library=gobject-2.0 --extra-library=gmodule-2.0 -L/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/base --extra-library=gstbase-1.0 -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/video -L/Users/user/repos/gstreamer/build/subprojects/orc/orc --extra-library=gstvideo-1.0 --extra-library=orc-0.4 -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/pbutils -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/audio -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/tag -L/Users/user/repos/gstreamer/build/subprojects/zlib-1.2.13 --extra-library=gstpbutils-1.0 --extra-library=gstaudio-1.0 --extra-library=gsttag-1.0 --extra-library=z -L/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/controller --extra-library=gstcontroller-1.0 -L/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gio --extra-library=gio-2.0 -L/Users/user/repos/gstreamer/build/subprojects/libxml2-2.10.3 --extra-library=xml2 -L/Users/user/repos/gstreamer/build/subprojects/gst-devtools/validate/gst/validate -L/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/check -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/app -L/Users/user/repos/gstreamer/build/subprojects/json-glib-1.6.6/json-glib --extra-library=gstvalidate-1.0 --extra-library=gstcheck-1.0 --extra-library=gstapp-1.0 --extra-library=json-glib-1.0 -L/Users/user/repos/gstreamer/build/subprojects/gobject-introspection-1.74.0/girepository --extra-library=girepository-1.0 -L/Users/user/repos/gstreamer/build/subprojects/gst-editing-services/ges --library ges-1.0 -L/Users/user/repos/gstreamer/build/subprojects/gstreamer/gst -L/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/glib -L/Users/user/repos/gstreamer/build/subprojects/proxy-libintl -L/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gobject -L/Users/user/repos/gstreamer/build/subprojects/libffi/src -L/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gmodule -L/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/base -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/video -L/Users/user/repos/gstreamer/build/subprojects/orc/orc -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/pbutils -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/audio -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/tag -L/Users/user/repos/gstreamer/build/subprojects/zlib-1.2.13 -L/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/controller -L/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gio -L/Users/user/repos/gstreamer/build/subprojects/libxml2-2.10.3 -L/Users/user/repos/gstreamer/build/subprojects/gst-devtools/validate/gst/validate -L/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/check -L/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/app -L/Users/user/repos/gstreamer/build/subprojects/json-glib-1.6.6/json-glib -L/Users/user/.local/share/uv/python/cpython-3.11.10-macos-aarch64-none/lib -L/opt/homebrew/opt/libffi/lib --extra-library=m --extra-library=iconv --extra-library=readline -L/Users/user/.local/share/uv/python/cpython-3.11.10-macos-aarch64-none/lib --extra-library=python3.11 --extra-library=dl --extra-library=resolv --sources-top-dirs /Users/user/repos/gstreamer/subprojects/gst-editing-services --sources-top-dirs /Users/user/repos/gstreamer/build/subprojects/gst-editing-services ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gstreamer/gst' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/glib' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/proxy-libintl' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gobject' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/libffi/src' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gmodule' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/base' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/video' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/orc/orc' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/pbutils' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/audio' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/tag' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/zlib-1.2.13' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/controller' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/glib-2.78.3/gio' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/libxml2-2.10.3' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gst-devtools/validate/gst/validate' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gstreamer/libs/gst/check' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/gst-plugins-base/gst-libs/gst/app' ignored ld: warning: duplicate -rpath '/Users/user/repos/gstreamer/build/subprojects/json-glib-1.6.6/json-glib' ignored ld: warning: duplicate -rpath '/Users/user/.local/share/uv/python/cpython-3.11.10-macos-aarch64-none/lib' ignored ld: warning: ignoring duplicate libraries: '-lgio-2.0', '-lglib-2.0', '-lgmodule-2.0', '-lgobject-2.0', '-lintl' Could not find platform independent libraries Could not find platform dependent libraries Python path configuration: PYTHONHOME = (not set) PYTHONPATH = (not set) program name = 'python3' isolated = 0 environment = 1 user site = 1 safe_path = 0 import site = 1 is in build tree = 0 stdlib dir = '/install/lib/python3.11' sys._base_executable = '/Users/user/repos/gstreamer/build/tmp-introspectk7kz6uwj/GES-1.0' sys.base_prefix = '/install' sys.base_exec_prefix = '/install' sys.platlibdir = 'lib' sys.executable = '/Users/user/repos/gstreamer/build/tmp-introspectk7kz6uwj/GES-1.0' sys.prefix = '/install' sys.exec_prefix = '/install' sys.path = [ '/install/lib/python311.zip', '/install/lib/python3.11', '/install/lib/python3.11/lib-dynload', ] Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding Python runtime state: core initialized ModuleNotFoundError: No module named 'encodings' Current thread 0x00000001ff8f7840 (most recent call first): Command '['/Users/user/repos/gstreamer/build/tmp-introspectk7kz6uwj/GES-1.0', '--introspect-dump=/Users/user/repos/gstreamer/build/tmp-introspectk7kz6uwj/functions.txt,/Users/user/repos/gstreamer/build/tmp-introspectk7kz6uwj/dump.xml']' returned non-zero exit status 1. ninja: build stopped: subcommand failed. ```

samypr100 commented 1 day ago

I tried to repro, but hit a bump on meson setup with subprojects/gobject-introspection-1.74.0/meson.build:129:11: ERROR: Recursive include of subprojects: gstreamer => glib => gobject-introspection => glib.

My initial thoughts is that it's possible this is related to meson issues discussed as part of https://github.com/astral-sh/uv/pull/7857 as meson doesn't work with build isolation. From a python meson setup I also expected to see /Users/user/repos/gstreamer/.venv or '$HOME/.local/share/uv/python/cpython-3.11* instead of /install.

Do you have a docker based reproduceable steps?

ben-xD commented 22 hours ago

Apologies @samypr100, there are many things that can go wrong with the gstreamer stuff. Gstreamer currently has a broken main branch lol where there is a hard circular dependency... Please try on branch 1.24, that works for me. I've also updated my first messsage

git checkout 1.24

A docker setup wouldn't let me use brew, native macOS etc so I haven't tried to set that up.

About build isolation, setting -Dwrap_mode=forcefallback will not use any of the system libraries, and download everything. That allows the build to be a bit more isolated, but I do need to install some things (currently python including headers from brew, ninja, meson).

samypr100 commented 3 hours ago

Hmm, still not happy. Same error.

Re: build isolation, I meant Python PEP 517 build isolation, not meson's build isolation with respect to system libraries 😅

astral-sh / uv