search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4k stars 3.47k forks source link

meson issues with glib2? #17308

Open MaurUppi opened 2 years ago

MaurUppi commented 2 years ago

Name:meson Maintainer: @Andre Heider Openwrt branch : 21.02.1 host: Ubuntu 20.04

Question: may i know how to fix this issue? I tried "make clean" after error happened first time then "make -j1 V=sc" again. same issue, so that i tried :

make -j1 package/feeds/packages/glib2/compile V=sc

same error copied below:

make[2]: Entering directory '/home/ouzy/kiddin9/openwrt/feeds/packages/libs/glib2'
mkdir -p /home/ouzy/kiddin9/openwrt/dl
SHELL= flock /home/ouzy/kiddin9/openwrt/tmp/.glib-2.66.4.tar.xz.flock -c '      /home/ouzy/kiddin9/openwrt/scripts/download.pl "/home/ouzy/kiddin9/openwrt/dl" "glib-2.66.4.tar.xz" "97df8670e32f9fd4f7392b0980e661dd625012015d58350da1e58e343f4af984" "" "@GNOME/glib/2.66"    '
rm -f /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/.configured_*
rm -f /home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/stamp/.glib2_installed
/home/ouzy/kiddin9/openwrt/staging_dir/host/bin/sed -e "s|@CC@|'ccache', 'gcc',|" -e "s|@CXX@|'ccache', 'g++',|" -e "s|@PKGCONFIG@|/home/ouzy/kiddin9/openwrt/staging_dir/host/bin/pkg-config|" -e "s|@CFLAGS@|'-O2', '-I/home/ouzy/kiddin9/openwrt/staging_dir/host/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/host/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/host/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/host/include',|" -e "s|@CXXFLAGS@|'-I/home/ouzy/kiddin9/openwrt/staging_dir/host/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/host/include',|" -e "s|@LDFLAGS@|'-L/home/ouzy/kiddin9/openwrt/staging_dir/host/lib', '-L/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib', '-L/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/host/lib', '-liconv', '-Wl,-rpath,/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib',|" -e "s|@PREFIX@|/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg|" < /home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/openwrt-native.txt.in > /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-native.txt
/home/ouzy/kiddin9/openwrt/staging_dir/host/bin/sed -e "s|@CC@|'ccache_cc',|" -e "s|@CXX@|'ccache_cxx',|" -e "s|@AR@|x86_64-openwrt-linux-musl-gcc-ar|" -e "s|@STRIP@|x86_64-openwrt-linux-musl-strip|" -e "s|@NM@|x86_64-openwrt-linux-musl-gcc-nm|" -e "s|@PKGCONFIG@|/home/ouzy/kiddin9/openwrt/staging_dir/host/bin/pkg-config|" -e "s|@CFLAGS@|'-O2', '-pipe', '-fno-caller-saves', '-fno-plt', '-fhonour-copts', '-Wno-error=unused-but-set-variable', '-Wno-error=unused-result', '-fmacro-prefix-map=/home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4=glib-2.66.4', '-Wformat', '-Werror=format-security', '-fstack-protector', '-D_FORTIFY_SOURCE=1', '-Wl,-z,now', '-Wl,-z,relro', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libiconv-stub/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libintl-stub/include', '-ffunction-sections', '-fdata-sections', '-I/home/ouzy/kiddin9/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/usr/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/include/fortify', '-I/home/ouzy/kiddin9/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libiconv-stub/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libintl-stub/include',|" -e "s|@CXXFLAGS@|'-O2', '-pipe', '-fno-caller-saves', '-fno-plt', '-fhonour-copts', '-Wno-error=unused-but-set-variable', '-Wno-error=unused-result', '-fmacro-prefix-map=/home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4=glib-2.66.4', '-Wformat', '-Werror=format-security', '-fstack-protector', '-D_FORTIFY_SOURCE=1', '-Wl,-z,now', '-Wl,-z,relro', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libiconv-stub/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libintl-stub/include', '-ffunction-sections', '-fdata-sections', '-I/home/ouzy/kiddin9/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/usr/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/include/fortify', '-I/home/ouzy/kiddin9/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libiconv-stub/include', '-I/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libintl-stub/include',|" -e "s|@LDFLAGS@|'-L/home/ouzy/kiddin9/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/usr/lib', '-L/home/ouzy/kiddin9/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib', '-znow', '-zrelro', '-L/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libiconv-stub/lib', '-Wl,-rpath-link=/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libiconv-stub/lib', '-L/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libintl-stub/lib', '-Wl,-rpath-link=/home/ouzy/kiddin9/openwrt/staging_dir/target-x86_64_musl/usr/lib/libintl-stub/lib', '-Wl,--gc-sections',|" -e "s|@ARCH@|"x86_64"|" -e "s|@CPU@|"64"|" -e "s|@ENDIAN@|little|" < /home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/openwrt-cross.txt.in > /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-cross.txt
/home/ouzy/kiddin9/openwrt/staging_dir/host/bin/python /home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/meson.py  --buildtype plain --native-file /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-native.txt --cross-file /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-cross.txt -Ddefault_library=both -Diconv=external -Dselinux=disabled -Dlibmount=disabled -Dinternal_pcre=true -Dman=false -Ddtrace=false -Dsystemtap=false -Dgtk_doc=false -Dbsymbolic_functions=true -Dforce_posix_threads=true -Dfam=false -Dinstalled_tests=false -Dnls=disabled -Doss_fuzz=disabled -Dglib_assert=false -Dglib_checks=false -Dxattr=true -Db_lto=true /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-build /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-build/..
Traceback (most recent call last):
  File "/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/meson.py", line 18, in <module>
    from mesonbuild._pathlib import Path
  File "/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/mesonbuild/_pathlib.py", line 38
    def resolve(self, strict: bool = False) -> 'Path':
                            ^
SyntaxError: invalid syntax
make[2]: *** [Makefile:133: /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/.configured_efb2d3cf01c707184fb77fd0fe149f94] Error 1
make[2]: Leaving directory '/home/ouzy/kiddin9/openwrt/feeds/packages/libs/glib2'
time: package/feeds/packages/glib2/compile#0.54#0.25#0.67
    ERROR: package/feeds/packages/glib2 failed to build.
make[1]: *** [package/Makefile:114: package/feeds/packages/glib2/compile] Error 1
make[1]: Leaving directory '/home/ouzy/kiddin9/openwrt'
make: *** [/home/ouzy/kiddin9/openwrt/include/toplevel.mk:230: package/feeds/packages/glib2/compile] Error 2

nano /home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/meson.py

#!/usr/bin/env python3

# Copyright 2016 The Meson development team

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import sys
from mesonbuild._pathlib import Path

# If we're run uninstalled, add the script directory to sys.path to ensure that
# we always import the correct mesonbuild modules even if PYTHONPATH is mangled
meson_exe = Path(sys.argv[0]).resolve()
if (meson_exe.parent / 'mesonbuild').is_dir():
    sys.path.insert(0, str(meson_exe.parent))

from mesonbuild import mesonmain

if __name__ == '__main__':
    sys.exit(mesonmain.main())

nano ./build_dir/hostpkg/pypi/meson-0.56.0/mesonbuild/_pathlib.py

# Copyright 2020 The Meson development team

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import sys
import typing as T

# Python 3.5 does not have the strict kwarg for resolve and always
# behaves like calling resolve with strict=True in Python 3.6+
#
# This module emulates the behavior of Python 3.6+ by in Python 3.5 by
# overriding the resolve method with a bit of custom logic
#
# TODO: Drop this module as soon as Python 3.5 support is dropped

if T.TYPE_CHECKING:
    from pathlib import Path
else:
    if sys.version_info.major <= 3 and sys.version_info.minor <= 5:

        # Inspired by https://codereview.stackexchange.com/questions/162426/subclassing-pathlib-path
        import pathlib
        import os

        # Can not directly inherit from pathlib.Path because the __new__
        # operator of pathlib.Path() returns a {Posix,Windows}Path object.
        class Path(type(pathlib.Path())):
            def resolve(self, strict: bool = False) -> 'Path':
                try:
                    return super().resolve()
                except FileNotFoundError:
                    if strict:
                        raise
                    return Path(os.path.normpath(str(self)))

    else:
        from pathlib import Path

from pathlib import PurePath, PureWindowsPath, PurePosixPath
paper42 commented 2 years ago 
  File "/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/mesonbuild/_pathlib.py", line 38
    def resolve(self, strict: bool = False) -> 'Path':
                            ^
SyntaxError: invalid syntax

This sounds like it's trying to use Python 2, because this is valid in Python 3.

MaurUppi commented 2 years ago 
  File "/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/mesonbuild/_pathlib.py", line 38
    def resolve(self, strict: bool = False) -> 'Path':
                            ^
SyntaxError: invalid syntax

This sounds like it's trying to use Python 2, because this is valid in Python 3.

So this is a bug?
how can it fix or avoid this error so that compile can move on?

BKPepe commented 2 years ago

@OUZYcn Thanks for reporting. Let's take a look.

MaurUppi commented 2 years ago

@OUZYcn Thanks for reporting. Let's take a look.

  • Could you check which Python3 version do you have in lang/python/python3-version.mk?
ouzy@ouzy:~/kiddin9$ find | grep python3-version.mk
./openwrt/feeds/packages/lang/python/python3-version.mk

So, we can rule out my pull request, which was mentioned. Also, it would be good which commit hashes from feeds are you trying to compile.

ref: https://github.com/kiddin9/OpenWrt_x86-r2s-r4s/blob/master/onekey/build.sh Before git repo, the build.sh had below apt get to prepare env. Could be this python3 python2.7 without specific package caused issue?

sudo apt-get -y install build-essential asciidoc binutils bzip2 gawk gettext git libncurses5-dev patch python3 python2.7 unzip zlib1g-dev lib32gcc-s1 libc6-dev-i386 subversion flex uglifyjs gcc-multilib g++-multilib p7zip p7zip-full msmtp libssl-dev texinfo libglib2.0-dev xmlto qemu-utils libelf-dev autoconf automake libtool autopoint device-tree-compiler ccache xsltproc rename antlr3 gperf curl screen upx-ucl jq

REPO_BRANCH="$(curl -s https://api.github.com/repos/openwrt/openwrt/tags | jq -r '.[].name' | grep v21 | head -n 1 | sed -e 's/v//')"
git clone -b v$REPO_BRANCH --depth 1 https://github.com/openwrt/openwrt
  • On your host system, could you please check that you don't have installed python3-minimal? If you do, be sure that you have installed python3-distutils. Also, you can try it with full-featured Python3 version (if it works in your case).
ouzy@ouzy:~/kiddin9$ apt list | grep python3-minimal

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

python3-minimal/groovy,now 3.8.6-0ubuntu1 amd64 [installed,automatic]

ouzy@ouzy:~/kiddin9$ apt list | grep python3-distutils

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

python3-distutils-extra/groovy 2.45 all
python3-distutils/groovy-updates,groovy-security,now 3.8.10-0ubuntu1~20.10 all [installed]

Therefore, should I " sudo apt remove python3-minimal" then openwrt compile should continue to works?

BKPepe commented 2 years ago 
ouzy@ouzy:~/kiddin9$ find | grep python3-version.mk
./openwrt/feeds/packages/lang/python/python3-version.mk

That's not helpful, though. I was asking which version is in that file. I'm looking for value in PYTHON3_VERSION_MICRO.

Unfortunately, I am not able to provide you any support for that script, which you linked.

  1. It says that you need to install python2.7 for your host system, which is wrong. Since OpenWrt 21.02, it is not required.

  2. It downloads firmware from tencent.com, which is not an official mirror according to https://openwrt.org/downloads

  3. It is questionable why you should use snapshots for 21.02 for the router in the production environment. Because 21.02 is a stable release and even released with fixup release - 21.02.1

  4. There are many patches, which I don't know what they are doing. Those patches are not included in our repositories, there are missing any details what these patches do except their name.

I am using Ubuntu 21.10 for development purposes and I am not able to reproduce your issue.

MaurUppi commented 2 years ago 
ouzy@ouzy:~/kiddin9$ find | grep python3-version.mk
./openwrt/feeds/packages/lang/python/python3-version.mk

That's not helpful, though. I was asking which version is in that file. I'm looking for value in PYTHON3_VERSION_MICRO.

Sorry , here you go. 

#
# Copyright (C) 2007-2016 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#

# Note: keep in sync with setuptools & pip
PYTHON3_VERSION_MAJOR:=3
PYTHON3_VERSION_MINOR:=9
PYTHON3_VERSION_MICRO:=7

PYTHON3_VERSION:=$(PYTHON3_VERSION_MAJOR).$(PYTHON3_VERSION_MINOR)

PYTHON3_SETUPTOOLS_PKG_RELEASE:=1
PYTHON3_PIP_PKG_RELEASE:=1

PYTHON3_SETUPTOOLS_VERSION:=57.4.0
PYTHON3_PIP_VERSION:=21.2.3

Unfortunately, I am not able to provide you any support for that script, which you linked.

  1. It says that you need to install python2.7 for your host system, which is wrong. Since OpenWrt 21.02, it is not required.

OK, I'm not using python2.7 for dev works on host. so I will remove it.

  1. It downloads firmware from tencent.com, which is not an official mirror according to https://openwrt.org/downloads

Well, about official mirror or not, maybe off topic but what I can tell you is Tencent , Ali, Huawei and some similar popular could services providers offering many linux dist mirrors, some .edu also. For Tencent cloud mirror, i did checked some files, sha256 checksum is same as openwrt.org repo.

  1. It is questionable why you should use snapshots for 21.02 for the router in the production environment. Because 21.02 is a stable release and even released with fixup release - 21.02.1

I think you're referring to below line 91, the point is when say this moment openwrt 21.02.1 released but SDK very likely not at same time or even same day. To avoid some network connectivity issue that someone who may had, the script will try to get it from Tencent cloud mirror to get repo sdk then try to get latest SDK snapshot of late official release. 

curl -fL -o sdk.tar.xz https://mirrors.cloud.tencent.com/openwrt/releases/$REPO_BRANCH/targets/x86/64/openwrt-sdk-$REPO_BRANCH-x86-64_gcc-8.4.0_musl.Linux-x86_64.tar.xz || curl -fL -o sdk.tar.xz https://downloads.openwrt.org/releases/21.02-SNAPSHOT/targets/x86/64/openwrt-sdk-21.02-SNAPSHOT-x86-64_gcc-8.4.0_musl.Linux-x86_64.tar.xz
  1. There are many patches, which I don't know what they are doing. Those patches are not included in our repositories, there are missing any details what these patches do except their name.

Somehow that patchs , my understanding, its enhancement mod such like fullconenat . I'm not sure which already upstream but some certainly isn't.

I am using Ubuntu 21.10 for development purposes and I am not able to reproduce your issue.

MaurUppi commented 2 years ago

Hi @BKPepe May i have you update about this issue?

jefferyto commented 2 years ago

meson in 21.02 uses your build host (system) Python, not host Python (compiled from the python3 package).

What do you see if you run at the terminal:

python3 --version

Also ping @dhewg

MaurUppi commented 2 years ago

meson in 21.02 uses your build host (system) Python, not host Python (compiled from the python3 package).

What do you see if you run at the terminal:

python3 --version

Also ping @dhewg

@jefferyto

Sorry for reply late. 

ouzy@ubuntuesxi:~/kiddin9$ python --version
Python 2.7.18
ouzy@ubuntuesxi:~/kiddin9$ python
python            python2.7         python3           python3-unidiff   python3.8-config  
python2           python2.7-config  python3-config    python3.8         pythontex
ouzy@ubuntuesxi:~/kiddin9$ whereis python
python: /usr/bin/python2.7-config /usr/bin/python /usr/bin/python3.8-config /usr/bin/python3.8 /usr/bin/python2.7 /usr/lib/python3.8 /usr/lib/python3.9 /usr/lib/python2.7 /etc/python3.8 /etc/python2.7 /usr/local/lib/python3.8 /usr/local/lib/python2.7 /usr/include/python3.8 /usr/include/python2.7 /usr/share/python
ouzy@ubuntuesxi:~/kiddin9$ ll /usr/bin/python
lrwxrwxrwx 1 root root 7 Apr 15  2020 /usr/bin/python -> python2*

After sudo apt remove python and sudo apt autoremove, here is python info


ouzy@ubuntuesxi:~/kiddin9$ python
python3 python3-config python3-unidiff   python3.8 python3.8-config  pythontex
ouzy@ubuntuesxi:~/kiddin9$ python
bash: /usr/bin/python: No such file or directory
ouzy@ubuntuesxi:~/kiddin9$ python3 --version
Python 3.8.10
eli-schwartz commented 2 years ago 
/home/ouzy/kiddin9/openwrt/staging_dir/host/bin/python /home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/meson.py  --buildtype plain --native-file /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-native.txt --cross-file /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-cross.txt -Ddefault_library=both -Diconv=external -Dselinux=disabled -Dlibmount=disabled -Dinternal_pcre=true -Dman=false -Ddtrace=false -Dsystemtap=false -Dgtk_doc=false -Dbsymbolic_functions=true -Dforce_posix_threads=true -Dfam=false -Dinstalled_tests=false -Dnls=disabled -Doss_fuzz=disabled -Dglib_assert=false -Dglib_checks=false -Dxattr=true -Db_lto=true /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-build /home/ouzy/kiddin9/openwrt/build_dir/target-x86_64_musl/glib-2.66.4/openwrt-build/..
Traceback (most recent call last):
  File "/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/meson.py", line 18, in <module>
    from mesonbuild._pathlib import Path
  File "/home/ouzy/kiddin9/openwrt/staging_dir/hostpkg/lib/meson/mesonbuild/_pathlib.py", line 38
    def resolve(self, strict: bool = False) -> 'Path':
                            ^
SyntaxError: invalid syntax

You are invoking meson using the (unversioned) python interpreter at /home/ouzy/kiddin9/openwrt/staging_dir/host/bin/python which apparently does not like the python3 syntax.

jefferyto commented 2 years ago

After sudo apt remove python and sudo apt autoremove, here is python info

You didn't actually need to remove Python 2. Both $(STAGING_DIR_HOST)/bin/python and $(STAGING_DIR_HOST)/bin/python3 are mapped to Python 3 (python3) not Python 2.

In any case, I suggest you start again with a clean checkout of buildroot / sdk and try compiling again.

neheb commented 2 years ago

is this still an issue?