search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
3.94k stars 3.45k forks source link

OpenWrt build status on macos #17669

Open svlobanov opened 2 years ago

svlobanov commented 2 years ago

Currently there are many packages with compilation issues on macos.

From Openwrt base (11 packages):

libs/elfutils,
libs/libaudit,libs/libaudit,libs/libsemanage,libs/libselinux,utils/secilc,utils/checkpolicy,libs/libsemanage,utils/policycoreutils,system/selinux-policy,system/refpolicy

(elfutils and a lot of selinux-related stuff)

From "packages" feed: 99 packages From "telephony" feed: 11 packages From "routing" feed: 2 packages

Most difficult to fix: glib2, packages related to selinux, boost (many packages depend on boost), python-cffi, packages that use host kernel headers for building host tools (e.g. xfsprogs), packages that use suid bit(FILE_MODES feature): apache, gnunet, mariadb (fixed by https://github.com/openwrt/openwrt/commit/53ebacacf93278c6e71052eb5b3708c599129e55 https://github.com/openwrt/openwrt/commit/1d4750fd50eeadad640fb62e1b11bf4cdde39dd1 https://github.com/openwrt/openwrt/commit/3869ccbcc891a7185550a2a422e2db01fd994b7d)

If someone knows how to fix packages from 'most difficult' list, any help is welcome.

packages with build issues on macos: err.log

CC: @aparcar @neheb @paper42 @ptpt52 @httpstorm

neheb commented 2 years ago

boost is surprising. glib2 is difficult, yes. I tried editing libiconv and gettext-full to be built as shared libraries as macOS demands but it didn't work.

champtar commented 2 years ago

We don't have MacOS CI and we can't right now has we don't have MacOS OpenWrt SDK. Also Mac are not always available so it's complicated to ask maintainers to carry patches they can't test ...

Maybe send an email to the mailing list to see what other Mac users are doing.

svlobanov commented 2 years ago

just a link to CI discussion: #17423 (to link this issue and CI issue)

Also Mac are not always available so it's complicated to ask maintainers to carry patches they can't test ...

yes, I understand it and most probably macos will never be officially supported even if I (or someone else) fix all the issues

I will continue fixing macos build issues. This issue might be a place to discuss general ideas (related to macos build) with another users

httpstorm commented 2 years ago

@svlobanov @neheb @champtar How did so many packages break all at the same time? Were they all updated or do they actually depend on another package which did? What I would usually do is start with two commits A and Z, where A works, Z doesn't. Then check the commit in the middle, etc. performing a binary search. Once I know which commit broke it, I would look into it. If a package was updated, I would switch that package to git. Then repeat the same binary search, since we have one commit which works and one which doesn't. Finally I would know what exact change broke it, and I would figure a way to solve it. If possible start with the first package that breaks, to avoid interference.

Edit: I just compiled af8a059bb41dd7afee49ce9a5d52e9a8e010bbd8 on macOS 11.6.2 (20G314) x64. Everything went smooth, so I would assume that the build fails on macOS 12 Monterey. Possible reasons:

Comparing the prepared and built directories for the affected projects might shine some light what is different and why. It helps if both sides use the same build path and commit, in my cause /Volumes/test/openwrt. I recommend using a sparse disk image labelled test, and we should use the same target and .config, x64 might be a good candidate. The reason I'm still on the now old Big Sur 11, is because many people on reddit complain about the poor reliability of Monterey 12. I might be able to install it as a second temporary OS, though this usually causes iCloud related issues.

What would it take to implement CI on macOS? How does it work, and how is it implemented/configured?

neheb commented 2 years ago

No they were always broken. Very few actually build on macOS.

champtar commented 2 years ago

What would it take to implement CI on macOS? How does it work, and how is it implemented/configured?

See #17423

httpstorm commented 2 years ago

Compile successfully on macOS 11.6.2

Failed host packages

Failed packages

Failed due to dependency on other packages which fail

svlobanov commented 2 years ago

@httpstorm which target do you compile? (x86_64 target?) do you compile all packages or only some packages? Have you tried to host-compile elfutils?

svlobanov commented 2 years ago

@httpstorm I can confirm that boost can be compiled on macos x86_64 for openwrt x86_64 target, I will also try for another targets, but my x86_64 mac is very slow and requires much time to compile anything

Also, @nbd168 added three new patches for macos: https://github.com/openwrt/openwrt/commit/53ebacacf93278c6e71052eb5b3708c599129e55 https://github.com/openwrt/openwrt/commit/1d4750fd50eeadad640fb62e1b11bf4cdde39dd1 https://github.com/openwrt/openwrt/commit/3869ccbcc891a7185550a2a422e2db01fd994b7d

chown-related issues should be gone (apache, gnunet, mariadb) after this

neheb commented 2 years ago

@svlobanov ccache can help a lot.

httpstorm commented 2 years ago

@httpstorm which target do you compile? (x86_64 target?) do you compile all packages or only some packages? Have you tried to host-compile elfutils?

Hi @svlobanov ! I see you, Rosen and Felix made some good progress. Yes, I compile the x86_64 target on MacBook Pro 2019 x64. I just upgraded to macOS 12.2.1. The list of packages I build for my servers is here: https://httpstorm.com/openwrt/x64/2022-02-13.r18804/ see config.buildinfo.

make package/libs/elfutils/host/compile V=sc
...
checking for __thread support... no
configure: error: __thread support required

Another issue I have during build is with staging_dir/toolchain-x86_64_gcc-11.2.0_musl/libexec/gcc/x86_64-openwrt-linux-musl/11.2.0/cc1, which gets corrupted when copied from /Volumes/x64/openwrt/build_dir/toolchain-x86_64_gcc-11.2.0_musl/gcc-11.2.0-initial/gcc/cc1. This was introduced in October 2021 when my version of coreutils got updated to version 9, because /usr/local/bin/ginstall supports sparse copy. Initially I thought that coreutils is to blame, since reverting to version 8 resolves the issue, however further investigation shows that the issue is either a bug in macOS 11 and 12, or a hardware issue with my laptop. In short, when toolchain/gcc/initial is compiled, cc1 and friends seem to be compiled sparse. While the utils under build_dir work, the copies installed under staging_dir contain huge blocks of unallocated zeroes and cannot be executed. Creating a regular copy works fine, only sparse copies are affected. Also if we eject the sparse disk image where OpenWRT is compiled, and then mount it again, sparse copies work. But if we recompile gcc, the issue occurs again.

Steps to reproduce: Create a new sparse disk image as described here httpstorm/OpenWRT build on macOS, and compile. toolchain/musl fails to build, because staging_dir/toolchain-x86_64_gcc-11.2.0_musl/libexec/gcc/x86_64-openwrt-linux-musl/11.2.0/cc1 is corrupted. Can you please try to reproduce this on your old x64 Mac, and then on M1? If you go past building the toolchain, it means everything works correctly. What version of macOS and coreutils (from brew) are you using?

Let me know if I can help with anything else!

svlobanov commented 2 years ago

@httpstorm I'm working on elfutils fix. It is not easy because a lot of GNU extensions are missing on macos. BTW, elfutils/host is required only for FRR. If I can't fix elfutils, maybe I will ask FRR developers to make an ability to use both methods for FRR build (new one - using elfutils and old one - using some python libs). They use elfutils (libelf) on compile time due to their old method was slow (https://github.com/FRRouting/frr/commit/5609b3af497c44d980aad2bc7d6332dedd7974e7 ). Also, it looks like, we can remove elfutils/host dependency if HOST_OS is Darwin because FRR has this check: AS_IF([test "$host" = "$build"]. Maybe I will try this way if can't fix elfutils build on macos

Regarding to cc1 sparse copy issue. I use big sur 11.6 on my old x86_64 mac (12.x is unsupported for this hw). Also I use big sur 11.6 on my new m1 mac, because I prefer not update host OS until current version becomes unsupported.

You can try to split your disk and format it to HFS+ case sensitive instead of using sparse image.

I use coreutils 9.0 on my both Apple macs

httpstorm commented 2 years ago

@svlobanov While building coreutils, I ran into a problem where the built-in cp and bison do not support new features, so I linked them ahead in the path. The main issue I face is when a script references /bin/sh or friends using an absolute path. Makes me wonder if it is possible to redirect this to the version installed by brew? I know /bin is read only. I'm rather thinking about mount --bind, alias or something similar. If I recall correctly, on Linux it is possible to mount a file on top of another, though that would take effect system wide, but might still help to complete a build and compare the difference to a failed one if we can do it on macOS.

Regarding to cc1 sparse copy issue. I use big sur 11.6 on my old x86_64 mac (12.x is unsupported for this hw). Also I use big sur 11.6 on my new m1 mac, because I prefer not update host OS until current version becomes unsupported. You can try to split your disk and format it to HFS+ case sensitive instead of using sparse image.

Then your system should be affected. Can you please verify using a sparse image with APFS? Sparse images with HFS+ work fine. APFS on physical volume also works.

If you ever need to build coreutils, here's a workaround to disable sparse copy: edit src/copy.c line 1045

- #ifdef SEEK_HOLE
+ #if defined(SEEK_HOLE) && !defined(__APPLE__)
  scan_inference->ext_start = lseek (fd, 0, SEEK_DATA);
svlobanov commented 2 years ago

Can you please verify using a sparse image with APFS?

I will try to check it a bit later.

If you ever need to build coreutils, here's a workaround to disable sparse copy: edit src/copy.c line 1045

I think, we need to implement it as a configurable feature (maybe as ./configure option) and send this to upstream.

BTW, OpenWrt recommends to use HFS+ sparse image so it is better to follow the recommendation. https://openwrt.org/docs/guide-developer/toolchain/buildroot.exigence.macosx

neheb commented 2 years ago

I have a potential fix for glib2: https://github.com/neheb/openwrt/commit/4127e3435203a7da3992928aa8e54070273177e2

@httpstorm can you test?

httpstorm commented 2 years ago

I have a potential fix for glib2: neheb/openwrt@4127e34

@httpstorm can you test? @neheb Hi Rosen! No difference, unfortunately: https://pastebin.com/Tgfkb7nL I applied your changes on top of a freshly cloned master. I could not checkout the commit directly, because it does not belong to a branch.

neheb commented 2 years ago

Unfortunate.

neheb commented 2 years ago 
cc  -o glib/gtester glib/gtester.p/gtester.c.o -Wl,-dead_strip_dylibs -Wl,-headerpad_max_install_names -Wl,-undefined,error -Wl,-framework,CoreFoundation -Wl,-framework,Carbon -Wl,-framework,Foundation -Wl,-framework,AppKit glib/libglib-2.0.a -lintl /Volumes/test/openwrt/staging_dir/hostpkg/lib/libpcre.dylib -lintl -liconv -lm
-Wl,rpath,$(STAGING_DIR_HOSTPKG)/lib

is needed here. It's specified in HOST_LDFLAGS. Wonder why it's missing...

Although currently libiconv-full/host is not built as a shared library to avoid any rpath hackery.

Then again, I also remember macOS not liking static libraries for some reason.

httpstorm commented 2 years ago

@neheb I did't understand where I should put this. I have a dozen of projects which relay on my static library and they build without any issues on macOS. Then last year I needed to add OpenSSL and it linked against the dynamic library, which caused problems because the brew version is not signed. The solution I found is to compile OpenSSL as a static library only. It turns out the linker tries to find a dynamic library first, and if that fails, it falls back to using the static library. The final tricky part was finding how to cross-compile OpenSSL for iPhone. Now my web server httpstorm runs happily on iPhones.

neheb commented 2 years ago

hah. while I have a jailbroken iPhone I never figured out a clever solution for it.

When I had a macOS VM I remember having issues with libiconv or gettext when built as a static library. Maybe I should get one of these new M1 machines and try again.

httpstorm commented 2 years ago

@neheb Hmm, what are you trying to do, perhaps I could help with a solution? Jaibreak is fun and freedom. I lost mine due to a mistake I made, which bricked the phone. I miss the ability to copy pictures over ssh. 4 years ago I had no programming experience with Linux. Eventually I got motivated to not depend on Windows, so now my apps work on pretty much anything: Linux, OpenWRT, FreeBSD, macOS, iPhone, Windows. Then I got a developer account, and now I can write and sign any app I need. If you need the macOS environment, I would say go for it. I think they released new devices this month. I wouldn't go below 32 GB RAM. With your experience, OpenWRT will get even better support. I personally find macOS comfortable, and after two years of fixes, OpenWRT and all the packages I need build correctly. The ability to mount virtual disks is very convenient when testing OpenWRT. Just keep in mind, Apple are evil. I was forced to spend 30 months on repairs, and they never miss the opportunity to erase your device if you send it for repair. I lost count on that, well at least the second laptop works after the final repair. 🤦🏼 I also spent two years trying to get a bounty for a remote exploit: visiting a web page can crash iOS and macOS. Their security team is very hard to work with, it took them over an year to start making any improvements. I can still exploit it with my network benchmark tool. No bounty and no CVE yet. If Linux can get your work done, and you only need macOS occasionally, use a VM. They don't deserve your money. One thing I learned to appreciate while contributing to OpenWRT is that people are friendly, and I see a good progress. 😎