search

opnsense / tools

OPNsense release engineering toolkit
https://opnsense.org/
BSD 2-Clause "Simplified" License
271 stars 195 forks source link

22.1 ARM base build failed #258

Closed yrzr closed 2 years ago

yrzr commented 2 years ago

Hi, I am trying to build 22.1 ARM with latest tools(3ed03c72237f0e12e06cb921c558824944529661) and src(1fca8e5780b58bdf99ede32bd0c9aaa23ef93e59). Both cross build on AMD64 or native build on ARM lead to the same error here:

--------------------------------------------------------------
>>> stage 4.4: building everything
--------------------------------------------------------------
...
/usr/local/aarch64-unknown-freebsd13.0/bin/ld: boot1.sym: error: PHDR segment not covered by LOAD segment
cc: error: linker command failed with exit code 1 (use -v to see invocation)
--- everything ---

make[1]: stopped in /usr/src
--- buildworld ---

make: stopped in /usr/src
*** Error code 2

Stop.
make: stopped in /usr/tools

Any ideas?

fichtner commented 2 years ago

This might be an issue with llvm 13 update? Some things there have been reverted by FreeBSD already.

wilmardo commented 2 years ago

Isn't this the error that comes from this bug? https://github.com/opnsense/tools/issues/250

That is what I assumed, I am also having this error for at least the time that issue has been open.

fichtner commented 2 years ago

No. This comes from FreeBSD source tree.

yrzr commented 2 years ago

I have tried the FreeBSD source tree on branch stable/13, which results in the same problem.

fichtner commented 2 years ago

It might be LLVM 13 which dropped in 22.1.b3. Wasn’t very eager about it but since we still follow stable/13 I don’t see a quick way out of this. Best to raise a bug at https://bugs.FreeBSD.org

wilmardo commented 2 years ago

For reference, it seems that the bug has been opened: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260972

TomRomeo commented 2 years ago

Would it be possible to build 22.1.b2 on arm until this has been fixed upsteam? I'm sorry if this his been asked before or if I simply missed the secion in the readme, but how exaclty do you downgrade to a stable release? Is it sufficient to just checkout the specific tag in /usr/tools?

fichtner commented 2 years ago

Well, you just need to reset src.git:

# git -C /usr/src reset --hard 22.1.b2
yrzr commented 2 years ago

@fichtner As I tested, the solution @efetropy mentioned in https://github.com/opnsense/tools/issues/264 that removing CROSS_BINUTILS_PREFIX=/usr/local/aarch64-unknown-freebsd${SRCREVISION}/bin in device config works. Maybe we follow him?

turboproc commented 2 years ago

@yrzr , got a lot further after downgrading pkg and having the build process run for a number of days. Getting stuck now at g-lang that refuses to build which is required to get e.g. dnscrypt-proxy2. Noticed in the files that you've published that the dnscrypt-proxy package is present but that go-lang is not present which at leat to me is a kind of interesting ;-)

dnsefe commented 2 years ago

@turboproc I was also running into issues trying to cross-build go (required to build dnscrypt-proxy2) which after many hours failed. Once the build of the packages set is done, you can actually find go inside there. However, if you build an image certain packages (go among others) are just discarded as they are only build dependencies and not runtime dependencies which is why it's missing there. The package set gets updated accordingly.

@yrzr actually had a packages set prior to building an image on his ftp server (I still have a backup and can confirm that go is included). Looks like he recently updated the packages set on his server and the latter has no build dependencies.

turboproc commented 2 years ago

@efetropy , I understand that go-lang is not required in the packages, but to build dnscrypt-proxy2 I would still require go-lang to build succesfully. Do I miss something ?

dnsefe commented 2 years ago

@turboproc Indeed, it is only required to build dnscrypt-proxy2, but not to run it.

turboproc commented 2 years ago

@efetropy : so brings me to the point of how to successfully get go-lang build or onboard in another way ;-) Without I wouldn't know how to move forward.

dnsefe commented 2 years ago

@turboproc If you are keen on building all the required sets by yourself, we can ask @yrzr about how he cross-compiled go and follow his instructions.

What I did during the beta was simply run everything on my amd64 host and let the entire procedure finish. Once it's done, you will get a packages set with logs where you can see which packages failed during the build. I simply rebuilt those packages on my arm64 device (go, dnscrypt-proxy2 and some other packages). It's a headache though.

Theoretically speaking you can just skip building the sets since they are already available on his ftp server. With the base, kernel and packages sets you can already start building images for your arm device(s).

yrzr commented 2 years ago

@turboproc Hi, as mentioned by @efetropy, go-lang is a build dependency but not a runtime dependency. So it will be built but not included in the final package. The reason that go-lang is included in my first built packages is that the ignore_list wasn't working. Therefore go-lang was pulled in by other packages that are now ignored.

@turboproc Could you show your error log?

turboproc commented 2 years ago

@yrzr , @efetropy , below the output when go-lang is to be build and it fails. I understand that we do not need go-lang as an end result but still require it is an intermediate result to be able to build some other packages:

===> go-1.17.6,1 depends on file: /usr/local/sbin/pkg - found [20220212234111] => go1.17.6.src.tar.gz doesn't seem to exist in /usr/ports/distfiles/. [20220212234112] => Attempting to fetch https://golang.org/dl/go1.17.6.src.tar.gz go1.17.6.src.tar.gz 21 MB 2499 kBps 08s [20220212234121] => go-freebsd-arm64-go1.14.tar.xz doesn't seem to exist in /usr/ports/distfiles/. [20220212234122] => Attempting to fetch https://github.com/dmgk/go-bootstrap/releases/download/go1.14/go-freebsd-arm64-go1.14.tar.xz go-freebsd-arm64-go1.14.tar.xz 33 MB 2522 kBps 13s [20220212234136] ===> Fetching all distfiles required by go-1.17.6,1 for building [20220212234136] ===> Extracting for go-1.17.6,1 [20220212234138] => SHA256 Checksum OK for go1.17.6.src.tar.gz. [20220212234141] => SHA256 Checksum OK for go-freebsd-arm64-go1.14.tar.xz. [20220212234147] ===> Patching for go-1.17.6,1 [20220212234148] ===> Applying FreeBSD patches for go-1.17.6,1 from /usr/ports/lang/go/files [20220212234148] ===> Configuring for go-1.17.6,1 [20220212234148] ===> Building for go-1.17.6,1 cd /usr/obj/usr/ports/lang/go/work/go/src ; /usr/bin/env XDG_CACHE_HOME=/usr/obj/usr/ports/lang/go/work GOROOT_BOOTSTRAP=/usr/obj/usr/ports/lang/go/work/go-freebsd-arm64-bootstrap GOROOT=/usr/obj/usr/ports/lang/go/work/go GOROOT_FINAL=/usr/local/go GOBIN= GOOS=freebsd GOARCH=arm64 GO386= GOARM= CC=cc /bin/sh make.bash -v -ap: not found go: not found qemu: uncaught target signal 4 (Illegal instruction) - core dumped Illegal instruction Building Go cmd/dist using /usr/obj/usr/ports/lang/go/work/go-freebsd-arm64-bootstrap. () cmd/dist qemu: uncaught target signal 4 (Illegal instruction) - core dumped Illegal instruction *** Error code 132

Stop. make[1]: stopped in /usr/ports/lang/go *** Error code 1

Stop.

turboproc commented 2 years ago

Getting there

image

JPBeltman commented 2 years ago

@turboproc what platform are you building on? Asking it because I was stuck at that point as well (RPI4), seems I'm right behind you in my progress haha.

Also, @yrzr, I decided to try some of your images while my own are compiling;

Would you happen to have any pointers on this? I've got some ideas I'd like to before test opening an issue (should I do that on your fork?),but you may already have an answer:) U-boot seems to be the biggest troublemaker, the steps on your site (very nice btw) haven't yet worked for me

To, almost, finish my mostly non-related comment; It's almost been a month after I've read my first post of an ARM OPNsense port, just a week before 22.1-release. From what I've found; you've really made a lot of progress in the past few weeks, not just in terms of ARM-availability but making it accessible for (relative) newcomers. So, thanks @fichtner, @yrzr for your work (towards ARM)!

I'm relatively new to git(hub), what are the steps for me to take to suggest improvements for documentation?

dnsefe commented 2 years ago

@turboproc regarding "RPI4, pve vm; after converting & importing, the vm shows the Opnsense bootscreen but disconnects display afterwards" I assume this happened after the first reboot, right? Do you have hw.uart.console="" in loader.conf.local (under /boot)? If not creating the latter might fix the issue.

yrzr commented 2 years ago

@JPBeltman For RPI, Did you write config.txt in the boot partition (the first partition)? There are also two sample files config_rpi3.txt and config_rpi4.txt for your reference in the partition. Please also use the new images. The old ones were built for test days ago.

And for VM, the size of the image should be 21G as defined here https://github.com/opnsense/tools/blob/a69f574dd23746ce2dfac6ba787e94fd38e1ec38/build/vm.sh#L37-L38 So you may need to ensure that there is enough space.

@efetropy The problem you mentioned shall be already solved here https://github.com/opnsense/tools/blob/a69f574dd23746ce2dfac6ba787e94fd38e1ec38/config/22.1/extras.conf#L56-L57

turboproc commented 2 years ago

@JPBeltman , @yrzr , @efetropy regarding platform I was following different routes;

OpnSense runs on a NanoPI R4S 4G.

Still need to understand why some packages failed. Also ' make distfiles' fails because of audio/beep not existing on arm64.

JPBeltman commented 2 years ago

@turboproc, that's where mine stopped too. Fixed it by updating the ignorelist, but that doesn't seem to work on cross-building

JPBeltman commented 2 years ago

@yrzr spot on, I was tired and forgot to rename the files.. aarch64-RPI.img works on rpi4b 4gb model!

turboproc commented 2 years ago

@yrzr , @efetropy please check this audio/beep issue. When I checked the distfiles script I had the impression it's not using the IGNORE status of ports and hence it will try to fetch to download all ports. This would be an issue with all arm based builds (not aarch64 or NanoPI speciic). Correct ?

dnsefe commented 2 years ago

@turboproc The issue is already reported in #268 and indeed persists in all arm-based builds

@JPBeltman That line unfortunately didn't work work for me. It's also mentioned in the forums where a manual intervention is unavoidable unless you incorporate the fix into your build script :/

yrzr commented 2 years ago

I encountered the same problem as @turboproc in cross-building. I suspect there is some problem with qemu.

[20220215013704] ===> Building for go-1.17.6,1
cd /usr/obj/usr/ports/lang/go/work/go/src ; /usr/bin/env  XDG_CACHE_HOME=/usr/obj/usr/ports/lang/go/work  GOROOT_BOOTSTRAP=/usr/obj/usr/ports/lang/go/work/go-freebsd-arm64-bootstrap  GOROOT=/usr/obj/usr/ports/lang/go/work/go  GOROOT_FINAL=/usr/local/go  GOBIN=  GOOS=freebsd  GOARCH=arm64  GO386=  GOARM=  CC=cc  /bin/sh make.bash -v
-ap: not found
go: not found
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction
Building Go cmd/dist using /usr/obj/usr/ports/lang/go/work/go-freebsd-arm64-bootstrap. ()
cmd/dist
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction
*** Error code 132

Stop.
make[1]: stopped in /usr/ports/lang/go
*** Error code 1
turboproc commented 2 years ago

@yrzr , @efetropy kind of off-topic but what would be the best solution for a bare metal ARM platform to build this on. Build cycle of one week becomes a bit long. I see Pine rock64 PRO, there is a similar Rock PI 64. Anything more capable than RK3399 based boards ?

dnsefe commented 2 years ago

@yrzr I was facing the same problem while cross-building go under qemu (even with xtools during the beta).

@turboproc If money is not an issue, you can always go for ARM server platforms, e.g., HoneyComb LX2 is supported by FreeBSD.

All in all, I wonder why xtools is causing problems. If we manage to fix that issue, cross-building will be feasible again.

wilmardo commented 2 years ago

I fired up my crossbuild again to check and the PHDR segment not covered by LOAD segment gone on my machine too. But I got hit with an other error then mentioned above:

===> libexec/rpc.rwalld (includes)
===> stand/efi (includes)
make[4]: make[4]: don't know how to make /usr/src/contrib/llvm-project/libcxx/include/__functional_03. Stop

make[4]: stopped in /usr/src/lib/libc++
===> bin/expr (includes)
===> lib/libc++experimental (includes)
===> cddl/lib/libzfs_core (includes)
--- includes_subdir_lib/libc++ ---

make[3]: stopped in /usr/src/lib
===> usr.sbin/clear_locks (includes)
--- includes_subdir_share ---
dnsefe commented 2 years ago

@wilmardo That's news to me. Didn't encounter that before. Can you provide more information about your build host? Are you running FreeBSD 13 STABLE or RELEASE?

wilmardo commented 2 years ago

@efetropy Good one! 

# freebsd-version
13.0-RELEASE-p7

I will switch to STABLE since that is specified, I will report back :)

JPBeltman commented 2 years ago

So, to have it clear/confirmed, OPN22.1-release on aarch64 needs BSD13.0-stable instead of 13.0-release as mentioned in the docs (for AMD64)?

Is it an idea to move the 'Cross building for other architectures' section to tools/device?

A few arguments;

That being said, if there is any interest of doing this, I'm willing to scour through the issues in the coming weeks to get a sort of development overview of current devices (R4S, RockPi, NanoPi, Rock64, Pi4). Together with expanded building steps for native/cross-building ARM64 images. I'm learning something new almost every time I read an issue, while I can't contribute really much to development itself (other than saying 'it works'), I hope I can by making information more accessible.

dnsefe commented 2 years ago

@JPBeltman Based on my own experience, I ran into way fewer issues with a FreeBSD 13 STABLE host both for cross-building and native build on my aarch64 device. In my build process, I always make sure to have matching source trees.

Concerning the points you mentioned, here are my few bits to it:

If you put in effort to get a development overview of current devices and even write expanded building steps, I am willing to go through it for suggestions/changes. We can maybe put everything (current state, building steps) under tools/device and eventually link there from the main page. Would have to discuss it with @fichtner though.

fichtner commented 2 years ago

Just a quick note on "officially support aarch64" ... if this cross build situation teaches us anything is that a lot of work is needed to stay on top of it either from FreeBSD or from OPNsense and for the latter I'm not willing to spend the necessary time (as indicated by various build issues) on it as it will eat up resources for other parts of the project which are far more vital. I would have hoped the FreeBSD side would be less problematic by now but it is what it is. Best option is to sit this one out still...

JPBeltman commented 2 years ago

@fichtner I'm sorry. I think the arguments I've made are a bit unclear in hindsight, maybe even rude.

Just a quick note on "officially support aarch64"

I did not mean it in a perspective of OPN should support aarch, rather that it should only be included if or when it is supported by OPN (maybe 15 years). I do not want to put pressure on aarch development, nor ask you to put even more time and effort in than you already do (for which I'm very grateful). This repo should first and foremost focus on the core-business of OPN/Deciso; providing a production ready OS-level firewall and tested & stable solutions on building it.

That was the reason behind my first argument, the userbase for amd64 shouldn't have to scroll through non-native building info. But the development does exist and I think you've come far enough by now to start some documentation, for transparancy and others to build upon. E.g. there are 6 working devices now, each sharing some bugs with one of the other devices, same for bug-fixes. If we could provide some sort of status for devices to get an overview of it's state, packages, bugs, workarounds, relation between devices, package lists etc etc.

To finish, I hope my intentions are well received. My RPI4 wouldn't be running OPN right now without the work you all put in, I'm hoping to relieve others from days of searching and reading to get it to work too (again, in the end, 99% of what I did was following steps you provided via issues etc :) ).

Edit: added package status

JPBeltman commented 2 years ago
  • As you already mentioned, a lot of documentation regarding the ARM build process is either in the issues or in the forums. STABLE vs RELEASE, pkg v1.16.3 vs v.1.17.*, stuck at MASK on boot etc. You're right in that not everything mentioned there is transparent (such as downgrading the pkg version). At least for the current release, we can gather all available information in one place and do some clean summary.

This indeed. Downgrading pkg, for example, took me too long to figure out (but just 4 weeks into freeBSD). I think I wrote down my steps for it somewhere (altering multiples .confs etc) and wouldn't mind sharing it.

I really am hoping that OPNsense will officially support aarch64 with the upcoming (22.7) release.

I don't hope that tbh, I'd rather have it being a slow-paced community effort. imo Deciso & contributors should focus on the firewall itself to keep it a top-notch and relevant product. I also think ARM will mature a lot in the coming years, after which it may be ready for the OPN group to look into officially supporting some curated devices (like they already do with their own solutions, DEC610-630).

fichtner commented 2 years ago

closed via 7680ca161a3, thanks @yrzr