[Cross build ARM] 20.1 config ready to be use?

[nekoprog]

Just wondering if it's okay to build using 20.1 config, because I encounter some error when running make base SETTINGS=20.1 with FreeBSD 12.0. Looks like the base.txz not found, but the script was looking at wrong directory where it supposed to look at STAGEDIR/usr/src, but it went to host /usr/src.

Will update the error with log later, I just close my VM.

[fichtner]

Or simply delete net/miniupnpd/files/patch-portinuse.c and commit ;)

[fichtner]

Due to popular demand I applied a fix: https://github.com/opnsense/ports/commit/eefcea86384

(the patch in the upstream FreeBSD ticket didn't work for me)

[rene-bayer]

Perfect, thanks :-)

[nekoprog]

Thanks a lot.

[dgktkr]

I've run into another snag when building kernel modules for the kernel target:

... /usr/local/arm-gnueabi-freebsd12.0/bin/ld -m armelf_fbsd -d -warn-common --build-id=sha1 -r -d -o cloudabi32.kld cloudabi32_vdso_blob.o cloudabi32_fd.o cloudabi32_module.o cloudabi> /usr/local/arm-gnueabi-freebsd12.0/bin/ld: error: source object cloudabi32_fd.o has EABI version 5, but target cloudabi32.kld has EABI version 0 /usr/local/arm-gnueabi-freebsd12.0/bin/ld: failed to merge target specific data of file cloudabi32_fd.o ...

It seems curious that quite a few .kld files are built before the cloudabi32.kld build fails.

I wondered whether the problem also occurs when building from stock FreeBSD source or whether OPNsense modifications are the cause. So I did an experiment: A build of the kernel for ClearFog using Github FreeBSD source release/12.1.0 and KERNCONF=ARMADA38X proceeds and completes without error. So it does appear that something in OPNsense's src.git is the culprit.

I also wonder why no-one else seems to have encountered this problem.

Does anyone have ideas or insights that might be helpful here?

[rene-bayer]

I had the same error.

Are you building on 12.1? Which device config are you using for building opnsense?

Edit: if you just want opnsense on your sg3100 ... i think i am able to release a first "ready to use" alpha in a few days.

[dgktkr]

I had the same error.

Are you building on 12.1?

The host system is 12.1-RC2. Same results with the host being release/12.1.0.

About a week ago Franco wrote that src.git was releng/12.1. The src.git I have is 3 days old. So the target system should be 12.1 also.

Which device config are you using for building opnsense?

CLEARFOG.conf, which is a modified BANANAPI.conf. I'm also using tools from OPNsense (not nekoprog).

Were you able to solve this problem for your sg3100?

[rene-bayer]

Were you able to solve this problem for your sg3100?

yes.

Try my device config: https://github.com/DarkSunOne/tools/blob/master/device/ARMADA38X.conf i extracted the /usr/usg3100/boot folder from the running pfsense image on the sg3100.

You also need to get the Netgate dts file for your kernel, you can grab it here

You also need a slightly changed kernel config, try it with that one

My SG3100 already runs with opnsense, but theres still a lot of work :-(

[dgktkr]

Were you able to solve this problem for your sg3100?

yes.

Try my device config: https://github.com/DarkSunOne/tools/blob/master/device/ARMADA38X.conf i extracted the /usr/usg3100/boot folder from the running pfsense image on the sg3100.

You also need to get the Netgate dts file for your kernel, you can grab it here

You also need a slightly changed kernel config, try it with that one

My SG3100 already runs with opnsense, but theres still a lot of work :-(

Hi Rene_,

Thanks for the suggestions. Changing the toolchain did indeed solve the EABI version mismatch problem. Now the build dies with undefined symbols (xz_dec_init, xz_dec_run, xz_dec_end) while linking the kernel. I'm sure it will take me a while to resolve that.

[rene-bayer]

Hi, Maybe the kernel config? As said i copied the 'default' ARMADA38X config to SMP-ARMADA38X, and changed the last lines (regarding the fdt settings) You also NEED the right dtb file! Else the kernel wont boot.

Let me know if you need something :)

[dgktkr]

Hi, Maybe the kernel config? As said i copied the 'default' ARMADA38X config to SMP-ARMADA38X, and changed the last lines (regarding the fdt settings) You also NEED the right dtb file! Else the kernel wont boot.

Let me know if you need something :)

Will do! I think I have the boot stuff sorted. Once I install the complete OPNsense image on the device SSD, I'll copy over proper dtb and ubldr files to their proper places and manually dd known good boot bits to the start of the drive.

Edit: kernel config it was! Putting device xz and options GEOM_UZIP in /usr/src/sys/arm/conf/SMP-CLEARFOG allowed the kernel to build!

[dgktkr]

With the suggestions given in this thread, I've gotten base, kernel, tools, packages and arm to build to completion, but there are two things of note that I've encountered.

1) This may be a cosmetic problem, but the packages set is called

packages-19.7.6_102-OpenSSL-armv7.tar

even though SETTINGS?= 20.1 is in /usr/tools/Makfile. The other sets have the expected names, for instance,

kernel-20.1.a_93-armv7-CLEARFOG.txz

2) I can boot the image built with make arm, but the kernel refuses to run /bin/sh, complaining: can't exec /bin/sh for /etc/rc: Exec format error. So, first I checked the boot/kernel/kernel in the image:

file ../boot/kernel/kernel ../boot/kernel/kernel: ELF 32-bit LSB executable, ARM, EABI5 version 1 (FreeBSD), dynamically >linked, interpreter /red/herring, BuildID[sha1]=fe91ba8f364ffe32a24cf99754aa14d85cfb3f94, >for FreeBSD 12.1, not stripped

Then I checked /bin/sh in the image:

file sh sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 12.1, >FreeBSD-style, stripped

How'd that happen? A sampling of executables in the image /bin all are for 64 bit x86-64. A sampling of libraries in the image /lib are all 32 bit ARM EABI5. A sampling of executables in the image /sbin are also all 32 bit ARM EABI5. A sampling of executables in the image /usr/bin: some are 32 bit ARM EABI5 (crypt, top, tip) and some are x86-64 (objcopy, objdump)

So why is some code being compiled as x86-64 and some as 32 bit ARM?

Edit: 3) To build arm successfully, I found you can go into the main Makefile and delete the # in line 67 or so:

SUFFIX?= #-devel

I'm very happy I discovered this as it takes about 16 hours to build packages on my system.

[rene-bayer]

Theres something wrong with your build environment, or how you use it.

Package name is as you said just cosmetic, they work on all my builds

First of all i would suggest you to stop build with DEVICE=CLEARFOG ... as i remember you want to build for the netgate sg3100, this board is Marvell Armada385 based, and the configs for this board, especially the kernel defconfog is already there (/usr/src/sys/arm/conf/ARMADA38X)

Next one is, dont change the Makefile, better use SETTINGS=20.1 in your commands. You even need to 'make update SETTINGS=20.1' to get the latest (FreeBSD12.1 based) source.

Regarding step 3 ... i didnt need that, arm builds very well even with that line, what is the error message with that line included?

[dgktkr]

Theres something wrong with your build environment, or how you use it.

Package name is as you said just cosmetic, they work on all my builds

First of all i would suggest you to stop build with DEVICE=CLEARFOG ... as i remember you want to build for the netgate sg3100, this board is Marvell Armada385 based, and the configs for this board, especially the kernel defconfog is already there (/usr/src/sys/arm/conf/ARMADA38X)

The board I'm trying to build for is the SolidRun ClearFog Base https://developer.solid-run.com/products/clearfog-base/ I have custom KERNCONF and dtb files that work well with a plain vanilla build of FreeBSD 12.1 on this board.

Next one is, dont change the Makefile, better use SETTINGS=20.1 in your commands.

This is worth looking into.

You even need to 'make update SETTINGS=20.1' to get the latest (FreeBSD12.1 based) source.

Regarding step 3 ... i didnt need that, arm builds very well even with that line, what is the error message with that line included?

The same one you encountered in this thread about 2 weeks ago. About halfway between here and the original post: When executing make arm, the build errors out with

Could not find package: os-dyndns

[rene-bayer]

The board I'm trying to build for is the SolidRun ClearFog Base https://developer.solid-run.com/products/clearfog-base/ I have custom KERNCONF and dtb files that work well with a plain vanilla build of FreeBSD 12.1 on this board.

Oh sorry, my fault.

The same one you encountered in this thread about 2 weeks ago. About halfway between here and the original post: When executing make arm, the build errors out with

Could not find package: os-dyndns

I found a better fix, you just need to remove "[0-9]" here and here will create a pr later

[dgktkr]

Next one is, dont change the Makefile, better use SETTINGS=20.1 in your commands.

Hi Rene_,

Thank you ever so much. Again, your suggestion solved my problem! The web GUI is currently running on my ClearFog Base.

Take care

[dgktkr]

OK. I have the webGUI running on my board, but the console goes unresponsive after showing:

Invoking start script 'beep'

Root file system: /dev/ufs/OPNsense Wed Nov 13 18:09:26 UTC 2019

OPNsense.localdomain: OPNsense 20.1.a_241 (armv7/OpenSSL)

LAN (mvneta0) -> v4: 192.168.1.1/24 WAN (mvneta1) ->

HTTPS: SHA256 EF 39 4D 05 D1 51 CF A2 7C AE 23 5C 12 60 CD C8 DD 0B 38 AC D6 E0 5D 60 74 8C E3 7E BE A9 93 7E

Is the console supposed to do that or is this a dtb problem?

[rene-bayer]

Nice work :+1:

Probably the exact problem i'm facing currently, already explained here

My workaround is to change this line to /usr/local/sbin/opnsense-shell

You can also enable ssh over the webgui, enable ssh, and change the line directly under /usr/local/etc/rc

[dgktkr]

Hi guys,

OPNsense has been running on my arm device for a couple of days now and so far most features are functioning as expected. Thank you nekoprog and Rene_ for your help.

One thing that isn't working as expected is logging for pf, so I'd like to ask a favor of nekoprog and/or Rene_: could you check and see if pf logging is working for you, or are Live View, Overview and Plain View empty with clog /var/log/filter.log showing nothing. I'm wondering if I'm the only one with this issue.

[rene-bayer]

Hi,

Perfect :smiley:

i will look at this the next days :+1:

[nekoprog]

Hi guys,

OPNsense has been running on my arm device for a couple of days now and so far most features are functioning as expected. Thank you nekoprog and Rene_ for your help.

One thing that isn't working as expected is logging for pf, so I'd like to ask a favor of nekoprog and/or Rene_: could you check and see if pf logging is working for you, or are Live View, Overview and Plain View empty with clog /var/log/filter.log showing nothing. I'm wondering if I'm the only one with this issue.

Hi, can you open new issue regarding this pf log not functioning at opnsense/core? It will help others who look for solution with the same issue next time. :)

[dgktkr]

Hi, can you open new issue regarding this pf log not functioning at opnsense/core? It will help others who look for solution with the same issue next time. :)

I posted on another forum hoping someone would have insight into the problem: https://forum.opnsense.org/index.php?topic=15036.0

Franco replied, indicating that since I was working with OPNsense 20.1, FreeBSD 12.1 and arm, I was on my own.

Well, I accepted the challenge and looked into the matter anticipating that it would save someone else some time.

I'm pretty sure I've found the cause of the issue and have implemented a solution that works for me.

The first hint came with a reading of /usr/src/UPDATING :

   20180406:
    In addition to supporting RFC 3164 formatted messages, the
    syslogd(8) service is now capable of parsing RFC 5424 formatted
    log messages. The main benefit of using RFC 5424 is that clients
    may now send log messages with timestamps containing year numbers,
    microseconds and time zone offsets.

    Similarly, the syslog(3) C library function has been altered to
    send RFC 5424 formatted messages to the local system logging
    daemon. On systems using syslogd(8), this change should have no
    negative impact, as long as syslogd(8) and the C library are
    updated at the same time. On systems using a different system
    logging daemon, it may be necessary to make configuration
    adjustments, depending on the software used.

    When using syslog-ng, add the 'syslog-protocol' flag to local
    input sources to enable parsing of RFC 5424 formatted messages:

            source src {
                    unix-dgram("/var/run/log" flags(syslog-protocol));
            }

    When using rsyslog, disable the 'SysSock.UseSpecialParser' option
    of the 'imuxsock' module to let messages be processed by the
    regular RFC 3164/5424 parsing pipeline:

            module(load="imuxsock" SysSock.UseSpecialParser="off")

    Do note that these changes only affect communication between local
    applications and syslogd(8). The format that syslogd(8) uses to
    store messages on disk or forward messages to other systems
    remains unchanged. syslogd(8) still uses RFC 3164 for these
    purposes. Options to customize this behaviour will be added in the
    future. Utilities that process log files stored in /var/log are
    thus expected to continue to function as before.

    __FreeBSD_version has been incremented to 1200061 to denote this
    change.`

Studying the version of syslog.c in FreeBSD 12.1, did, indeed, show that syslog() adds an RFC 5424 formatted header to the log message being sent to syslogd.

Apparently syslogd hasn't been modified to completely accommodate the change and neither has OPNsense. My solution is to revert to the old version of syslog.c, which formats log messages with an RFC 3164 type header. I expect the OPNsense authors will prefer a different solution.

[nekoprog]

Don't take it personally. What he meant was that 20.1 and 12.1 build is still in alpha/beta development branch, hence there will be bug along the road. As a tester, you need to address this issue on core.git hoping that core dev will look deeper into this issue. This will surely be fixed on stable 20.1 release next year, still got a few weeks then.

[dgktkr]

No worries. One reason I like open source software is that if something bothers me enough I can further develop my coding skills and fix it myself.

@nekoprog, with your encouragement I've opened an issue for OPNsense src.git on GitHub https://github.com/opnsense/src/issues/49 I hope that's the right place to do it.

[dgktkr]

FWIW OPNsense src.git has progressed and is now based on HardenedBSD. And it even builds OK.