search

grisp / grisp2-rtems-toolchain

Basic tools and libraries to get an RTEMS system running on GRiSP2.
Other
13 stars 6 forks source link

Update to RTEMS 6 #52

Open ecpeterson opened 11 months ago

ecpeterson commented 11 months ago

Currently, GRiSP uses RTEMS 5. I'd like to upgrade it to use RTEMS 6 or later. This was originally motivated by wanting to build GRiSP against a Xilinx target for which RTEMS only gained support on version 6. I'm not interested in that any longer, someone else may be, and I feel it's worth considering the upgrade anyway.

I'm opening this issue to track related work, rather than writing observations into a Slack workspace and losing them to its 90 day time horizon. I'm going to focus on GRiSP 2, since that's the board I have.

ecpeterson commented 11 months ago

The first batch of changes is straightforward:

  1. Update the RTEMS submodules to point to version 6: that's the master branch for most, but 6-freebsd-12 for rtems-libbsd.
  2. A bunch of obvious variables like RTEMS_VERSION=5 have to get changed. These are smeared all over the place, including in grisp_tools.
  3. RTEMS configuration is now handled by a config.ini, processed by ./waf configure. Settings like IMX_CCM_IPG_HZ go in there. Similarly, -specs bsp_specs should be erased everywhere. (This probably has to be handled less brutally in order to preserve support for GRiSP 1.)
  4. Add #define RTEMS_BSD_CONFIG_BSP_CONFIG to the rtems-libbsd setup at the bottom of erl_main.c .

Chasing these changes all the way out is enough to get it to compile, but not to boot all the way to an Erlang shell prompt.

ecpeterson commented 11 months ago

The first hurdle is that the boot routines in erl_main.c have a hard time finding the configuration files, including grisp.ini.

It seems that SD card bring-up has drifted slightly. I'm not sure exactly what's wrong, but putting an extra sleep(1) after the call to grisp_init_wait_for_sd is enough for it to settle out. I can see some other MMC-related devices continue to be brought up after this call returns; maybe that's related, and maybe not. In any case, this is a successful stopgap.

ecpeterson commented 11 months ago

The second hurdle is that ERTS's call to pipe returns enosys. This is because RTEMS's definition of pipe has changed in two important ways:

  1. The "virtual memory filesystem", IMFS, has a new configuration option which governs FIFO creation. It is disabled by default. You can turn it on by adding #define CONFIGURE_IMFS_ENABLE_MKFIFO to the settings at the bottom of erl_main.c.
  2. However, this IMFS definition of pipe causes ERTS to deadlock! I'm not sure why. There used to be a BSD-level definition of pipe, which has been removed in the upgrade to RTEMS 6: rtems-libbsd 2e5f80. Reinjecting this definition into the locally checked out copy of rtems-libbsd avoids the deadlock.

This gets us all the way to a working Erlang shell!

The author of the rtems-libbsd commit, Chris Johns, says that the change was motivated by not wanting users to get two different pipe behaviors depending on linker order, and it won't do to simply rollback this change to get GRiSP working again. That said, I don't think there is something "wrong" with the BSD definition — if the parts of it which are needed to avoid deadlock could be lifted to the IMFS definition, then they'd probably take the patch.

ecpeterson commented 11 months ago

Networking doesn't seem to work out of the box. The interfaces are there, link events are detected and reported, and all the configuration tools start successfully, but they don't install any settings. The timeout pattern is (again!) slightly different from RTEMS 5, staged between IPv4 and IPv6 and baked into ifconfig. Maybe this means the fix is just tinkering with ifconfig arguments.

It will be useful to have a debugger to sort out the pipe conflict, and bringing up a debugger is easiest done either by bringing up networking or getting QEMU to emulate the board (or both).

ecpeterson commented 11 months ago

Not much success getting QEMU to work either. It offers the machine type mcimx6ul-evk which — while not identical to the GRiSP SoC — I was hoping would suffice to see the initial RTEMS boot sequence happen en route to booting the BEAM. No dice.

As part of its build process, RTEMS offers to generate a "testsuite" of smaller applications (e.g., hello.exe which prints to the console and halts, and ticker.exe which prints some timer information and halts). Our settings enable these secondary build targets, and the generated binaries can be found at grisp2-rtems-toolchain/external/rtems/build/arm/imx7/testsuites/samples. These fare no better: QEMU pegs the CPU and prints no output.

However, these build instructions for linux-imx do yield a binary which generates console output through QEMU. Perhaps I can interpolate between the two to figure out what I'm missing. One kernel boot operation that stands out is console=ttymxc0; perhaps RTEMS is putting output on a different serial port.

c-mauderer commented 11 months ago

Hello @ecpeterson,

I tried to run the RTEMS master on GRiSP2 some-when during 2022. Back then it worked fine. I didn't try Erlang. Please note that the following patches are still necessary, because they are not in the upstream sources:

rtems:

For the libbsd patches, I tried to get them into RTEMS-libbsd. It was suggested to upstream them to FreeBSD instead. But FreeBSD never integrated them: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=249231

Also note that currently libbsd on the 6-freebsd-12 branch might has some problems. The NFS patch set added quite some changes. It increased the size of the binaries quite a bit. Due to some differences how to handle that, patches from Sebastian that would also fix some vfs-related problems haven't been accepted. I think for example ntpd didn't work with the 6-freebsd-12 version from 2022. The increased size on the 6-freebsd-12 is unusable for quite some of our projects so I didn't test that branch recently. I'm not sure whether all vfs-related problems are now fixed. So if something doesn't work, you might want to try libbsd master instead. The NFS related patches are not there on that branch.

Best regards

Christian

peerst commented 9 months ago

@c-mauderer any updates on this?

We have an additional problem now because our current GRiSP toolchain doesn't build on Apple Silicon which are getting the majority of our users system. Or is there a version combination of RSB, libbsd and RTEMS which would build fine on Apple Silicon.

ecpeterson commented 9 months ago

(No update from me. I started a fresh job and have stopped GRiSPing for the time being.

Point taken that one need not use 6-freebsd-12; I had initially adopted that branch because I intended to build GRiSP against a ZynqMP board, which was most convenient to do on a branch with support for board devices. Perhaps that’s a separate project. Heads up, though, about the pipe issue, which will probably eventually rear its head on main.)

c-mauderer commented 9 months ago

@c-mauderer any updates on this?

We have an additional problem now because our current GRiSP toolchain doesn't build on Apple Silicon which are getting the majority of our users system. Or is there a version combination of RSB, libbsd and RTEMS which would build fine on Apple Silicon.

Also I didn't test recently, GRiSP2 should work on an up to date libbsd if the three patches from my last post are applied.

But I'm not sure whether the tools build on the Apple Silicon. I think I have seen at least one mail thread on the list that there are some problems. But that's already at least a few month back so the issue could be resolved by now. Back then it was an upstream problem in gcc or gdb.

I don't have an Apple device to test anything. But I just pushed an update to our CI branch of the rtems-source-builder:

https://github.com/embedded-brains/rtems-source-builder/actions/runs/7897978927

If the build of the ARM tools on MacOS works, It might could be worth to try it on an Apple Silicon too.

Edit: It seems that some download currently failed. I try to add a retry for that.