search

MichaIng / DietPi

Lightweight justice for your single-board computer!
https://dietpi.com/
GNU General Public License v2.0
4.88k stars 497 forks source link

Quartz64 | Migrate to kernel with more features #6151

Closed MichaIng closed 1 year ago

MichaIng commented 1 year ago

There are a couple of issues related to Peter Geis' kernel which has quite some limitations (missing kernel modules):

It seems to be intended since he is not reacting to my merge requests to add those features.

I now tested a bunch of alternative kernel builds, but none of them is great:

I also tested the Debian kernel: It boots, but at least Linux 6.0 from Bullseye backports lacks major features. One cannot check or change CPU governor or see current frequencies, to start with. And it is ~400 MiB vs ~100 MiB, as a single kernel for ALL 64-bit ARM devices.

So what do I aim to do instead?

I'm gonna merge the needed changes for Peter's builds into my own fork, adjust the CI pipeline, hope that it works just the same way, so that we just do our own builds and updates. This is also an opportunity to gather some experience in kernel and bootloader builds.

diederikdehaas commented 1 year ago

If you want to try the Debian kernel (again), you need to have one from the 6.1 series in which several kernel modules for Quartz64 devices got enabled and later on several patches (mostly DT related) got backported from 6.2 to Debian's 6.1 kernel.

MichaIng commented 1 year ago

Okay, but this is applicable for Bookworm images only to get updates via APT. If we are lucky, the Bullseye backport kernel gets a bump to v6.1 as well.

diederikdehaas commented 1 year ago

Correct. I have no idea (if/)when a 6.1 backports will be made though. Best to periodically check https://tracker.debian.org/pkg/linux for that.

diederikdehaas commented 1 year ago

I just checked and all 3 mentioned issues should be resolved with Debian's 6.1 kernel:

$ grep -E "CONFIG_CAN_GS_USB|CONFIG_SQUASHFS_XZ|CONFIG_BRIDGE_NETFILTER" /boot/config-6.1.0-4-arm64 
CONFIG_BRIDGE_NETFILTER=m
CONFIG_CAN_GS_USB=m
CONFIG_SQUASHFS_XZ=y
MichaIng commented 1 year ago

Yes it does, also v6.0 and earlier versions. And all Armbian kernels have it as well. Peter's build is quite special in being very minimal 😉. Also the fact that he uses RC versions only shows that it is more for testing than for productive systems. I hope I'm able to rebase the kernel branch onto e.g. v6.1.8.

MichaIng commented 1 year ago

Own builds are running: https://gitlab.com/MichaIng/quartz64_ci/-/pipelines/776450060

Looks pretty good. Tomorrow I'll try merging Linux v6.1.11.

diederikdehaas commented 1 year ago

Shouldn't be too hard. Usually it's the upgrade to a new major version which can be difficult.

AFAICT Peter's primary 'concern' is getting things in the upstream kernel and for that you should base your patches on the most recent RC1 version.

MichaIng commented 1 year ago

Makes sense. Same for U-Boot. Another approach would be to just build with unmodified upstream Linux and U-Boot.

diederikdehaas commented 1 year ago

What do you mean by unmodified upstream (Linux)? I don't have much experience with using upstream's defconfig's, but I'd assume they would be too limited for most. Certainly if you support multiple devices.

One major reason the Debian kernel for arm64 is (way?) bigger, is that it supports a whole bunch of different arm64 devices. Next to the 'standard features' which are present in all Debian's kernels. And that all gets put into 1 kernel build. If you only target (certain) Pine64 devices, which I guess Peter does, you can leave out all the other arm64 devices.

MichaIng commented 1 year ago

I mean the quartz64_defconfig from Peter with our additions. That one is copied in place by the CI pipeline and used for the build.

And another approach again would be (probably when Linux 6.2 has been released) to also use upstream defconf but applying customisation with a patch.

Ah I see now upstream is just a single arm64 defconf: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/configs/defconfig?h=linux-6.1.y

diederikdehaas commented 1 year ago

And another approach again would be (probably when Linux 6.2 has been released) to also use upstream defconf but applying customisation with a patch.

I'm not so sure you'd get what you want with that. It will add support for various devices which you don't seem to support.

MichaIng commented 1 year ago

Note the second part of my sentence "applying customisation" which implies removing support for all but the Quartz64 variants. We are still DietPi, so I'm gonna build an own kernel, it will have minimal unnecessary overhead 😉.

MichaIng commented 1 year ago

Good news:

Bad news:

MichaIng commented 1 year ago

There it is: https://about.gitlab.com/pricing/

400 minutes. That is done with 2-3 kernel builds 😄. Moving to GitHub tomorrow.

MichaIng commented 1 year ago

All done. U-Boot finally is only the stable v2022.04 (was rc1). I was able to migrate some parts for latest v2023.01, but others I wasn't able to figure out, especially the ATF (ARM Trusted Firmware) integration (actually optional(?) but nice to have). And also some SPL related RAM size/offset/address value which is suddenly needed and I don't know what to set it correctly.

However, images working now very well, ~only Quartz64 B test outstanding~: https://dietpi.com/downloads/images/testing/

Kernel (and bootloader) can be upgraded with firmware packages from here: https://dietpi.com/downloads/binaries/ Will be offered as well next DietPi update.

MichaIng commented 1 year ago

The kernel configuration is now hosted here: https://github.com/MichaIng/DietPi/blob/dev/.build/images/Quartz64/quartz64_defconfig

This is kinda new land for me, so please give feedback about it. I'm sure there are still a lot of features missing, which one typically expects, respectively which is needed for any of all the software titles we offer. And on the other hand I'm sure there are features enabled which are not needed, or which are not needed builtin but can be made a kernel module, to keep the main kernel slim. I already found and disabled a bunch of features, like VirtIO drivers, legacy PTY nodes and such things, not sure how they made their way into this config in the first place.

diederikdehaas commented 1 year ago

Assuming the request was directed at me, I'm not sure what kind of feedback you're looking for.
And whether I can give feedback at all as my experience is with the Debian kernel and that uses a completely different build system.

What I do know is making your own initial kernel config is a huge undertaking. You need to figure out what you need and don't need/want and figure out how to make that work as 1 kernel module possibly depends on another, which depends on several others ... turtles all the way down. I know for improving Debian's kernel config for Rock64, I made a LOT of kernel builds. And there all the groundwork was already done. I did look a lot at other projects kernel config, especially LibreELEC as I wanted to improve multimedia support.

MichaIng commented 1 year ago

Assuming the request was directed at me

Just to anyone who reads this. The new kernel solves a lot of reported issues/requests, so if anyone has some kernel config/build experience/knowledge, I'm happy for any input 🙂.

Debian kernel and that uses a completely different build system

It still uses a _defconfig in the end, isn't it? I can imagine that there are tools to manage it on an abstracted basis.

making your own initial kernel config is a huge undertaking

This is why I used Peter's which worked "well" (with the limitations) since we offer Quartz64 images, and extended/modified it as much as needed to solve reported requests and a little more about things I know about/looked up 🙂. No starting from zero intended.

what you need and don't need/want

Apart of all features we offer in our scripts and support for all software titles, that is indeed likely an iterative undertaking. This is why I am asking for input. There may be (there were, removed already) features enabled which are nonsense, probably only there for testing/debugging with downsides/overhead in production systems. VirtIO drivers are still some included, I didn't radically remove everything directly, but made them modules at least, the way they are on Debian. But I think they are actually pointless. There is CONFIG_ANDROID=y enabled, like on Debian, which is needed for Android emulations at least, which isn't needed, I'd say. But it has other impacts I read about, just not certain enough yet to remove it. And I have not much knowledge about all the encryption and hashing features, which ones are commonly used, which ones nearly never (can be made modules or removed) on modern Debian versions.

1 kernel module possibly depends on another

AFAIK this is internally solved. Linux has a dependency system, and I've seen features being available which are/were not explicitly enabled in your config, disabled by default, but found them to be needed for other features.

I know for improving Debian's kernel config for Rock64

Exactly what I am looking for 🙂. Not same SoC, but same manufacturer for SoC and SBC, so I lot of things apply to both the same way.

diederikdehaas commented 1 year ago

Debian kernel and that uses a completely different build system

It still uses a _defconfig in the end, isn't it? I can imagine that there are tools to manage it on an abstracted basis.

No, upstream's _defconfig files are irrelevant for Debian. Debian has 1 kernel for f.e. all arm64 devices and consists of:

That's the general gist of things, but there are more factors which play a role, like -rt or not. And possibly more; I still don't fully grasp it.

making your own initial kernel config is a huge undertaking

This is why I used Peter's which worked "well" (with the limitations) since we offer Quartz64 images, ...

That helps ... for Quartz64 devices, but I thought DietPi supported more devices? Or do they all get their own kernel (config)? As said, it took me quite a while to properly enable a few options, while the base was already fine. You need to make the config, build the kernel, verify whether the kernel config matches what you intended and then test it to verify it's working properly. And repeat that cycle for every feature you want. Hopefully I'm wrong and it is simple though.

There is CONFIG_ANDROID=y enabled, like on Debian

FTR: it's no longer enabled in Debian's kernels.

1 kernel module possibly depends on another

AFAIK this is internally solved. Linux has a dependency system, and I've seen features being available which are/were not explicitly enabled in your config, disabled by default, but found them to be needed for other features.

I don't know enough about the upstream build system, but I think it's declarative and that the proper tooling, like menuconfig, can make use of that. So with the proper tooling several things can be automated, but I'm not so sure that Linux does that all automatically for you.

MichaIng commented 1 year ago

No, upstream's _defconfig files are irrelevant for Debian.

I do not mean upstream files, but that they use the same type of config files with CONFIG_ definitions inside. And they do. The files are not called _defconfig, but otherwise it's exactly the same: https://salsa.debian.org/kernel-team/linux/-/blob/master/debian/config/arm64/config

That helps ... for Quartz64 devices, but I thought DietPi supported more devices?

The issue here is about the Quartz64 kernel only. In all other cases we do no own kernel builds but use Debian/Armbian/vendor kernel builds.

We may build more own kernels in the future, possible even taking the config from here as a reference, at least the learnings or find of features we definitely need (independently of hardware). But for now Quartz64 only.

You need to make the config, build the kernel, verify whether the kernel config matches what you intended and then test it to verify it's working properly. And repeat that cycle for every feature you want. Hopefully I'm wrong and it is simple though.

Do not take me wrong, I'm happy for your input, but I know about general development procedures 😉. I really only need feedback about the individual config entries of the above linked file, hints about features we definitely do not need or those we do need, or do not need as builtins, as far as hardware-independent or known for Rockchip SoCs (like, well, they do not contain VirtIO devices). All onboard hardware features work fine already and all software titles which did not work before (without workarounds) do work now, so it's about fine tuning, disabling nonsense and enabling missing things we might not be aware of yet, e.g. possibly hardware acceleration for video features, encryption or hashing which is currently missing but just done without noise on software basis.

upstream build system

In the end, its make, and this seems to resolve config/module dependencies automatically. Also Debian's debhepler tools apply some patches and then run make: https://buildd.debian.org/status/fetch.php?pkg=linux-signed-arm64&arch=arm64&ver=6.1.12%2B1&stamp=1676661599

MichaIng commented 1 year ago

feedback about the individual config entries of the above linked file

EDIT: That one I mean: https://github.com/MichaIng/DietPi/blob/dev/.build/images/Quartz64/quartz64_defconfig

MichaIng commented 1 year ago

I'll close this issue now. Major issues have been solved and the new kernel + bootloader seem to work pretty well. Further optimisations can be tracked in dedicated issues.