Closed ArcEye closed 7 years ago
If we could have a meta package that links to RCNs site for instance that would be ideal, but I am not sure that he builds outside of ARM.
Unfortunately that does seem to be the case, can only find varieties of armhf etc. Cannot easily locate any other kernel image source either.
Building a rt-preempt kernel is child’s play by comparison with rtai kernels for instance. I have just built and am running 4.8.2-rt3, which appears to be the latest patch set available and it works fine.
The process of building is quite straight forward, but the support ramifications are non-trivial. We end up going backwards towards wholesale kernel support, or hope that Debian sorts out its kernel building, with this as an interim measure.
A bunch of folks are using PREEMPT_RT kernels from Debian backports snapshots. What about picking those into the Machinekit repo? I don't know what the support implications are, except that it seems easy to pick those out, and not having to build packages is a big win. It looks like Ben Hutchings adds the RT patches whenever he builds a new kernel for which they exist, so we'd have periodic updates.
I don't know how the Machinekit repo is built, but I can provide a reprepro
configuration that pulls from snapshots. It pulls the linux
and linux-latest
source packages themselves, plus all binary packages needed for running and developing.
A bunch of folks are using PREEMPT_RT kernels from Debian backports snapshots.
Those are probably the ones to use, I looked at them previously. I seem to recall that they disappear without notice when new ones appear, so we would need to stash them somewhere.
I think the Machinekit repo uses reprepro already, at least that was what was discussed, but I don't have access to it.
Everybody can build his/her own rt-preempt kernel, can't they? I remember writing down in the docs how to do that. So why do we need to support that?
@ArcEye, exactly, PREEMPT_RT kernel packages occasionally show up in Jessie backports when the phase of the moon is right, and then disappear again when Ben Hutchings builds a newer kernel. I suggest these be picked out of backports (or a snapshot, when not currently available) and added to deb.mk.io.
@luminize I don't understand the questions, can you elaborate? Anyone can build a kernel, yes. This issue is about pre-built kernel packages where its Provides:
and MK package's Requires:
package relationships match so that apt-get install machinekit-rt-preempt
works. Are you suggesting we build our own kernel packages for deb.mk.io?
I think that @cdsteinkuehler has weighed in on this topic before, but I've forgotten where. Hopefully he'll chime in.
Everybody can build his/her own rt-preempt kernel, can't they? I remember writing down in the docs how to do that. So why do we need to support that?
Yes I wrote a brief entry about it, but to 75% of users, who barely know how to fetch packages, it will probably be so esoteric that we cannot expect large take up.
The main issue is linking the machinekit-rt-preempt package dependencies, to something that won't disappear again.
On 27 Oct 2016, at 17:50, John Morris notifications@github.com wrote:
Are you suggesting we build our own kernel packages for deb.mk.io?
No, i mean that if it's trivial to build a rt-preempt kernel, then users can build one themselves. My remark is about why take on the support chore if it's trivial to build a rt-preempt one.
On 27 Oct 2016, at 18:57, ArcEye notifications@github.com wrote:
Everybody can build his/her own rt-preempt kernel, can't they? I remember writing down in the docs how to do that. So why do we need to support that?
Yes I wrote a brief entry about it, but to 75% of users, who barely know how to fetch packages, it will probably be so esoteric that we cannot expect large take up.
Ok, that's sound logical.
The main issue is linking the machinekit-rt-preempt package dependencies, to something that won't disappear again.
Ok, so build one every 6 months? That might be easier instead of guiding people thru building kernels themselves.
On 10/27/2016 12:04 PM, Bas de Bruijn wrote:
No, i mean that if it's trivial to build a rt-preempt kernel, then users can build one themselves. My remark is about why take on the support chore if it's trivial to build a rt-preempt one.
Oh, I get it now. I see things differently.
IMO, there's not a big chore maintaining a repo with kernel packages pulled from backports, less work than maintaining the Xenomai kernels that MK ships. Initially, it should be relatively easy to configure reprepro to pull the packages from Jessie backports snapshots (I can supply a config). By contrast, the Xenomai kernel packages were an awful chore to get working the first time. Thereafter, the ongoing support burden should be minimal: if someone sees Ben Hutchings has built a new RT kernel package and if there's a need for it, picking it is a matter of pointing reprepro to a new URL and re-running reprepro. If users have kernel-related problems, there's not much support burden: tell them they can try downloading a different (older?) kernel from snapshots, or they can try building their own from instructions, or they can buy a motherboard that others are known to have had success with. That's as easy as the minimal ongoing Xenomai kernel support.
Providing RT kernel packages makes life much easier for users. With
upstream kernels pulled into the MK package repo, users will
automatically get the whole runtime environment including the right
kernel with apt-get install machinekit-rt-preempt
, and with upstream
packages, users can be sure the update process will always go smoothly.
There are various options for users to obtain their own kernels. Users could go grab those packages themselves, but they're hard to find in Jessie backports snapshots unless one knows the date, and then the one must configure the repo separately in apt. And like you say, users could follow instructions 1 and build a kernel from source (IMO not trivial), and MK packagers would need remove kernel package requirements. My experience has been that maintaining a kernel config that works with a variety of hardware is the most difficult part of upgrading to new kernel versions, and the MK docs seem to take on that challenge (BTW, the kernel config provided is broken by the markdown formatting); I'd rather let the Debian kernel maintainers deal with this. And one more point, it's arguable that user support could actually become more difficult with everyone running different kernels.
To me, there are overwhelming reasons for going this way, but I also respect that others might have very different ideas that are equally valid, and I'd like to hear the alternatives.
Forgive my naive look at this stuff, but....
We build the custom SoCFPGA armhf kernels out of necessity rather than out of choice; mostly to do with FPGA overlay stuff that is missing from more mainstream kernels. The nice thing about those two arm kernels is they only targeting very few devices with a very known set of peripherals.
For x86/amd64 kernels those choices in the kconfig would have to be much more generic, but I think that could be easily overcome by steal... I mean borrowing the default configs from the debian group and simply tacking on patches for rt. The required options to turn on full realtime with preemption are easy to add.
We could probably even automate the process, since the debian kernels are very clearly tagged, and the rt preempt patches are also similarly tagged and apply very nicely to the proper kernels. Something like the Jenkins hooks we are using now -
1) Watch the debian repo for a new kernel 2) Check the preempt-rt server for a matching patchset If there is a matching patch: 3a) send an email to the mailing list so a maintainer could manually build 3b) Automatically grab the kernel source, apply the patchset and kconfig options 4) Build and publish to the repo
From the discussion above, I would think the order of choices from least to most painful would be:
1) Cherry pick kernels from backports if they support the common features needed for MK 2) Maintain a MK kernel with hooks 3) Manually maintain a MK kernel
I'm a little behind with my deb package chops, but I am trying to remedy that and I could give a stab at an automated approach.
@dkhughes I think you're right. If it is one thing the project should have learned by now is that if we don't automate things, in a good and proper way, no shortcuts and one off hacks, we'll tie our own feet and hands with support issues. Then we'll run into a wall. So I'd think after reading your comment is that it's automation to go for.
Automation has my vote too.
We have to watch https://www.kernel.org/pub/linux/kernel/projects/rt/ for the patches, rather than watch the kernels. They do not always match the current kernel, currently 0.2 versions behind (patch 4.8.2, latest kernel version 4.8.4)
If there is a way to pick up Ben Hutchins builds and pull across to our repo, that has to be the way to go for simplicity. However checking Jessie backports, currently there are no rt-preempt versions of the 4.7.0 kernel in use and as @zultron and I alluded to, they just disappear as soon as a new one comes out anyway. https://packages.debian.org/source/jessie-backports/linux
Here are some instructions for locating the latest kernel in Jessie backports snapshots. Currently that is v. 4.6.4. There's also a 4.8.4 that might be for Sid.
These directions only apply to the x86 architectures. For armhf, RCN ships Beaglebone kernels, but I don't know how to address kernel packages for other SoCs.
linux-image-rt
kernel:-rt-amd64
on page; find next until the most recent package is
found, e.g. linux-headers-4.6.0-1-rt-amd64
Check the date and build a snapshot URL
date --date='2016-07-19 16:21:50' \
+http://snapshot.debian.org/archive/debian/%Y%m%dT%H%M%SZ`
http://snapshot.debian.org/archive/debian/20160719T162150Z
Configure the apt repo
echo "deb http://snapshot.debian.org/archive/debian/20160719T162150Z" \
"jessie-backports main" | \
sudo tee /etc/apt/sources.list.d/kernel-snapshot.list
Update apt cache, and deal with expired snapshot repo key
sudo apt-get update -o Acquire::Check-Valid-Until=false
echo 'Acquire::Check-Valid-Until "false";' | \
sudo tee /etc/apt/apt.conf.d/90ignore-release-date
sudo apt-get update
Install kernel package
sudo apt-get -t jessie-backports install linux-image-rt-amd64
Here's a reprepro update configuration (untested) to pull packages from the snapshot into a reprepro-managed package repo:
Name: jessie-linux-rt
Method: http://snapshot.debian.org/archive/debian/20160719T162150Z
Suite: jessie-backports
Components: main
Architectures: amd64 i386 source
VerifyRelease: !7638D0442B90D010
DownloadListsAs: .gz
GetInRelease: yes
FilterFormula:
package (% linux-headers-*-rt*) | package (% linux-image-*-rt-*)
| package (% linux-headers-rt-*) | package (% linux-image-rt-*)
| package (% linux-kbuild-*)
Delightfully arcane information @zultron :+1:
Armed with it I have pulled the 4.6.0-1-rt kernels, headers and linux-base-4.6 into my private repo (Manually rather than with your reprepro script) It all seems to work ( http://deb.mgware.co.uk )
Caveat:
If you do try it just add the jessie-backports
The repo was set up Dec 2015 as a kind of mirror of the MK one, when we were in the process of changing across to the current one, in case of disaster.
Therefore all the MK packages are of that vintage and out of date in many cases.
Snapshot of linux-image-4.8.0-1-rt-(amd64 and 686pae) is out now and I have added both plus headers to my private repo.
I have been running the previous version 4.6.0-1 for a week now without problems.
What I need now is access to the http://deb.machinekit.io repo. Then I can set up the reprepro scripts etc so we can host copies of the packages and link them to the debian control file in the machinekit-rt packages, thus fixing the original problem this thread relates to.
I don't know where this physically lives and thought it may be a redirection from @mhaberler s private server for which I have a user account. However the /var/www directories and files look to be rather dated and the repo is not there.
Unless a whizz-bang automatic solution is forthcoming, I am happy to update manually for now, at the same time as I check for updates for my own system.
On 11/5/2016 5:52 AM, ArcEye wrote:
I don't know where this physically lives and thought it may be a redirection from @mhaberlers' private server for which I have a user account. However the /var/www directories and files look to be rather dated and the repo is not there.
It's on Michael's server. The apache config shows the debian repository to be in /home/debrepo/www and not /var/www.
Charles Steinkuehler charles@steinkuehler.net
OK thanks, I'll look at it in a bit.
The 4.8.0-1-rt-amd64/686pae
linux images and headers are now in our repo.
This actually appears to be the kernel currently selected in Stretch, but no sign of it being back-ported to Jessie by Debian.
The main problem is that the linux-image-rt-amd64/i386
meta package from Debian is still pointing to an unavailable kernel (4.1.0-0-bpo-1-rt)
What I propose is creating our own meta packages, say linux-image-mk-rt-amd64/686pae
, changing the machinekit-rt package control file to that dependency.
Then the meta package can be updated whenever there is a new rt kernel in the repo, without having to touch anything else.
It is only ever going to be a stop gap measure, until either Debian get their act together (unlikely, they seem to have moved on to Stretch and Jessie is history) or our user base moves to Stretch. We don't need to track every new version, just update now and then once the kernels have been tested.
The installing documentation will need a change and hopefully any error message from apt re machinekit-rt, will be detailing a required package which CAN be installed, thus solving the current problem.
If a way to hook into the snapshot.debian.org repo and pluck out new kernels is found, a method already in use for the component manual pages packages, to push them automatically to the repo, has been suggested as adaptable to this.
What I propose is creating our own meta packages, say linux-image-mk-rt-amd64/686pae, changing the machinekit-rt package control file to that dependency.
Well that seemed like a good idea at the time and I found how to tag it on to the machinekit-rt debian build so as it built automatically.
However after reflection, the whole circular nature of the dependency problem, where both wheezy and jessie machinekit-rt packages are built from the same control files, but only wheezy is catered for by debian, probably requires a simpler and cleaner solution.
linux-image-4.1.19-rt22mah
package.
It is new enough to support most new hardware and there is a newer kernel in the repo if required by the user.I am leaning towards 1.
It provides a working kernel, so that installs work without any additional knowledge or input.
There is nothing to stop us adding further rt kernels from the snapshots to our repo, to make them available easily, but does not introduce a requirement to do so.
A short entry in the docs installation guide can point the user to how to check the repo for newer kernels if they desire them.
What say you all? Shall we close this and get on with something else?
What say you all? Shall we close this and get on with something else?
Aye!
If there's a kernel that works then that's probably good enough. If people want to have the latest kernels for the latest hardware then it's logical to conclude that they have the skills to build their own.
This arises from #917 which highlighted that machinekit-rt-preempt package had become uninstallable due to the Debian meta package it specified no finding any kernels.
1087 provided a temporary fix by depending upon a kernel from our own repo
@zultron wrote:
@arceye wrote: I appreciate it is sub-optimal, not as unsatisfactory as depending on a meta package which does not link to any kernels however. Debian were building rt-preempt kernels for Jessie, but this seems to have slipped for some reason.
Our only other option is to build our own packages and host them, something we have tried to get away from.
@mhaberler very reluctantly independently uploaded his kernels when no others were available but did not like the support implications.
If we could have a meta package that links to RCNs site for instance that would be ideal, but I am not sure that he builds outside of ARM.