Jessie rt-preempt kernel availability

[ArcEye]

This arises from #917 which highlighted that machinekit-rt-preempt package had become uninstallable due to the Debian meta package it specified no finding any kernels.

1087 provided a temporary fix by depending upon a kernel from our own repo

@zultron wrote:

The issue with this change is that Debian RT kernels can no longer be used with Machinekit packages. It's also troublesome because the kernel version is hard-coded into the linux-image-4.1.19-rt22mah package name, which could complicate upgrades.

@arceye wrote: I appreciate it is sub-optimal, not as unsatisfactory as depending on a meta package which does not link to any kernels however. Debian were building rt-preempt kernels for Jessie, but this seems to have slipped for some reason.

Our only other option is to build our own packages and host them, something we have tried to get away from.
@mhaberler very reluctantly independently uploaded his kernels when no others were available but did not like the support implications.

If we could have a meta package that links to RCNs site for instance that would be ideal, but I am not sure that he builds outside of ARM.

[ArcEye]

If we could have a meta package that links to RCNs site for instance that would be ideal, but I am not sure that he builds outside of ARM.

Unfortunately that does seem to be the case, can only find varieties of armhf etc. Cannot easily locate any other kernel image source either.

Building a rt-preempt kernel is child’s play by comparison with rtai kernels for instance. I have just built and am running 4.8.2-rt3, which appears to be the latest patch set available and it works fine.

The process of building is quite straight forward, but the support ramifications are non-trivial. We end up going backwards towards wholesale kernel support, or hope that Debian sorts out its kernel building, with this as an interim measure.

[zultron]

A bunch of folks are using PREEMPT_RT kernels from Debian backports snapshots. What about picking those into the Machinekit repo? I don't know what the support implications are, except that it seems easy to pick those out, and not having to build packages is a big win. It looks like Ben Hutchings adds the RT patches whenever he builds a new kernel for which they exist, so we'd have periodic updates.

I don't know how the Machinekit repo is built, but I can provide a reprepro configuration that pulls from snapshots. It pulls the linux and linux-latest source packages themselves, plus all binary packages needed for running and developing.

[ArcEye]

A bunch of folks are using PREEMPT_RT kernels from Debian backports snapshots.

Those are probably the ones to use, I looked at them previously. I seem to recall that they disappear without notice when new ones appear, so we would need to stash them somewhere.

I think the Machinekit repo uses reprepro already, at least that was what was discussed, but I don't have access to it.

[luminize]

Everybody can build his/her own rt-preempt kernel, can't they? I remember writing down in the docs how to do that. So why do we need to support that?

[zultron]

@ArcEye, exactly, PREEMPT_RT kernel packages occasionally show up in Jessie backports when the phase of the moon is right, and then disappear again when Ben Hutchings builds a newer kernel. I suggest these be picked out of backports (or a snapshot, when not currently available) and added to deb.mk.io.

@luminize I don't understand the questions, can you elaborate? Anyone can build a kernel, yes. This issue is about pre-built kernel packages where its Provides: and MK package's Requires: package relationships match so that apt-get install machinekit-rt-preempt works. Are you suggesting we build our own kernel packages for deb.mk.io?

I think that @cdsteinkuehler has weighed in on this topic before, but I've forgotten where. Hopefully he'll chime in.

[ArcEye]

Everybody can build his/her own rt-preempt kernel, can't they? I remember writing down in the docs how to do that. So why do we need to support that?

Yes I wrote a brief entry about it, but to 75% of users, who barely know how to fetch packages, it will probably be so esoteric that we cannot expect large take up.

The main issue is linking the machinekit-rt-preempt package dependencies, to something that won't disappear again.

[luminize]

On 27 Oct 2016, at 17:50, John Morris notifications@github.com wrote:

Are you suggesting we build our own kernel packages for deb.mk.io?

No, i mean that if it's trivial to build a rt-preempt kernel, then users can build one themselves. My remark is about why take on the support chore if it's trivial to build a rt-preempt one.

[luminize]

On 27 Oct 2016, at 18:57, ArcEye notifications@github.com wrote:

Everybody can build his/her own rt-preempt kernel, can't they? I remember writing down in the docs how to do that. So why do we need to support that?

Yes I wrote a brief entry about it, but to 75% of users, who barely know how to fetch packages, it will probably be so esoteric that we cannot expect large take up.

Ok, that's sound logical.

The main issue is linking the machinekit-rt-preempt package dependencies, to something that won't disappear again.

Ok, so build one every 6 months? That might be easier instead of guiding people thru building kernels themselves.

[zultron]

On 10/27/2016 12:04 PM, Bas de Bruijn wrote:

No, i mean that if it's trivial to build a rt-preempt kernel, then users can build one themselves. My remark is about why take on the support chore if it's trivial to build a rt-preempt one.

Oh, I get it now. I see things differently.

IMO, there's not a big chore maintaining a repo with kernel packages pulled from backports, less work than maintaining the Xenomai kernels that MK ships. Initially, it should be relatively easy to configure reprepro to pull the packages from Jessie backports snapshots (I can supply a config). By contrast, the Xenomai kernel packages were an awful chore to get working the first time. Thereafter, the ongoing support burden should be minimal: if someone sees Ben Hutchings has built a new RT kernel package and if there's a need for it, picking it is a matter of pointing reprepro to a new URL and re-running reprepro. If users have kernel-related problems, there's not much support burden: tell them they can try downloading a different (older?) kernel from snapshots, or they can try building their own from instructions, or they can buy a motherboard that others are known to have had success with. That's as easy as the minimal ongoing Xenomai kernel support.

Providing RT kernel packages makes life much easier for users. With upstream kernels pulled into the MK package repo, users will automatically get the whole runtime environment including the right kernel with apt-get install machinekit-rt-preempt, and with upstream packages, users can be sure the update process will always go smoothly.

There are various options for users to obtain their own kernels. Users could go grab those packages themselves, but they're hard to find in Jessie backports snapshots unless one knows the date, and then the one must configure the repo separately in apt. And like you say, users could follow instructions 1 and build a kernel from source (IMO not trivial), and MK packagers would need remove kernel package requirements. My experience has been that maintaining a kernel config that works with a variety of hardware is the most difficult part of upgrading to new kernel versions, and the MK docs seem to take on that challenge (BTW, the kernel config provided is broken by the markdown formatting); I'd rather let the Debian kernel maintainers deal with this. And one more point, it's arguable that user support could actually become more difficult with everyone running different kernels.

To me, there are overwhelming reasons for going this way, but I also respect that others might have very different ideas that are equally valid, and I'd like to hear the alternatives.

[dkhughes]

Forgive my naive look at this stuff, but....

We build the custom SoCFPGA armhf kernels out of necessity rather than out of choice; mostly to do with FPGA overlay stuff that is missing from more mainstream kernels. The nice thing about those two arm kernels is they only targeting very few devices with a very known set of peripherals.

For x86/amd64 kernels those choices in the kconfig would have to be much more generic, but I think that could be easily overcome by steal... I mean borrowing the default configs from the debian group and simply tacking on patches for rt. The required options to turn on full realtime with preemption are easy to add.

We could probably even automate the process, since the debian kernels are very clearly tagged, and the rt preempt patches are also similarly tagged and apply very nicely to the proper kernels. Something like the Jenkins hooks we are using now -

1) Watch the debian repo for a new kernel 2) Check the preempt-rt server for a matching patchset If there is a matching patch: 3a) send an email to the mailing list so a maintainer could manually build 3b) Automatically grab the kernel source, apply the patchset and kconfig options 4) Build and publish to the repo

From the discussion above, I would think the order of choices from least to most painful would be:

1) Cherry pick kernels from backports if they support the common features needed for MK 2) Maintain a MK kernel with hooks 3) Manually maintain a MK kernel

I'm a little behind with my deb package chops, but I am trying to remedy that and I could give a stab at an automated approach.

[luminize]

@dkhughes I think you're right. If it is one thing the project should have learned by now is that if we don't automate things, in a good and proper way, no shortcuts and one off hacks, we'll tie our own feet and hands with support issues. Then we'll run into a wall. So I'd think after reading your comment is that it's automation to go for.

[ArcEye]

Automation has my vote too.

We have to watch https://www.kernel.org/pub/linux/kernel/projects/rt/ for the patches, rather than watch the kernels. They do not always match the current kernel, currently 0.2 versions behind (patch 4.8.2, latest kernel version 4.8.4)

If there is a way to pick up Ben Hutchins builds and pull across to our repo, that has to be the way to go for simplicity. However checking Jessie backports, currently there are no rt-preempt versions of the 4.7.0 kernel in use and as @zultron and I alluded to, they just disappear as soon as a new one comes out anyway. https://packages.debian.org/source/jessie-backports/linux

[zultron]

Here are some instructions for locating the latest kernel in Jessie backports snapshots. Currently that is v. 4.6.4. There's also a 4.8.4 that might be for Sid.

These directions only apply to the x86 architectures. For armhf, RCN ships Beaglebone kernels, but I don't know how to address kernel packages for other SoCs.

Find the snapshot URL for the most recent linux-image-rt kernel:

• Browse to http://snapshot.debian.org/binary/?cat=l
• Find -rt-amd64 on page; find next until the most recent package is found, e.g. linux-headers-4.6.0-1-rt-amd64
• Click the link
• On the next page, click the link to the most recent version

• Check the date and build a snapshot URL

  • e.g. "Seen in debian on 2016-07-19 16:21:50":

  date --date='2016-07-19 16:21:50' \
      +http://snapshot.debian.org/archive/debian/%Y%m%dT%H%M%SZ`

  • prints: http://snapshot.debian.org/archive/debian/20160719T162150Z

Install the kernel package on a system:

• Configure the apt repo

  echo "deb http://snapshot.debian.org/archive/debian/20160719T162150Z" \
     "jessie-backports main" | \
     sudo tee /etc/apt/sources.list.d/kernel-snapshot.list

• Update apt cache, and deal with expired snapshot repo key

  • Once:

  sudo apt-get update -o Acquire::Check-Valid-Until=false

  • Permanently:

  echo 'Acquire::Check-Valid-Until "false";' | \
      sudo tee /etc/apt/apt.conf.d/90ignore-release-date
  sudo apt-get update

• Install kernel package

  sudo apt-get -t jessie-backports install linux-image-rt-amd64

Reprepro configuration

Here's a reprepro update configuration (untested) to pull packages from the snapshot into a reprepro-managed package repo:

    Name: jessie-linux-rt
    Method: http://snapshot.debian.org/archive/debian/20160719T162150Z
    Suite: jessie-backports
    Components: main
    Architectures: amd64 i386 source
    VerifyRelease: !7638D0442B90D010
    DownloadListsAs: .gz
    GetInRelease: yes
    FilterFormula:
      package (% linux-headers-*-rt*) | package (% linux-image-*-rt-*)
      | package (% linux-headers-rt-*) | package (% linux-image-rt-*)
      | package (% linux-kbuild-*)

[ArcEye]

Delightfully arcane information @zultron :+1:

Armed with it I have pulled the 4.6.0-1-rt kernels, headers and linux-base-4.6 into my private repo (Manually rather than with your reprepro script) It all seems to work ( http://deb.mgware.co.uk )

Caveat: If you do try it just add the jessie-backports The repo was set up Dec 2015 as a kind of mirror of the MK one, when we were in the process of changing across to the current one, in case of disaster. Therefore all the MK packages are of that vintage and out of date in many cases.

[ArcEye]

Snapshot of linux-image-4.8.0-1-rt-(amd64 and 686pae) is out now and I have added both plus headers to my private repo.

I have been running the previous version 4.6.0-1 for a week now without problems.

What I need now is access to the http://deb.machinekit.io repo. Then I can set up the reprepro scripts etc so we can host copies of the packages and link them to the debian control file in the machinekit-rt packages, thus fixing the original problem this thread relates to.

I don't know where this physically lives and thought it may be a redirection from @mhaberler s private server for which I have a user account. However the /var/www directories and files look to be rather dated and the repo is not there.

Unless a whizz-bang automatic solution is forthcoming, I am happy to update manually for now, at the same time as I check for updates for my own system.

[cdsteinkuehler]

On 11/5/2016 5:52 AM, ArcEye wrote:

I don't know where this physically lives and thought it may be a redirection from @mhaberlers' private server for which I have a user account. However the /var/www directories and files look to be rather dated and the repo is not there.

It's on Michael's server. The apache config shows the debian repository to be in /home/debrepo/www and not /var/www.

Charles Steinkuehler charles@steinkuehler.net

[ArcEye]

OK thanks, I'll look at it in a bit.

[ArcEye]

The 4.8.0-1-rt-amd64/686pae linux images and headers are now in our repo. This actually appears to be the kernel currently selected in Stretch, but no sign of it being back-ported to Jessie by Debian.

The main problem is that the linux-image-rt-amd64/i386 meta package from Debian is still pointing to an unavailable kernel (4.1.0-0-bpo-1-rt)

What I propose is creating our own meta packages, say linux-image-mk-rt-amd64/686pae, changing the machinekit-rt package control file to that dependency. Then the meta package can be updated whenever there is a new rt kernel in the repo, without having to touch anything else.

It is only ever going to be a stop gap measure, until either Debian get their act together (unlikely, they seem to have moved on to Stretch and Jessie is history) or our user base moves to Stretch. We don't need to track every new version, just update now and then once the kernels have been tested.

The installing documentation will need a change and hopefully any error message from apt re machinekit-rt, will be detailing a required package which CAN be installed, thus solving the current problem.

If a way to hook into the snapshot.debian.org repo and pluck out new kernels is found, a method already in use for the component manual pages packages, to push them automatically to the repo, has been suggested as adaptable to this.

[ArcEye]

What I propose is creating our own meta packages, say linux-image-mk-rt-amd64/686pae, changing the machinekit-rt package control file to that dependency.

Well that seemed like a good idea at the time and I found how to tag it on to the machinekit-rt debian build so as it built automatically.

However after reflection, the whole circular nature of the dependency problem, where both wheezy and jessie machinekit-rt packages are built from the same control files, but only wheezy is catered for by debian, probably requires a simpler and cleaner solution.

1. Leave things as they now are, ALL i386/amd64 machinekit-rt packages (jessie and wheezy) are depended upon the linux-image-4.1.19-rt22mah package. It is new enough to support most new hardware and there is a newer kernel in the repo if required by the user.
2. Change this dependency to the newer kernel. Don't know if this is a good idea for Wheezy, the linux-base package etc might end up being too new for it.
3. Remove the dependency altogether and add information to the documentation about how to install a suitable kernel for the install type (linux-image-rt for Wheezy, linux-image-4.1.19-rt22mah or linux-image-4.8.0-1-rt from our own repo for Jessie)

I am leaning towards 1.
It provides a working kernel, so that installs work without any additional knowledge or input. There is nothing to stop us adding further rt kernels from the snapshots to our repo, to make them available easily, but does not introduce a requirement to do so. A short entry in the docs installation guide can point the user to how to check the repo for newer kernels if they desire them.

What say you all? Shall we close this and get on with something else?

[luminize]

What say you all? Shall we close this and get on with something else?

Aye!

If there's a kernel that works then that's probably good enough. If people want to have the latest kernels for the latest hardware then it's logical to conclude that they have the skills to build their own.