search

canonical / ubuntu-mir

Ubuntu Main Inclusion Process - formerly on https://wiki.ubuntu.com/MainInclusionProcess
14 stars 11 forks source link

Testing requirements for hardware enablement #30

Closed seb128 closed 1 year ago

seb128 commented 1 year ago

The current MIR position on testing leads to a situation where we can't enable new/specific hardware unless Canonical/one of our team members or contributors own the required hardware to conduct the testing.

Since we don't really have budget to allow the owning teams to buy testing hardware and the certification team has its own priority the consequence is that Ubuntu is falling behind other distributions/competitors on hardware support which I don't think is the wanted outcome.

One example is https://bugs.launchpad.net/ubuntu/+source/libqrtr-glib/+bug/1963707/comments/9 where we failed to enable support for qualcomm hardware in modemmanager

Recently pipewire added an optional depends on libffado to enable FireWire audio device support. I'm reluctant start the MIR processus for it though because in the current policy state I believe the outcome is going to be similar to the libqrtr-glib one.

I would ask the MIR to reconsider their current position on the topic. Ideally we would be able to test those features, but if we are not in the position of doing so I believe that having untested support which might be buggy is still better than plainly not handling the devices.

The policy also seems inconsistent, as far as I know we don't request our kernel team to own hardware and do testing for every driver they enable.

Thanks for considering

seb128 commented 1 year ago

Seems like Christian was already working on submitting a ticket for that, let me close mine and I will follow up on his

cpaelzer commented 1 year ago

I'm happy to provide all my input as answer, as I've said on IRC - thanks for bringing it up.

cpaelzer commented 1 year ago

Hi, this aspect of the MIR ruling has gone through a few phases. First one of abuse by being too lenient forcing us to bolster the requirements and now inhibiting innovation maybe a bit too much. Furthermore the Ubuntu world around a universe->main discussion has changed with Ubuntu Pro extending the services and guarantees offered. Let me outline that in a bit more detail in the form of an historical essay before we go into conclusions/solutions ...

Disclaimer: this intentionally reads between the lines and summarized so that people can understand how things evolved without reading 250+ bugs and then still needing a time machine.

Phase 1: Get more strict to prevent abuse by being too lenient

In the past we only said something like:

Rule: does it have a build time or autopkgtest?`

And that was used and interpreted very differently by different teams ...

a) If think of DPDK, or GPGPU support, there teams ended up saying something like:

"It only has rudimentary build and autopkgtest because it needs special HW, therefore we have ensured working with partners or by buying it that such HW is present and now have a test plan which we execute on every (sometimes only the major) upload. Some even got included in regular more CI style testing."

b) I do not remember an example I could point to, but we could find one if necessary, the statement was like:

"It only has rudimentary build and autopkgtest because it needs special HW, and no one has this special HW so no in depth testing will be possible"

This was used up to a point that sometimes felt like "tests are effort I hate, let us just state it needs special HW and we can go on". Maybe not in intentional but in a bit of an "who cares, we won't be there when it breaks" style.

And we need to remember that, being in main was a very big step that gave a package extra promises. Like it can be supported and will get deeper security coverage. It is easy to say "this can't be tested, let it in", but then the pain will be at the people providing the support/security which will be unable to test what they do.

So as a consequence this rule got extended by the MIR to be more explicit. Aligning it more and more with how some teams already did interpret the old lenient rule and keeping the pain of later support in mind. It didn't really change, but spelled out how people always felt it should be:

Rule: if special HW does prevent build/autopkgtest is there a test plan, code, log provided?

Later extended to be more readable:

Rule: This does seem to need special HW for build or test so it can't be automatic. Is there a test plan, code, hardware available and committment to cover that continuously?

Then we had a phase where it seemed people might just throw words at the MIR team like "run it on this HW" or give us "fake plans" never intended to execute (e.g. plans had obvious mistakes). So we started to also ask for a proof by asking for a sample run of that test plan:

Rule: This does seem to need special HW for build or test so it can't be automatic. Is there a test plan, code, hardware available and committment to cover that continuously? Please outline in detail and provide a sample log of a successful run.

IMHO this never intended to fully rule out anyone with a piece of software that couldn't be well tested. But to make it hard to misuse this to pass too easily. Teams should be fully aware of the long term risk and pain associated with weakly tested code and proof that they really tried and there really really really isn't a way (could be a simulator like in ipmitool, or budget to get HW like for dpdk) to test these things.

Phase 2: Unintended suffering

Almost everything has two sides, so the consequence of tightening rules obviously is the problem that @seb128 outlines in this report here.

Ubuntu is falling behind other distributions/competitors on hardware support which I don't think is the wanted outcome

I agree that this wasn't the wanted outcome, so it is great to discuss it here.

But as you read above, these rules have not been meant to make us fall behind and provide inferior solutions to our users. They are IMHO just means to ensure good quality for the better of Ubuntu users and Ubuntu developers alike.

The policy also seems inconsistent, as far as I know we don't request our kernel team to own hardware and do testing for every driver they enable.

While it is true that they do not test everything, they have budget, HW, people and many systems to cover as much as possible of it. This is the pattern also other use - I mentioned DPDK above, there we test only a subset of more common cards and work closely with the community covering the rest, but we do not test "none".

Meanwhile the Ubuntu world changed

Ubuntu Pro provides security coverage for all of universe and one can also get support for that as well. So, without going into detail (it is great, but this isn't the place) - the old argument of "It needs to be testable as we now need to service it" was rendered somewhat moot. Nowadays it needs to get service no matter how few attention it might have got upfront.

To be clear, Main is still intended to be an even safer, even more tested, opinionated and well integrated set of software compared to universe. But it no more strictly is the only set that will need to be serviced. And for that aspect anything going into main, even if it would stay as-is and get no QA and security check, would still be an improvement. Because it would have an owning team which security and support can reach out to :-).

A way out ...

I do understand that sometimes the effort to untangle dependencies is too high to do or maintain. So you end up not supporting whole devices or features at all. While I have a hard time agreeing "having untested support which might be buggy is still better", I think there should be a way to get into main. There need to be a way to compromise for certain cases. Yet without being a quick neglected "This needs special HW, I'm out of responsibility to try to test this"

Let us revisit what nowadays is the consequence if we let such cases into main (those cases which are no more true due to Ubuntu Pro are strikethrough):

So with that put down so straight I agree, there should be a way to let components into main if (IF) teams plausibly demonstrate that they did what is possible and are willing to add more whenever they can (I do not want to go back to the abuse days).

IMHO it needs to change from a strict requirement to a, as high of a priority as possible, recommendation

But also some things you consider impossible in your report shouldn't be considered that way. For example you said "We don't really have budget", but TBH that is just the point - try to fix this: Get such budget! And also: Get a process how you could use HW of solutions-qa, certification or other departments! Do research to find a simulator! While not always 100% reliable, try to get Community (users as well as upstreams) testing contacts!, Get in contact with the HW providers if they are willing to help!, ...

Overall it is a temporal balance of pain: Does it happen now, or does it potentially happen later (then usually bigger and with more urgency). The owning teams that want/need to go with an case that isn't fully testable, reproducible should be aware of what they sign up for. And to ensure that is the case - process wise - we usually ask them to state that in the MIR bug.

cpaelzer commented 1 year ago

I really want to help to get this settled, therefore

We should try to get the state of this issue and PR in a good state until the mid cycle meeting, then we can high-level Ack/Nack there and then make it active (or not).

seb128 commented 1 year ago

Thanks Christian for the detailed post, one comment

But also some things you consider impossible in your report shouldn't be considered that way. For example you said "We don't really have budget", but TBH that is just the point - try to fix this: Get such budget! And also: Get a process how you could use HW of solutions-qa, certification or other departments! Do research to find a simulator!

Right, those things are not impossible, it's just a matter of time and energy.

I will articulate my perspective a bit more in details.

Those features/hardware support aren't of strategic value for us (desktop / Canonical) and aligning the pieces needed to get the testing story in place isn't going to trivial and would impact other Canonical teams. While I agree that we should work toward that direction, seeing out roadmap/backlog of more important items I don't think we will be able to make a case to be able to commit the time/efforts investments required in the next cycles. During my previous discussions with the cert/oem teams it was clear that they have lot of important things to work on, and I didn't feel like our ask was realistically important enough that I could make a case for them to reconsider other priority work to accommodate us.

The conclusion to me is that Canonical/Ubuntu doesn't have the available resources at this point to reach the bar set for those non-strategic goals. We need to build a path that leads us there but that's probably going to be a multiple years journey

We should reach out to community/partner/upstream yes. In the libqrtr-glib case we tried, just failed to get any traction.

I would personally be happy with an outcome where the MIR team would consider granting exceptions based on demonstration of <enough investigation / proof of why there is currently no other option> (+ optionally a drafted plan to address the gap at some point)

cpaelzer commented 1 year ago

I would personally be happy with an outcome where the MIR team would consider granting exceptions based on demonstration of <enough investigation / proof of why there is currently no other option> (+ optionally a drafted plan to address the gap at some point)

That is almost exactly what I'm after atm. Just interrupted too often to get it completed, but soon (tm)

setharnold commented 1 year ago

Ubuntu Pro does change the calculus (the security team does indeed fix far more software with a much wider array of requirements than previously) but places the costs on the larger organizations that subscribe.

I think Christian's proposed text does a good job of trying to balance all the tradeoffs but I fear:

I know that we can't possibly own all the hardware we support. It mostly seems to work okay anyway.

Users are usually forgiving when we make mistakes. However, if we get a reputation for making mistakes often, users will be less forgiving over time, even if there's good reasons for why the mistakes are inevitable.

I don't know if I have a conclusion, but this is a reasonable enough place to put my thoughts.

cpaelzer commented 1 year ago

Thanks @setharnold for sharing your thoughts! And yes, that is the place to put thoughts and also yes, all we do is trying to balance all the tradeoffs

if we're not actually fixing a package in main because we can't test it, what's the point of it being in main?

Oh, please get me right - if something would not be convoluted in dependencies there would be no reason and it would stay in main. But it shouldn't make the user unable to use something by being not enabled. The reason behind that is usually the way how dependencies and features work.

To be clear we have cases where sources in main intentionally build e.g. some drivers/plugins into binaries in main and others (lack of testing or maturity) for universe. But sometimes that isn't an option.

I'll amend my text to also cover that aspect.

cpaelzer commented 1 year ago

FYI - this was discussed and agreed by all related Directors at the sprint. The PR should be ready to be merged at the next team meeting.