QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
534 stars 46 forks source link

Change the OS used in dom0 #1919

Open mfc opened 8 years ago

mfc commented 8 years ago

We have discussed this numerous times but don't have an issue to track these discussions. It would be worth understanding what would be needed to change dom0 from Fedora to Debian (say Debian 8). Benefits include:

This ticket does not encompass modifying the desktop environment.

andrewdavidwong commented 8 years ago

The other issue about changing/upgrading dom0's OS: #1807

TNTBOMBOM commented 8 years ago

the problem is , why anyone would take fedora which is very known for its issues (from about 2000) and their enterprise lockin on the code (redhat sucking stuff...) ?

is there anyone can give me good reasons why taking fedora at the first place ? why not debian ? why not even gentoo (which is regarding of code easiness/freedomcy better than fedora)?

im afraid the one who took the first (lets call it "bad") decision is still continuing to take these decisions which will make qubes a disaster even if u shifted to debian or X distro.

so i think consultations on these decisions is very important. and fedora to be the core of qubes or any other linux distro is a mistaken step.

woju commented 8 years ago

Fedora is currently hosting the graphical environment and, like it or not, it has the best compatibility with graphic cards. Debian generally has old X drivers and older kernel. This is especialy problematic with Qubes, because the hardware selection is already constrained by virtualisation extension.

We will surely consider switching the distro when we implement GUI domain. It is on out roadmap and depends on GPU passthrough, which does not currently work, at least not without patching Xen and libvirt. We this it will be hard enough, and the effort should be directed there.

When switching, it will be neither Fedora nor Debian or any general purpose distro, but something very minimal, like Alpine.

Jeeppler commented 8 years ago

What exactly is the issue with GPU passthrough and Xen?

marmarek commented 8 years ago

Take a look at this thread - it's a good summary of GPU passthrough and PCI passthrough to HVM at all: https://groups.google.com/d/topic/qubes-devel/mvZBIUuyjv0/discussion

minad commented 8 years ago

It would be really nice to have a choice here. Like @woju I think it would be best to have a minimal distribution. However it might be nice to try something more declarative like NixOS.

I guess you wouldn't want to maintain different distributions, at least not for the official distribution? Do you have an idea how many Fedora specific configuations are in the qubes builder and the applications?

marmarek commented 8 years ago

I guess you wouldn't want to maintain different distributions, at least not for the official distribution?

Exactly.

Do you have an idea how many Fedora specific configuations are in the qubes builder and the applications?

During the current Fedora 20 -> 23 upgrade I'm trying to collect that list. Actually two lists:

I think the only really Fedora specific thing is update mechanism, other parts are just some system configuration which may need some adjustment when porting to other distribution (like config file locations etc).

Yes, we're considering NixOS here, but no decision has been made yet.

unman commented 8 years ago

I think the only really Fedora specific thing is update mechanism, other parts are just some system configuration which may need some adjustment when porting to other distribution (like config file locations etc).

I've got some experience here as I've been running Debian in dom0 for a while. It's an unholy mix of packages, alien and hand crafted stuff,(hand crafted like the Borja Jesus), and not full Qubes. But it works well enough for my slight requirements. I'm hoping that the upgrade to 3.1 will be somewhat closer to full Qubes. So far I would say there is almost nothing Fedora specific that comes outwith the normal repackaging process.

It's interesting that @mfc suggests that Debian provides increased hardware compatibility, while @woju says Fedora has best compatibility with graphic cards. Is there any evidence either way?

minad commented 8 years ago

The Borja jesus :joy: That still sounds somehow encouraging! What are your reasons for prefering Debian on the dom0? I just started using qubes and I had much more experience with Debian, Arch, ... but basically none with Fedora. It took a short while to get accustomed - but in the end it feels all the same with some minor package manager differences. Since qubes takes a more radical approach in general, I think it would also make sense to consider a change to something conceptually "better" or more advanced like nix.

h01ger commented 8 years ago

unman, could you please share more details about your setup, maybe even commits? ;-) Are you running this on a jessie based dom0 or stretch? How did you build+install the qubes specific stuff?

I'm still pondering doing the same, and looking at the cubes git repos I've identified these as being needed for dom0:

Is that correct? Which are missing? Cause once dom0 is working one should be able to deploy VMs from a backup… ;-)

marmarek commented 8 years ago

On Wed, May 18, 2016 at 09:03:43AM -0700, Holger Levsen wrote:

I'm still pondering doing the same, and looking at the cubes git repos I've identified these as being needed for dom0:

  • qubes-core-admin
  • qubes-core-admin-linux
  • qubes-core-agent-linux
  • qubes-core-libvirt
  • qubes-core-vchan-xen

Is that correct? Which are missing?

In addition to above:

And mgmt-salt stuff (everything with "base" and "dom0").

Some of them already have debian packaging.

Packaging Xen for dom0 would be tricky because of stubdomains. It consists of some external libraries/tools built as part of Xen build and statically linked. Something that AFAIR is against Debian policy (for example there are multiple source tarballs, from different projects).

And template packages needs to be converted to deb. Or not packaged at all and installed some other way - which I think would be even better option (installing templates as packages have a lot of side effects...).

Cause once dom0 is working one should be able to deploy VMs from a backup… ;-)

Yes, it should be possible.

Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

h01ger commented 8 years ago

Marek, thanks for the hints!

For a start I don't mind violating Debian policy if needed. That said, while statically linking and embedded code copies are discouraged in general, there are also exceptions to these rules.

But really, at first I just want to be able to use a Debian based system as dom0.

unman commented 8 years ago

@h01ger Worked through the build logs from fedora and the spec files. I followed the same build order and worked through each package in turn. I started with a base jessie install, and built up from there. I tried cherry picking Xen packages from sid and applying the Qubes patches, and that worked reasonably but I found difficult to incorporate stubdomains, and got pretty lost. In the end I basically built everything from source. It was relatively easy to identify Debian equivalents for the fedora requirements, and referencing the native Debian packages also helped, particularly for Xen. I had some success using alien for simpler stuff, although some paths and scripts needed to be bodged by hand. I didn't build manager or salt stack, but I cant imagine they would be difficult. Some stuff just doesn't work at all - can't boot from iso in HVM, for example. And nothing is really packaged so it's probably fragile on updates.

I'm currently working on 3.1, and trying to do the right thing, packaging properly and (sometimes) using qubes-builder framework, but I have limited time before my next trip. When I have anything sensible to share I will certainly commit, but I'm a way off that at present.

When I started there was a clear thought that Qubes would be moving to Debian, and I wanted to see what would be involved. Now, of course, that's no longer the case, so I suspect time is better spent working on Debian issues on the vm side.

Jeeppler commented 8 years ago

I thought one of the main reasons to switch from Fedora to Debian are reproducible builds. Debian made and make a lot of progress in this area.

The other question which was discussed before is moving from KDE to GTK (Gnome) for the desktop.

@woju proposed above a switch to Alpine Linux. I understand that Alpine Linux has nice security features, but it seems to be a radical switch. A switch to Alpine would be less radical if the management GUI and dom0 would be separated. In the case the separation between GUI and dom0 would be possible and implemented, dom0 would be Alpine, but what about the GUI domain?

Why not going step by step. First moving from Fedora to Debian, because of the ability to create reproducible builds. Then moving from KDE to GTK (Gnome). The final step would be to separate Dom0 into separate domains. The management domain would be Alpine whereas the graphical engagement domain would be still Debian.

Another good point of @woju is the problem with drivers, older kernel and drivers in Debian. This is true for Debian stable, but Debian has a testing, unstable and experimental repository which contain more up to date packages.

Debian and Linux in general has many distribution derivatives. Why not using one of the derivatives which fulfills the requirements:

Tanglu is a distribution which fulfills all the requirements. Tanglu is based on Debian, but uses software packages from testing, unstable and sometimes experimental to provide the newest drivers and other software packages, but is still based on Debian stable. Tanglu has three main editions KDE, GNOME and core. In addition Tanglu uses Calamares the independent system installer.

adrelanos commented 7 years ago

For reference:

h01ger commented 7 years ago

https://wiki.debian.org/Qubes/Devel is the relevant URL for this issue.

https://wiki.debian.org/Qubes just describe how to use Qubes from the point of view of someone familar with Debian - and that wiki page is also a bit outdated.

adrelanos commented 7 years ago

@unman in https://github.com/QubesOS/qubes-issues/issues/1781#issuecomment-262856034

I don't think that #1919 is still a target.

How did you come to that conclusion?

In any case it seems to me that most users should be kept well away from dom0 and so what is running there is irrelevant. If anyone has the nous to tinker in dom0 then they should be able to handle the differences.

I don't think Fedora can be as transparent as that. Once one wants to do certain things, one example is #1375 and there are others, one is back to learning about Fedora. Same when issues happen or when debug info is requested. And that probably is not going to go away. Also higher effort for developers to know two systems. For example makes getting grsecurity in Qubes harder since an rpm package / repository is required.

marmarek commented 7 years ago

Once one wants to do certain things, one example is #1375

No, it isn't - if you have enabled USB VM. Generally we want to keep user away from manipulating dom0 as much as possible. The extreme case (on the roadmap!) is GUI domain, where user have no way for direct interaction with dom0. Anyway, we're not going to switch dom0 to any other distribution until implementation of GUI domain.

adrelanos commented 7 years ago

No, it isn't - if you have enabled USB VM.

Which does not work for me due to effectively only 1 usb controller and all usb devices including internal keyboard connected to that, no ps2 either.

Other issues #2387 / #2360 also require quite some Fedora knowledge to sort out.

Anyway, we're not going to switch dom0 to any other distribution until implementation of GUI domain.

That sounds great and also sounds like there is still hope for dom0 Debian.

marmarek commented 7 years ago

That sounds great and also sounds like there is still hope for dom0 Debian.

Yes, that's right. But because of complexity of GUI domain, it will not be soon.

jpouellet commented 7 years ago

@adrelanos re 1 USB controller: have you tried https://github.com/QubesOS/qubes-app-linux-input-proxy?

It provides questionable security benefit, but (of particular relevance here) does further the goal of users not needing to do much in dom0.

v6ak commented 7 years ago

What about Ubuntu LTS? It provides 5Y support with releases every 2Y. So, we could stick with one version for 3Y-5Y.

It seems that there is some support of new hardware: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

adrelanos commented 7 years ago

Ubuntu has the wrong mindset privacy wise: https://www.eff.org/deeplinks/2012/10/privacy-ubuntu-1210-amazon-ads-and-data-leaks

Ubuntu is problematic license wise:

That's also the reason why there is no Ubuntu template for Qubes.

https://www.qubes-os.org/doc/templates/ubuntu/

Jeeppler commented 7 years ago

@v6ak Ubuntu LTS or Debian "Stable" does not matter. Ubuntu is based on Debian "Stable". Additionally, both offer support for 3/5 years. However, as @adrelanos pointed out there are license and privacy issues regarding Ubuntu.

Fedora is a nice choice, but the 9 months release cycle is a problem for dom0 in Qubes OS. The release cycle of Fedora is too short as a result most Qubes OS versions ship with an soon to be outdated or outdated version of Fedora as Dom0. Debian "Stable" and Ubuntu LTS on the other hand do not always support new hardware and kernel/packages are even on release date kind of out-of-date. I would prefer if Qubes OS would use a Debian "Testing" based OS for Dom0 with a semi-rolling or rolling release model. SparkleLinux (https://sparkylinux.org) could be an option.

v6ak commented 7 years ago

@v6ak https://github.com/v6ak Ubuntu LTS or Debian "Stable" does not matter. Ubuntu is based on Debian "Stable". Additionally, both offer support for 3/5 years.

AFAIK Debian does not provide 5Y support for new hardware.

However, as @adrelanos https://github.com/adrelanos pointed out there are license and privacy issues regarding Ubuntu.

Yes, this is a good point.

as a result most Qubes OS versions ship with an soon to be outdated or outdated version of Fedora as Dom0.

If Fedora has a better support for new hardware, it is effectively nullified by having outdated version, itn't is?

I would prefer if Qubes OS would use a Debian "Testing" based OS for Dom0 with a semi-rolling or rolling release model. SparkleLinux (https://sparkylinux.org) could be an option.

Well, uh rolling release in dom0? I want my dom0 to be working and stable. I know this might be a little bit contradictory (support for new hardware requires changes), but still. Introducing a new drivers to dom0 is probably rather wanted, major upgrades of some packages (especially Xfwm/Kwin) are however likely to be breaking in context of Qubes.

Jeeppler commented 7 years ago

Sorry, for using the term outdated. Unsupported is the right term. Fedoras release model is as following: "Release X is supported until one month after the release of Release X+2." [1] as a result Fedora 24 will reach it's end of life (EOL) approx. one month after the Fedora 26 release. The current supported versions are Fedora 24 and 25. Therefor Fedora 23 which is currently used for Dom0 is unsupported by the Fedora team. Unsupported does not mean Fedora is outdated. Fedora comes always a lot of new software in every release. The hardware support is still superior compared to other distribution.

Kalilinux is doing pretty well since they based their OS on the Debian "Testing" branch. Other distributions are successful doing the same. The Debian "Testing" branch is far more conservative compared to Arch Linux. Arch Linux has new software shortly after it is available. With other words Arch Linux follows a pretty aggressive rolling release model. Debian "Testing" on the other hand accepts packages only if they have been proven to work in the Debian "Unstable" branch [2]. As a result, I would consider Debian "Testing" as having conservative rolling release model. In addition, I think it fits into Qubes OS release model. Qubes OS receives minor upgrades and features even during the lifetime of a major release. For example, Qubes VM Manager updates or kernel updates during the lifetime of Qubes OS 3.2.

[1] https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle [2] https://wiki.debian.org/DebianReleases

v6ak commented 7 years ago

Therefor Fedora 23 which is currently used for Dom0 is unsupported by the Fedora team. Unsupported does not mean Fedora is outdated.

It depends. With my new laptop, Fedora 23 is surely outdated and a newer kernel was needed.

Debian "Testing" on the other hand accepts packages only if they have been proven to work in the Debian "Unstable" branch [2].

This doesn't make it suitable for Qubes, see my note on Xfwm/Kwin. Maybe this could be solved by adding Xfwm/Kwin with high version number, but I am not sure about all the consequences. -- Sent from my fruity BlackBerry pocket computer powered by Android with K-9 Mail. Please excuse my brevity.

Jeeppler commented 7 years ago

@v6ak I do not get your point of Xfwm/Kwin. Please elaborate on that a little bit more.

adrelanos commented 7 years ago

Whonix was based on Debian testing. (now on stable) The short summary "testing was absolutely horrible". Long summary:

https://www.whonix.org/wiki/Dev/Operating_System#Why_is_Whonix_based_on_Debian_Stable.2C_not_Debian_Testing.3F

Jeeppler commented 7 years ago

@adrelanos that make sense. Thanks. I thought Debian "testing" is more reliable.

tasket commented 7 years ago

An Ubuntu-based distro (not Ubuntu itself) could be a better choice if long-term support in the form of patches is a priority, which I think it should be. There is also the benefit of Ubuntu's hardware testing program... they're the only distro I'm aware that has validated an extensive list of PC desktop systems and components and done the tweaking necessary to get them working smoothly.

Some of this effort trickles down to Debian, but the trend is unreliable. Debian does not really care if desktop users are stymied by a server-centric kludge or bad UI choice or naive defaults from upstream.

Trisquel Linux specifically tracks Ubuntu LTS releases. But if FOSS-only is too confining (and I'm not sure it is), it may be worthwhile to remove the few offending components from Ubuntu ourselves and put the distro under a Qubes moniker.

OTOH, Fedora is really a problem: It is the only major distro that does not sign its repo metadata as a full set, allowing attacks that selectively hold-back patches -- yes, this is a big deal! Yet Fedora-based distros like RHEL and CentOS do fully sign. The reason I surmise for the disconnect is Red Hat's desire to direct users to paid RHEL subscriptions as the "serious choice with security". Fedora also lacks considerably in package selection. Its a testbed distro, not for serious use.

Jeeppler commented 7 years ago

@tasket Trisquel would be a nice choice. However, it is 100% free and libre software.

I like the idea of an Ubuntu-based distro. However, removing offending components from Ubuntu is not worth the effort. There are several other distros based on Ubuntu. My favorite among them would be Linux Mint. Another advantage would be that Linux Mint offers Cinnamon, Mate, Xfce and KDE flavors out of the box. Other desktop environments can be installed afterwards (i3, GNOME etc.). All Linux Mint 18.x editions are based on Ubuntu 16.04 LTS [1].

[1] http://blog.linuxmint.com/?p=3145

tasket commented 7 years ago

@Jeeppler How many components are there to remove from Ubuntu: Amazon and the Ubuntu font and...?

Linux Mint switched to Debian years ago and had a history of negligence re: not passing along security patches. So I take it they switched back?

jpouellet commented 7 years ago

Mint got owned and served backdoored install ISOs not to long ago, and yet even after that they still don't make sigs available on their downloads page. I don't know anything about their internal processes or track record, but this as my first impression suggests there are likely serious security concerns about their development process and their goals as a project are not aligned with Qubes', especially for use in dom0.

Remember also that ideally we want to move towards a completely reproducibly built system. Debian has been making lots of progress on that front. Is this a priority for Ubuntu or derivatives as well?

v6ak commented 7 years ago

I guess that no privacy offending component would actually send anything in network-disconnected dom0. Yes, one might complain about unclean solution or about ideological background, but not about the result.

@v6ak I do not get your point of Xfwm/Kwin. Please elaborate on that a little bit more.

Window managers in Qubes generally require some patching in order to have properly colored window borders. Additionally, Qubes is tied to X11 (i.e., it currently cannot be used with Wayland in dom0) and just a change in external component (e.g., Xfce) cannot fix it without a modification in Qubes.

When switching from Fedora 20 to Fedora 23, KDE was upgraded from 4. to 5.. No matter how much stable KDE Plasma was, it required KDE Qubes integration to be rewritten from scratch, with a different design. The new Kwin is not directly tied to X11, so its API tries to be display server agnostic.

While this might be seen as painful, it is natural part of such project. Now, developers have at least some option to decide when to upgrade Fedora. For example, they have decided that Qubes 4 will be still based on Fedora 23 in order to release it sooner. But try to imagine this with a rolling release distro. One day, some (even well-tested) update with breaking effect on Qubes will come. While we can try to skip its installation (e.g., by depending on an exact version of a package), we can get another classes of problems like dependency hell and requiring old unavailable package versions. It might be manageable, but it would probably generate urgent tasks for developers.

Jeeppler commented 7 years ago

@tasket @marmarek once pointed out that the problem about Ubuntu is that the Ubuntu license prevents the Qubes team from redistributing a modified Ubuntu under the Ubuntu name. Even if you remove all the privacy concerning parts of Ubuntu you still end up with the trademark issues.

Linux Mint is based on Ubuntu. I think you got confused between Linux Mint and Linux Mind Debian Edition (LMDE). LMDE is based on Debian.

@jpouellet your security concerns are maybe valid. The data breach which happened to them is one thing, that can happen to anybody. Linux Mint is a community project, Qubes OS developers and users can always work together with them to improve their security standard.

Debian and GuixSD are the two distributions which made some serious progress towards reproducible builds. Ubuntu on the other hand is not even listed in the reproducible-builds.org [1] page.

[1] https://reproducible-builds.org/who/

tasket commented 7 years ago

I don't think re-branding is a hurdle... more of a threshold. But I wasn't aware Ubuntu was ignoring reproducability. That, together with full repo signing and alternative architectures, makes Debian the more interesting choice.

v6ak commented 7 years ago

Sure, legal problems with Ubuntu remain if it is not rebranded.

Ideally, we would use some Ubuntu LTS fork that's well maintained and debranded. But I am afraid it is easier said than done.

Jeeppler commented 7 years ago

@v6ak qubuntu or qubesuntu ;-)

msusi commented 7 years ago

Until Fedora remains the default dom0 for QubesOS it would be nice to deblob it using for example packages from freed-ora: https://www.fsfla.org/ikiwiki/selibre/linux-libre/freed-ora.en.html

ghost commented 6 years ago

Related: https://github.com/QubesOS/qubes-issues/issues/833

Dom0 should definitely have the smallest possible TCB, so something like Alpine would make sense.

tasket commented 6 years ago

I'm a bit skeptical about alternatives like Alpine. They are rarely discussed or deployed, and the fact that the Qubes community has not adopted it for domU use may indicate a lack of flexibility or utility in the distro. I sometimes find even Fedora's selection of CLI tools too limiting.

Further, it adds a third Linux flavor that contributors and admins must become familiar with. This saps attention and energy away from exploring true alternative operating systems.

v6ak commented 6 years ago

I'm a bit skeptical about alternatives like Alpine. They are rarely discussed or deployed, and the fact that the Qubes community has not adopted it for domU use may indicate a lack of flexibility or utility in the distro. I sometimes find even Fedora's selection of CLI tools too limiting.

+1

While GUIVM will reduce the need of various drivers etc., I still find helpful if I can Google for various issues and find Fedora-related (or Debian-related or so) issues. I am not sure if I know all the relevant drivers, but:

Jeeppler commented 6 years ago

My biggest issue with Fedora as Dom0 is, that the version which is used is in Qubes OS is already unsupported by the Fedora community. Qubes 4.0 is a good example. Qubes 4.0 was released on the 29th March 2018. Qubes 4.0 ships with Fedora 25. The end of life for Fedora 25 was the 12th December 2017. I would prefer to have a long term distribution as basis for Qubes OS. Both Debian and CentOS would be better options, because of there long term support. However, the problem with both is the lack of up-to-date driver for new computer models. In case of Debian this could be avoided by using Debian 'testing'.

v6ak commented 6 years ago

Note that up-to-date Alpine might not get more community support than outdated Fedora. For CentOS and Debian, you're right.

IMHO this will not be done before dom0+GUIVM split. After the split, it should be much easier to make changes to dom0 or GUIVM, either upgrading or switching to another distro.

adrelanos commented 6 years ago

From Whonix linux distribution maintenance experience I can tell, that Debian testing is a nightmare.

https://www.whonix.org/wiki/Dev/Operating_System#Why_is_Whonix_based_on_Debian_Stable.2C_not_Debian_Testing.3F

I would very much like to see Debian stable in Qubes dom0 (and also sys-net / sys-firewall / sys-usb) by default. As a developer and user it is very difficult to cope up with two base distributions at the same time.

tasket commented 6 years ago

Seems like it would be easy for Debian stable instances to pull in drivers (especially) from testing. I've done this with various packages over the years with few issues.

DemiMarie commented 6 years ago

It seems that the problem is twofold:

One way around this is that the Linux kernel has a fantastic backwards-compatibility story when it comes to its userspace APIs. Perhaps we could drop-in a newer kernel? No, that won’t work.

Jeeppler commented 6 years ago

It would not be a huge issue to upgrade Dom0 once in a while. The upgrades should not break anything. Rolling release would be to fast and LTS is to slow. The best would probably be a semi-rolling release model. I think, having every 2-3 months a kernel upgrade would be acceptable.

DemiMarie commented 6 years ago

What about using Fedora’s system-upgrade mechanism to upgrade dom0?

On Mon, Apr 9, 2018, 3:15 PM Jeppler notifications@github.com wrote:

It would not be a huge issue to upgrade Dom0 once in a while. The upgrades should not break anything. Rolling release would be to fast and LTS is to slow. The best would probably be a semi-rolling release model. I think, having every 2-3 months a kernel upgrade would be acceptable.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/1919#issuecomment-379862714, or mute the thread https://github.com/notifications/unsubscribe-auth/AGGWB2Vbky0VQ9YKCwSrMcZPnZ7hRuwlks5tm7NbgaJpZM4IJ5V1 .