Open Rspigler opened 6 years ago
madscientist159
commented
6 years ago My vote would be seL4. Not only are they responsive to the threat posed by closed, highly privileged firmware, but the kernel is already being used in very high security installations.
Rspigler
commented
6 years ago I would truly love seL4 as well and hope that is the end goal; however, I believe that that course of action would take the longest of all 3. I would argue that porting Qubes for ppc64 (by way of 1 or 3) is of higher priority than working on substituting the seL4 microkernel for Xen, since that offers the quickest route to Qubes running a truly open platform. The security benefit of seL4 could then come after.
I could be wrong though.
madscientist159
commented
6 years ago Of the two, I suspect 3 is the fastest and probably a decent stopgap solution. 1 would be duplicating effort that could go into the seL4 port IMO.
jaesharp
commented
6 years ago If needed, I can provide a significant quantity of build and test infrastructure for this effort.
andrewdavidwong
commented
6 years ago There are various ways to achieve this compatibility, so I thought that this issue could be a way to track them/discuss.
The qubes-devel mailing list is the appropriate place for this sort of discussion. By contrast, qubes-issues is reserved for actionable issues as described in our issue reporting guidelines. If this issue is to remain open, it needs to be action-oriented rather than just a place for tracking and discussion. I'll change the title to reflect this.
awokd
commented
6 years ago I humbly propose this thread: https://groups.google.com/d/msg/qubes-devel/IFLyyCUbLmQ/_HtgOJK4AQAJ as a good place to continue the discussion!
Peter-Easton
commented
6 years ago I have had some difficulty getting to Google Groups, but I'd like to put my vote in as well for a port to the Talos II, whichever the professionals decide it will take the form of. I'm not well-versed enough in hypervisor design to be able to make any meaningful comment on which one would be best, but on the PPC64el, Debian runs KVM quite well with Libvirt for running other Debian and CentOS guests and is happily usable as a desktop with a minimum of work. I'd be very happy to beta-test a port to the PPC64, if you guys need testers.
Not only that but the Talos Lite is now here, which is about the price of high-end business grade or one of those high-end name-branded gaming laptops, and offers far better upgradeability for the future than a gaming laptop even when we don't look at the owner control/openness aspect of it. For many people interested in and serious about security, the price equivalent of a good quality laptop isn't a high one to pay, especially if the user isn't a firmware hacker that has the skills and know how to create their own "trusted stick" like what Joanna Rutkowska was suggesting for a stateless, verified boot x86 laptop.
tlaurion
commented
6 years ago The OTF funding form is here.
Time to tag developers, fund grant filling people (@mfc!) that might help so the discussion can follow on google groups and the information for the grant gets filled for the next OTF round!
mfc
commented
6 years ago hi all, i don't have any capacity to help out right now, but realistically i don't think OTF would fund such effort.
their target group is non-tech-savvy users in at-risk environments. these users have the current computers they own, they don't have the money/luxury of buying new hardware. Qubes' existing hardware requirements are already a very high barrier for these users and their target audience.
so if anything they would be more interested in funding Qubes development onto phones than Qubes on ppc64.
not stopping you all from submitting a proposal to OTF of course. maybe this is more something for a crowdfunding approach, for example.
Rspigler
commented
6 years ago Understandable. Going ahead with both might be possible. If a crowdfunding approach were to be done, what would be the estimated target goals for the KVM approach, and the seL4 approach? (seL4 could possibly be set as a stretch goal).
marmarek
commented
6 years ago As for KVM:
Well, the first non-trivial task would be research what really needs to be done. There are two main parts here:
The first one is much more predictable, I think it basically is:
I think we should assume 50-100kEUR goal for this part alone. Note it may be also beneficial for other architectures, not only ppc64.
The second part may take several months of work, but also may turn out to be trivial. Require someone knowing ppc64 architecture (which I don't know).
For seL4, I don't even try to provide any estimate, without doing some research on the current state of it and what features crucial for Qubes are missing. See FAQ for some hints.
Rspigler
commented
5 years ago Thank you for your detailed response and all your work on Qubes.
What would your estimate be for approach #1? (ppc64 port of Xen?)
marmarek
commented
5 years ago What would your estimate be for approach #1? (ppc64 port of Xen?)
Take a look at this xen-devel thread
tlaurion
commented
5 years ago GSoC proposal, anyone?
marmarek
commented
5 years ago In theory deadline for setting up GSoC idea page was Feb 6, but in practice period for students to choose projects haven't started yet. Feel free to open PR against gsoc.md. It's ok to have a proposal with very basic description, although this ticket already contain content that could be re-used.
Rspigler
commented
5 years ago -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
I am placing 1 bitcoin as a bounty for completing this task in porting Qubes.
As multiple developers will most likely be involved, I will divide the bounty between based on work done as I see fit. -----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEwsYOJ56G8Q1Wl3glNc4P5sIUGCMFAlxjj+AACgkQNc4P5sIU GCNDWw//YeGQtVH2xc6+qNRezvcBG3M4Ql0DaPFndSPqQ3tBaHsaBoAH7BT6o3Ih EU/pTCBWDAItC13yot4FejYp2eLRY4VtHPd3wHB+e5WVCZdKbW7Z1F2SWsPJutDo NSKC+Rb8CEZO0sxSe9CYP7plc6a6GK+WF+ZjgiSN0ISeBf/VMr1V7aXJgZ+cIjSp eaJCzQ0XSah0oO2brJXUTvPOHUKsFHv3vJLcZag+N5T3HZ5P6Zt6jYorCGRtRBzK aWVaqGb8MVR9CQUOcLJ8c0ubyRHwwpTWMgr0QQ0QUzwRTBKMyt8jAfPVshI1l/T5 xzr067/sX1cUIWXGp5kpvK32aX9Gb+Sh7e+BqYour4hGiSpq2Vzz2UAakLutM2s4 hkFfnZA1HiZsqS0bxR9/iNi1+dvEBc2U4KUd6ou++p4osve+PpHqWXJGd5Acewgf fcdhNDPSSif5ON56gZRzvEMhtbE4MXfIdEI8NQGihZfdyO+R+muKxDRDPq9HFK3s hH1RcAYx6jC1T7E4UENZMzw2n9xKIcBnbkURmx6byzRhZACcX/I6YMciZks5Xxdb yIItqpTqjW7IdabmDxEYUlG7g+9G8kzZMNFNiqbbxmvsd83O4OUuPOcYj9XAIRtw w5QmHMVKbPMoKPN9s15i3KXGTnpIQ80E0nGWai8NykgzzzMOH3c= =Tehn -----END PGP SIGNATURE-----
shawnanastasio
commented
5 years ago As for KVM:
Well, the first non-trivial task would be research what really needs to be done. There are two main parts here:
1. add KVM support in general 2. solve ppc64 specific problems (including verification of devices isolation etc)The first one is much more predictable, I think it basically is:
* implementing vchan using KVM-specific mechanism
I have been working on a reimplementation of vchan using KVM with the ivshmem shared memory backend, available here (comments welcome!).
I originally started it to assist in porting individual Qubes projects (mainly qubes-gui-daemon), but it would be great if it was useful for a port of Qubes as a whole.
marmarek
commented
5 years ago I have been working on a reimplementation of vchan using KVM with the
ivshmemshared memory backend, available here (comments welcome!).
Here you are: https://github.com/shawnanastasio/libkvmchan/issues/1
Rspigler
commented
5 years ago In the Qubes Architecture spec (Version 0.3), in the rational for choosing Xen over KVM, the authors mention how KVM relies on the Linux kernel to provide isolation. They also argue that Xen is much easier to audit due to a smaller code base, and that the I/O emulator, networking code, and drivers can be moved out of Dom0, further minimizing TCB.
This document was written 2010. These concerns still present?
marmarek
commented
5 years ago Mostly yes. In the meantime, some extra precautions were developed to isolate device emulator (I think the most relevant is seccomp filtering), so it is slightly better now. But the arguments of a) Linux vs Xen code size, and b) backend drivers on the host system still stands. The "b" point is especially bad, as it makes it hard to have host system isolated from the outside world (networking, usb etc).
On the other hand, a lot of development have happened on KVM since 2010, some of it very relevant for Qubes. For example much better support for GPU passthrough. And (the main point of this discussion) support for other architectures.
marmarek
commented
5 years ago Very interesting development: KVM support for Xen guests, including PV drivers. Given that Xen PV drivers supports backends in another guest, there is a chance this mechanism will support that too. Especially those patches emulates grant tables and event channels, which are designed with non-host backend in mind.
llebout
commented
5 years ago I'll add up another Bitcoin to the bounty, split across people same as above..
llebout
commented
5 years ago At current market rate, the bounty is worth more than $20,000 - I think that's good money.
shawnanastasio
commented
5 years ago A quick update. My vchan implementation for KVM is nearing completion. Once it's in a usable state, I plan on testing it with qrexec and other miscellaneous qubes utilities. If all goes well I'll look into getting qubes-gui working which will require a mechanism to map guest pages from the host which I'm still not sure how to (safely) do on KVM.
0spinboson
commented
5 years ago it's y'all's money of course, but as someone who doesn't see ppc ever becoming mainstream, while the project is rather under-staffed (considering the number of open issues on the tracker doubled since late 2016), I do think it's a strange priority to not want to give this to the Qubes project in general, but only to someone willing & able to port Qubes to a yuppie platform.
llebout
commented
5 years ago it's y'all's money of course, but as someone who doesn't see ppc ever becoming mainstream, while the project is rather under-staffed (considering the number of open issues on the tracker doubled since late 2016), I do think it's a strange priority to not want to give this to the Qubes project in general, but only to someone willing & able to port Qubes to a yuppie platform.
I am not interested in furthering Qubes OS support on x86. The architecture is doomed in regards to providing freedom to it's users. OpenPOWER is not a "yuppie" platform, I use it as my main workstation for day to day programming and administrative work. It would not benefit me at all to donate to the Qubes OS project if it's only goal is supporting x86 because I don't use x86 hardware anymore. So when Qubes OS is ported to ppc64[le] then I can explore donating for the rest of the project, otherwise, it's of no use to me..
A quick update. My vchan implementation for KVM is nearing completion. Once it's in a usable state, I plan on testing it with qrexec and other miscellaneous qubes utilities. If all goes well I'll look into getting qubes-gui working which will require a mechanism to map guest pages from the host which I'm still not sure how to (safely) do on KVM.
Awesome!
0spinboson
commented
5 years ago to clarify: by 'yuppie' I mean 'a platform for the quite well-off (who tell themselves Must Have Total Platform Security Or I Might As Well Move In With The NSA)', as 1300$ (ex VAT, shipping) for a 4c CPU in 2019 just means you have money to burn. Not that you aren't allowed to tell yourself these things, of course, but the thing you're afraid of is the govt and intelligence complex (which is of course interwoven with the Tech companies), not the arch and its shortcomings. Anyway, I've said my piece, so I'll leave you to it.
llebout
commented
5 years ago to clarify: by 'yuppie' I mean 'a platform for the quite well-off (who tell themselves Must Have Total Platform Security Or I Might As Well Move In With The NSA)', as 1300$ (ex VAT, shipping) for a 4c CPU in 2019 just means you have money to burn. Not that you aren't allowed to tell yourself these things, of course, but the thing you're afraid of is the govt and intelligence complex (which is of course interwoven with the Tech companies), not the arch and its shortcomings. Anyway, I've said my piece, so I'll leave you to it.
The CPU is cheaper than Intel in that regard, what costs money is the motherboard, and if I don't order RaptorCS offerings it will last even longer at that price, so that's meant to evolve of course. Personally I'm not afraid of surveillance or anything like that, I just think that by principle a machine you own means a machine you can hack. Not the case with Intel, AMD etc. You can't hack it.
tasket
commented
5 years ago I don't agree with the simplistic "thing you're afraid of". Governments are claiming justification for their expanded reach partly on the great success criminal syndicates have had in expanding theirs. And the former will (these days) typically pay ransoms for expediency over ethics, thus feeding a vicious circle.
Price points are an issue, but there are examples about how open source products can gain traction in the marketplace and eventually reduce costs via economies of scale. What we should avoid is becoming obsessed with attaining price-parity vs WinTel (the platform that offers us cheap-and-nasty, or cheap-and-nasty-on-steroids).
0spinboson
commented
5 years ago I don't agree with the simplistic "thing you're afraid of". Governments are claiming justification for their expanded reach partly on the great success criminal syndicates have had in expanding theirs. And the former will (these days) typically pay ransoms for expediency over ethics, thus feeding a vicious circle.
With respect, 'organized crime' has nothing to do with why the intelligence complex is asking Intel and AMD to add backdoors to the IME/PSP. The only reason they do, is because they want to be able to spy on citizens, corporations and foreign governments. They don't care about organized crime piggybacking (not least because they're not infrequently working together). The fact that lower govts are bothered by the same doesn't bother them either, "National security" trumps all.
andrewdavidwong
commented
5 years ago Reminder: This is an issue tracker, not a forum for discussion. The discussion thread for this issue is here. Please keep comments here on-topic and actionable for any developers who might work on this issue. Please see our issue tracker guidelines for more information.
Rspigler
commented
5 years ago In the discussion thread, @marmarek made clear that the Qubes team has no intention of porting to an open platform, and wants to keep focus on expanding the user base and hardware compatibility. However, he did state that if someone could provide enough funding to work on the port without sacrificing x86 support, Qubes would happily help with the port. I wonder if it would be possible to get an estimate on such a number?
I agree with @leo-lb - thanks to him and I, there are considerable funds available now available to developers willing to start working on this. @shawnanastasio has already started working on KVM support, although there is a long way to go. I would really like to start seeing more donations from the wider community. Spreading this on Twitter and Reddit I think would go along way. We probably need a website, with a secure donations page.
I am willing to match donations up to another 1 bitcoin. However, I am very concerned with what seems to be the consensus in choice of KVM over Xen. I understand that KVM has much better support for more architectures, but I have many security concerns. As I wrote before:
In the Qubes Architecture spec (Version 0.3), in the rational for choosing Xen over KVM, the authors mention how KVM relies on the Linux kernel to provide isolation [and therefore the isolation quality is limited to that of the Linux kernel]. They also argue that Xen is much easier to audit due to a smaller code base, and that the I/O emulator, networking code, and drivers can be moved out of Dom0, further minimizing TCB.
I would like this to be further discussed.
marmarek
commented
5 years ago I wonder if it would be possible to get an estimate on such a number?
See https://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-425749018
I would like this to be further discussed.
If host system would be sufficiently isolated (no external devices, no networking, possibly also no GUI), I would be less worried about KVM kernel part. Although it's still part of monolithic kernel there is very little direct interface to other kernel parts. We could have a separate kernel build with most features (not needed on host) disabled. It's still less ideal than Xen architecture, but acceptable. On the other hand, I/O emulator (aka qemu) is a bigger issue. There are multiple approaches to this issue:
tlaurion
commented
5 years ago For information, Raptor engineering confirmed they would be willing to provide free hardware for development.
tlaurion
commented
5 years ago @jpouellet : your thoughts in the discussion we had at OSFC on the current challenges of KVM approach would be of great importance here.
Your insights would clearly benefit the development of ppc64le support. :)P
tlaurion
commented
4 years ago From Timothy Pearson, from RaptorEngineering.
@jpouellet @marmarek : "KVM is already functioning reliably in production on ppc64le including PCI device isolation / passthrough. I can vouch for the integrity of the lower levels of that stack; we've even had people try generating random PCIe I/O using fairly high end equipment and the PHB blocked the accesses, which is more than can be said for many x86 systems.
I suspect most of the work is centered in the Qubes <--> KVM interfacing, which is not really architecture dependent or an area that I have any experience with. If Qubes could offer a bit more info on what they need from a ppc64 developer, I could offer more concrete suggestions and probably loop some folks in, but right now the requirements are so ill defined all I'd get is a "sure, that sounds interesting, what do they want help with? oh, they don't know, let me know when they have more info" kind of response."
marmarek
commented
4 years ago I've prepared more detailed list of tasks here:
-net socket,... can be used to connect two VM instances directly) (libvirt, qubes-core-admin) (P1, KVM)Tags explanation:
I've also added affected component names, may simplify further planning.
As you can see, indeed most tasks are specific to KVM, not necessary to the architecture change. This obviously assume the KVM features we want to use are actually supported on PPC64 - if not some of the tasks may have a hidden architecture-specific part.
Rspigler
commented
4 years ago This is awesome! Thanks everyone
stacktrust
commented
4 years ago This video on ppc64 virtualization may be of interest, includes Q&A from some people in this thread: https://www.platformsecuritysummit.com/2019/speaker/hunt/
tlaurion
commented
4 years ago 
liilac
commented
4 years ago As someone who would have interest in working on this, some things I think need to be resolved:
gmaxwell
commented
4 years ago If someone was willing to move forward on this on the basis of a bitcoin bounty and would be willing to set the conditions very clearly. I'd be willing to offer my services as a multisig escrow. Setting that up to everyone's satisfaction would be the most minor of difficulties-- one of the advantages of bitcoin is that it makes such things easy. They can be setup so that the funds are controlled by a 2 of 3 quorum between the offering party, the accepting party, and a escrow mediator.
This has the benefit that the mediator can't steal the funds without colluding with one of the other parties-- which should make it easier to choose the mediator. In the likely case that everyone is honest, the mediator doesn't need to do anything at all beyond agree in advance to participate.
or work of this nature, after the fact bounties are not realistic,
I fully agree. There are lots of reasons bounties don't work well for this sort of thing. ... but escrowing funds shouldn't be one of them.
tlaurion
commented
4 years ago @leo-lb @Rspigler ?
llebout
commented
4 years ago If someone wants proof of ownership for my funds, I can provide one through a private communication channel. I do not wish to link this github account to my Bitcoin funds publicly.
Otherwise, I also do not want to put my funds into a multisig scheme with people I do not know/or trust as they could keep me from using my funds, locking them up forever.
tlaurion
commented
4 years ago @liilac : what about doing things the other way around and proposing work for items you want to contribute to and funds you would require to accomplish them?
Other funding sources normally ask for an overview of tasks required for completion of the project (done) and then requires for tasks to have a committed amount of funds (WPD, work-person-day) required to do the work, on which sub-tasks, once approved by funder, gets the funds delivered upon proof of work.
Obviously, developers want to make sure they are going to be paid for work they are willing to contribute to, funders want to make sure work is gonna be done, and hardware providers want to see work commitment prior to give away hardware since what they want is code upstreamed.
Please everyone (@leo-lb, @Rspigler, @marmarek @stacktrust, @shawnanastasio) let's try to get out of this chicken and egg situation. :)
tlaurion
commented
4 years ago Rspigler
commented
4 years ago Hey all, glad to see this moving again! I am willing to get the ball moving and fund a multisig with my 1 bitcoin. I trust someone in this space who is public, especially @gmaxwell who has already offered.
My offer to match donations up to 1 more bitcoin still stands by the way. For those of you who have more of a following than I do, if you don't mind broadcasting this once we get an address set up, that could double our fund from 2 to 4 bitcoin, and I think really help our development efforts.
One thing I'm still concerned about (last time I'm mentioning it, I promise), is what seems like the consensus in choice of KVM over Xen for this port. IMO, this will be irreversibly adding magnitudes more code to the TCB. I would be willing to donate more to a Xen port.
llebout
commented
4 years ago If that is the consensus for all parties, I am ready to consider @gmaxwell as a multisig escrow.
As for KVM or Xen, I do have strong opinions for what would be ideal, but I think that goes a little beyond the scope of this issue.
I'd rather get something that works with KVM and that, at the same time, is way more portable.
The ideal situation would be to port sel4 and then port Qubes OS to sel4 but that's even harder. Also all the performance problems that may arise because Xen provides many features that help making the whole experience a lot smoother.
So no matter which one of the 3 it is, I agree to fund it.
llebout
commented
4 years ago @shawnanastasio How are things going with your work? Any blockers or expertise that you'd need about KVM? I can try poking around for some people.
QubesOS is the most secure operating system available, by far. However, it unfortunately only runs on the x86 instruction set, which runs on unauditable and insecure firmware. The Power Architecture is a much more secure ISA. Products like the Talos II (edit: and now much more affordable Blackbird) with the Power9 CPU are fully open, with auditable schematics, firmware, and software - and being able to run QubesOS on such devices would be a huge win for the infosec community.
There are various ways to achieve this compatibility, so I thought that this issue could be a way to track them/discuss.
1 - Xen could have a ppc64 port (Raptor Computing Systems has offered free hardware to incentivize) 2 - Using the seL4 microkernel (https://github.com/QubesOS/qubes-issues/issues/3894), which is already looking into supporting the Power Architecture 3 - Qubes' Hypervisor Abstraction Layer (HAL), which utlizes libvirt to support multiple hypervisors, yet currently only supports Xen, could be expanded to support KVM, to run on ppc64.
March 26, 2022: We are now all in agreement for Xen+Power (option 1).
Funds available as of May 7th, 2022: I (Robert Spigler) have 0.35 bitcoin & Blackbird Bundle @leo-lb has pledged 0.8 btc (need to confirm) Total 1.15 btc
@madscientist159 Has offered to do the Xen port for 2 btc (just Xen port; no Qubes integration yet)
Power Foundation has made a statement of support (https://twitter.com/OpenPOWERorg/status/1504112361975730186?s=20), but this needs to be clarified.
We will be moving from Github -> Gitlab for development. (https://gitlab.com/groups/xen-project/-/epics/6)
We have made a Mailing List and Matrix Room: qubes_port@lists.riseup.net; https://lists.riseup.net/www/info/qubes_port https://matrix.to/#/#qubes-port:matrix.org
We have now adopted this milestone approach for this Port: (done here)
Phase 1: 0.65BTC. Build tooling, minimal boot to serial console of a Xen kernel on a single core (no SMP, missing drivers, core locked at 100% power).
(Proposed) Phase 1.5: 0.65BTC (Pricing subject to change due to economic fluctuations): SMP, some driver integration (possible power state management?) required to get a usable system in preparation for Phase 2
I (Robert) donated 0.65 bitcoin out of my remaining 1 bitcoin bounty to fulfill the Phase 1 requirement. See here
@Rudd-O donated the entirety of his bounty (0.5 bitcoin) towards Phase 1.5. He no longer has any remaining pledge, and Phase 1.5 has 0.15 btc left to fulfill. See here
We are still waiting for @leo-lb to re-confirm his pledge.
Last updated May 7th, 2022
Details/History of Funding Below:
Please see the below chronological updates to funding:
In summary, we have a 3 bitcoin bounty,
and an additional 0.5 bitcoin remaining for matching funds(deadline passed with 0.5 matching funds filled out of 1 bitcoin matching funds offered - see here). The match offer expired on July 28th 2021.Details of the bounty are below:
@leo-lb paid @shawnanastasio 0.2 btc out of his 1 bitcoin bounty here: https://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-630972681
@Rspigler (me) paid Shawn 0.5 bitcoin out of his 1.5 bitcoin bounty here. I have also offered hardware (Blackbird mainboard and one 4 core Power9 CPU) for a developer who will use it towards this project. See post here.
@Rudd-O pledged 0.5 bitcoin here (has paid 0).
I (Robert) have a remaining 0.5 matching bitcoin offer that expires on July 28th 2021.Last updated: July 31st, 2022