Port Qubes to ppc64 [3 bitcoin bounty]

[Rspigler]

QubesOS is the most secure operating system available, by far. However, it unfortunately only runs on the x86 instruction set, which runs on unauditable and insecure firmware. The Power Architecture is a much more secure ISA. Products like the Talos II (edit: and now much more affordable Blackbird) with the Power9 CPU are fully open, with auditable schematics, firmware, and software - and being able to run QubesOS on such devices would be a huge win for the infosec community.

There are various ways to achieve this compatibility, so I thought that this issue could be a way to track them/discuss.

1 - Xen could have a ppc64 port (Raptor Computing Systems has offered free hardware to incentivize) 2 - Using the seL4 microkernel (https://github.com/QubesOS/qubes-issues/issues/3894), which is already looking into supporting the Power Architecture 3 - Qubes' Hypervisor Abstraction Layer (HAL), which utlizes libvirt to support multiple hypervisors, yet currently only supports Xen, could be expanded to support KVM, to run on ppc64.

March 26, 2022: We are now all in agreement for Xen+Power (option 1).

Funds available as of May 7th, 2022: I (Robert Spigler) have 0.35 bitcoin & Blackbird Bundle @leo-lb has pledged 0.8 btc (need to confirm) Total 1.15 btc

@madscientist159 Has offered to do the Xen port for 2 btc (just Xen port; no Qubes integration yet)

Power Foundation has made a statement of support (https://twitter.com/OpenPOWERorg/status/1504112361975730186?s=20), but this needs to be clarified.

We will be moving from Github -> Gitlab for development. (https://gitlab.com/groups/xen-project/-/epics/6)

We have made a Mailing List and Matrix Room: qubes_port@lists.riseup.net; https://lists.riseup.net/www/info/qubes_port https://matrix.to/#/#qubes-port:matrix.org

We have now adopted this milestone approach for this Port: (done here)

1. Phase 1: 0.65BTC. Build tooling, minimal boot to serial console of a Xen kernel on a single core (no SMP, missing drivers, core locked at 100% power).

2. (Proposed) Phase 1.5: 0.65BTC (Pricing subject to change due to economic fluctuations): SMP, some driver integration (possible power state management?) required to get a usable system in preparation for Phase 2

I (Robert) donated 0.65 bitcoin out of my remaining 1 bitcoin bounty to fulfill the Phase 1 requirement. See here

@Rudd-O donated the entirety of his bounty (0.5 bitcoin) towards Phase 1.5. He no longer has any remaining pledge, and Phase 1.5 has 0.15 btc left to fulfill. See here

We are still waiting for @leo-lb to re-confirm his pledge.

Last updated May 7th, 2022

---

Details/History of Funding Below:

Please see the below chronological updates to funding:

• 1
• 2
• 3

In summary, we have a 3 bitcoin bounty, and an additional 0.5 bitcoin remaining for matching funds (deadline passed with 0.5 matching funds filled out of 1 bitcoin matching funds offered - see here). The match offer expired on July 28th 2021.

Details of the bounty are below:

@leo-lb paid @shawnanastasio 0.2 btc out of his 1 bitcoin bounty here: https://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-630972681

@Rspigler (me) paid Shawn 0.5 bitcoin out of his 1.5 bitcoin bounty here. I have also offered hardware (Blackbird mainboard and one 4 core Power9 CPU) for a developer who will use it towards this project. See post here.

@Rudd-O pledged 0.5 bitcoin here (has paid 0).

I (Robert) have a remaining 0.5 matching bitcoin offer that expires on July 28th 2021.

Last updated: July 31st, 2022

[llebout]

Just for everyone's information: I think https://dornerworks.com/ has the expertise and is willing to port Xen to ppc64[le] but that is unlikely happening in the context of a bounty.

[shawnanastasio]

@shawnanastasio How are things going with your work? Any blockers or expertise that you'd need about KVM? I can try poking around for some people.

For vchan itself there are no blockers. There were a couple of kernel bugs I discovered, but I got the patches for them merged. At this point, all I need is some spare time to finish it up.

Aside from that, there's the issue of mapping guest pages from the host which qubes-gui-daemon requires. I haven't found a clean way to do this with KVM/QEMU, so if you or anybody else has some pointers it'd be greatly appreciated.

[marmarek]

Aside from that, there's the issue of mapping guest pages from the host which qubes-gui-daemon requires. I haven't found a clean way to do this with KVM/QEMU, so if you or anybody else has some pointers it'd be greatly appreciated.

In Qubes 4.1 we have abstraction for that. By default it's plugged into grant-tables mechanisms, but something else can be used instead. The most important point: we no longer require userspace on the host to map arbitrary page of the guest. I think it can be easily plugged with ivshmem.

[shawnanastasio]

The most important point: we no longer require userspace on the host to map arbitrary page of the guest. I think it can be easily plugged with ivshmem.

Sweet! So the guest-side software can be handed ivshmem pages directly? If I recall correctly, last time I checked the guest software would allocate its own memory and pass the physical address over vchan for the host to manually map. If this was fixed then it will make porting much easier.

[marmarek]

If this was fixed then it will make porting much easier.

Yes, that's exactly what was changed.

[marmarek]

Offtopic, but related interesting development: someone started working on RISC-V support in Xen: https://lists.xenproject.org/archives/html/xen-devel/2020-01/msg01731.html

[madscientist159]

it's a real shame no one is working on Xen support for POWER -- out of the two, POWER would produce usable results right now, today, on open firmware systems, whereas we have yet to see any open firmware RISC-V SBC (either on the market or planned) that would actually be able to use Xen in any meaningful way.

Both are open ISA systems, and we've repeatedly offered to provide hardware so that cost isn't a barrier. :disappointed:

[tlaurion]

@madscientist159 I asked the Q10 question here (Xen support plan from OpenPower?), on which @stacktrust mostly answers, just before Q11: https://www.platformsecuritysummit.com/2019/speaker/hunt/

There lies the Xen chicken/egg problem.

[madscientist159]

@tlaurion Yeah, so he's kind of glossing over the entire part where we were offering the systems for free to Xen developers, but I definitely get that IBM doesn't seem to want to commit developer effort right now. That being said, it looks like the Xen RISC-V initial discussion was started by community developer, so I just wonder what it would take to get things kicked off and some momentum going here...

[tlaurion]

@madscientist159 @Rspigler : Xen path would be preferable, of course.

You got the part at the end of first Q10 question, where Hunt gives contact information to make it happen with OpenPower collaboration?

@madscientist159: That would be awesome for you to reach out and update this ticket with gathered information.

Meanwhile, PowerPPC Xen support is dropped. Until revived. Maintainership problem. KVM way is preferred from all other parties meanwhile (availability over TCB security), it seems.

[Rspigler]

Has anyone reached out to Dimitrios Pendarakis yet?

I also think it could be useful to reach out to Bobby Eshleman and Andrew Cooper on that RISC-V support in Xen announcement thread. I agree completely with @madscientist159. Personally, I don't understand why developers would want to waste time porting to an open ISA that doesn't have any current or planned open implementations. And I'm not sure if the general public is very well aware of that fact, and the fact that RCS is offering free hardware, as well as that there is this existing bounty.

(My offer to match donations up to one more bitcoin still stands, and my donation will increase if we can land Xen support).

[tlaurion]

@Rspigler

Has anyone reached out to Dimitrios Pendarakis yet?

Not that i'm aware of. @madscientist159? @marmarek ?

Q: "Is there a possibility for joint efforts between OpenPower and developers to port Xen?" Answer: "You have to play by the rules when you do something like this..." "There is a guy named Dimitrios Pendarakis linked to this government project, [...]CTO of the security team for Power. Talk to this guy."

Author of this talk's referred paper Contact information: Dimitrios Pendarakis

@stacktrust brief historical recap of Xen support having been dropped by IBM and going the KVM way since then

[stacktrust]

Some context on Xen and Power:

• More than 50 IBM engineers were moved from Xen to KVM in the late 2000s. As a result, there is no Power expertise in the current Xen community. Given that IBM was previously involved in Xen, might there be Power developers who have prior Xen expertise? Step one is to find an individual who has, or has an interest in acquiring, Xen and Power knowledge.

• The previous Power PC port of Xen was done on old hardware and optimized for PV guests. A modern Xen port to Power would require HVM and PVH support, which would be new development to support guests and platform hardware .

• As for why RISC-V is receiving more attention, even though the virtualization spec is not finalized and hardware not available, one potential reason is the diversity of potential use cases, compared to the datacenter focus of Power. On the datacenter/cloud front, vendors are either doing custom CPUs (extending vertical integration that started with smartnics) based on Arm, or using the Power precedent to push incumbents for open-ish firmware + off-CPU roots of trust.

• The Ultravisor architecture has several benefits. If/when Ultravisor supports nesting, there may be a place for Xen L1 hypervisor (instead of KVM) running on Ultravisor L0 in firmware, looking forward to public availability of Ultravisor open-source code. Note that Xen is being modified to improve nesting on Hyper-V.

[ghost]

Just bumping this to let everyone know I am starting to work on porting Xen over to Power9 with the help of Raptor! Feel free to directly message me via Discord @ no#4880 or via email at davis123watts4@outlook.com!

[Rspigler]

@no-112 excellent! Thank you!

Have you let the Xen team know as well?

[Peter-Easton]

@no-112 That's awesome! Fingers crossed and thank you so much! Best of luck!

[tlaurion]

@madscientist159 can you confirm this?

[ghost]

@Rspigler Hey, I have been in contact with Olivier Lambert, from the Xen-ng. My next step is to send an email to Xen informing them of my plan, which will be done today.

@Peter-Easton Thank you!

[tlaurion]

@no-112 : If there is anywhere where we can see progress/participate, please let us know. Repository

[ghost]

@tlaurion Hey, you can follow my git repository at https://github.com/no-112/xen or you can join my Matrix room at #no-112-projects:matrix.org for updates and live progress!

[Rspigler]

Wonderful.

Along with @tlaurion, I would like to hear confirmation from @madscientist159, although I do see that he liked @no-112's announcement post, so I am not too concerned.

@leo-lb and I should get together, with input from the community and @no-112, to come up with possible sub-tasks/milestones for updates and partial funding.

@gmaxwell - Can we continue with set up of the bounty address?

As I promised earlier, since we chose Xen over KVM, I will be increasing my donation from 1 btc to 1.5. My offer to further match donations of 1 bitcoin is increased to a match of 1.5. Hopefully this brings in more supporters and at least one more developer. My match offer expires 4 months after we can really start advertising it (once the address is set up).

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

My donation and match have both increased from 1 to 1.5 bitcoin.

March 3rd 2020.

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEwsYOJ56G8Q1Wl3glNc4P5sIUGCMFAl5eCDgACgkQNc4P5sIU GCM9lBAAmnYKzHev993wIJZU4zxcNlCD1zfQ6JQ2NjoIzriOIoXWn+1iLcKIt//2 VzTCCN0q1Zx7yuGQatE9zbi92C1iWBM02cFRUmv7b2BK5QziDmH0OBmSySPbHN06 LhO2LgzTY06845YwLS3KJpgrv0y7Ihdpgltdx6p5XnM2P5SB5VYA43GNkhCBzcyR ZSp6tLGZ0xsmkvYVr1Q08TqRJTEwosAztoVUUSK/cba7LvzumWNIYIDPUElsAbc9 IounwiuVrZ7ny5PJ37P+cIAP+DM6K5J+Gg68jUk8yKY0zEy4wTdh6PX5vR+pkmII djFtHAE7KHfN6OOSK8Xnn2qPFvOqiccPloY6RpRIhoO2sWCNCFIiVkvxTOI3aBaz 5KoYLY0nIDYxeAx+JE5KT0ssw+BGFTq4N2uJznl3YModC18tszK+w+nTy+0iGgqw pSpj8K5Vzi+2XX1wdREWVngKSXa5226v5ONP4QICNAnWhlO+g7mlRlPqqtncYv0z pz5w9Rw3UiUvEUidhNXWsCbWSSZ1TlmHkLN4GJ5uQlp5oJMxVI7MI2HQ5+GmiwZ4 zMK9UNNefaDh9I5L1k6TWrfzw7iWJgWvyKqwV0U2uq02cQNFzLbMP4iNFcf1Lmyx Bd3xhSWrVAli92YhyDI8DlYfHBe4LgbYLJmH1uWdnkNHWv9jmaA= =ME/T -----END PGP SIGNATURE-----

[tlaurion]

@no-112 @madscientist159 , any updates somewhere else then https://github.com/no-112/xen-power9 which has no activity whatsoever?

[ghost]

Hey. There is zero activity due to me having some hardships due to this pandemic. As soon as I can get my financial situation sorted I am going to commence work. I will post an update here as soon as I start working on it!

On 16 Apr 2020, at 12:38 pm, tlaurion notifications@github.com wrote:



@no-112https://github.com/no-112 @madscientist159https://github.com/madscientist159 , any updates somewhere else then https://github.com/no-112/xen-power9 which has no activity whatsoever?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-614380660, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOM7OKP4OIMOSDG44FXKPT3RMZVSNANCNFSM4FVUJIKQ.

[tlaurion]

@no-112 money could be released on defined milestones if they were defined. The smaller the milestones, and consequent deliverables verified by community, the faster money is released upon task completion. That is how grant released money works in the open source world.

This project could involve community and other experts, following https://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-593809167 from @Rspigler:

@leo-lb and I should get together, with input from the community and @no-112, to come up with possible sub-tasks/milestones for updates and partial funding.

Please let's stop this chicken egg situation. Money is there. Everything goes slow for everyone. Those seem the best ingredients for collaboration torward a goal. There is a need for a plan, collaboration (not lonesome work here. You can do everything?) and money will be released. This is actually an opportunity, depending of how one sees things.

@no-112: What are your needs? What about we start the discussion? If you have expertise and there is money (and now time...); Where is the problem?

[tlaurion]

Have you reached out to Dimitrios Pendarakis ? (https://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-579186130)

Is there other people in the community willing to jump in?

Where can we read that exchange?

[llebout]

@tlaurion

Money is there.

I don't think it's even nearly enough to cover the actual porting costs. Most of the time is still voluntary here.

The reason this is stuck is because there's no one around competent enough to undergo such a port, and that's certainly not a single person's job.

The ARM port alone is 90k lines of code and they say it's been cleaned up and shortened.

[llebout]

I think the port to KVM is where the effort is best spent and it's much easier, even though I don't really care which hypervisor gets to power a working Qubes OS on POWER - Ultravisor mode with the Protected Execution Facility and it's future developments promises a bright future for an application like Qubes OS, which could make the usage of KVM on IBM POWER chips supporting Ultravisor and Protected Execution Facility (DD2.3+) even more secure than Xen on x86.

[shawnanastasio]

I agree. While I applaud efforts to take on a Xen port, I think that KVM is the most viable option at this point.

On that note, I will continue to work on libkvmchan, though development has slowed greatly recently due to a lack of time on my part. I'll provide updates here when I am able to make more progress.

[Jeeppler]

The ARM port was a lot of work and started before ARM 64 bit machines with hardware virtualization extensions were available. I think a Xen port to OpenPower is easier, because hardware is there and OpenPower has already hardware virtualization extensions.

[madscientist159]

@no-112 Sorry about the delay, the COVID19 situation has led to abnormal demands on my time. I think we had left it where I needed to get you a box -- it would probably be easiest to get you immediate remote access to one in our datacenter, or did you need the physical machine to get started?

[llebout]

@no-112 Sorry about the delay, the COVID19 situation has led to abnormal demands on my time. I think we had left it where I needed to get you a box -- it would probably be easiest to get you immediate remote access to one in our datacenter, or did you need the physical machine to get started?

I think that remote access to OpenBMC would make the remote experience just as good as the physical experience :-)

[shawnanastasio]

A quick update.

With https://github.com/shawnanastasio/libkvmchan/commit/dd4917c3eb1babce633a4b07f18581838b0312a3, the libvchan API is fully implemented in libkvmchan! I have tested open, close, packet read/write, and stream read/write in a small demo program and everything seems good so far.

Next, I'll write a qubes-core-vchan-kvm wrapper and attempt to use it with qubes utilities.

[tlaurion]

Time to revisit https://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-549986749

[llebout]

@shawnanastasio when the wrapper is done and is adequate for Qubes utilities I'll send you 0.2 BTC, is that OK with everyone? Does it amounts right compared to the rest of the work? I tend to think that's one of the more complicated parts.

[tlaurion]

@madscientist159 @Rspigler : Xen path would be preferable, of course.

You got the part at the end of first Q10 question, where Hunt gives contact information to make it happen with OpenPower collaboration?

@madscientist159: That would be awesome for you to reach out and update this ticket with gathered information.

Meanwhile, PowerPPC Xen support is dropped. Until revived. Maintainership problem. KVM way is preferred from all other parties meanwhile (availability over TCB security), it seems.

@madscientist159: have you reached out or followed up on this? Something to share before the energy is directed to KVM because XEN seems to complicated and no XEN devels showed up? Question were directed at you here. Now would be a good time to do some social magic from someone who is playing from inside or peripheral for things to reach outside. One OpenPower developer reaching out seems a good plan to gather people on reputation in field. Not the other way around. My 2 cents.

[tlaurion]

@marmarek @madscientist159 : We missed the starting point.

Mind following the "given advice" below to make this go forward and contact both Xen and PowerPC people on/off list in your direct inner circles following recommended advice?

Direct requote: Q: "Is there a possibility for joint efforts between OpenPower and developers to port Xen?" Answer: "You have to play by the rules when you do something like this..." "There is a guy named Dimitrios Pendarakis linked to this government project, [...]CTO of the security team for Power. Talk to this guy."

Author of this talk's referred paper Contact information: Dimitrios Pendarakis

[shawnanastasio]

A major update!

With the latest libkvmchan and the new qubes-core-vchan-kvm, qrexec now runs (mostly*) unmodified on ppc64le!

Still to be implemented is support for determining whether a vchan was closed, but in its current state everything seems sufficient for usage with qrexec. Below is a screenshot of qrexec executing a remote shell on a Fedora VM (left) from a Gentoo host (right). (Excuse the debug printfs :stuck_out_tongue_winking_eye:)

*A bodge in qrexec was add to sleep for a little bit before vchan client connection, since libkvmchan takes a little longer than libxenvchan to initialize new vchans. I'm still pondering the best way to solve this.

[shawnanastasio]

Just confirmed that I received 0.2BTC of the bounty from @leo-lb. Thanks!

[llebout]

I trust that @shawnanastasio will probably go on from here and continue polishing this implementation along with integration with Qubes utilities

I hope to reach a point where I can also contribute myself, I probably could try to peak in already.

Otherwise I can help with everything that touches build systems or packaging, "integrator" work.

[tlaurion]

@madscientist159 Room: Aprils PPC Projects : Porting Xen, Anarchy Chat, Porting Arch

Fri, Apr 17 2020
April left the room.

https://github.com/QubesOS/qubes-issues/issues/4318#issuecomment-622466631

[tlaurion]

@andrewcooper?

[cgchinicz]

Hi All,

Very inspiring this thread. After reading up to here I realized it will "Qubes" as a concept but not necessarily a port of the existing Qubes, running under Xen.

Following these lines, I'd like to add a suggestion: openpower architecture counts with an "Abstraction Layer" (OPAL), which may replace sys-net and sys-usb, at least for Linux/kernel based VMs which know to work with OPAL directly.

[shawnanastasio]

Hi All,

Very inspiring this thread. After reading up to here I realized it will "Qubes" as a concept but not necessarily a port of the existing Qubes, running under Xen.

It will still be very much Qubes - all Qubes utilities that facilitate creation of VMs and communication between them will still be in use, just under a different hypervisor.

Following these lines, I'd like to add a suggestion: openpower architecture counts with an "Abstraction Layer" (OPAL), which may replace sys-net and sys-usb, at least for Linux/kernel based VMs which know to work with OPAL directly.

I think you're confusing OPAL with a more traditional Power firmware/hypervisor. OPAL is a thin wrapper around platform-specific code for interrupt management, PCIe configuration, and BMC services like RTC, Serial, NOR Flash, etc. Notably, it is not a full operating system kernel nor does it handle anything like network or USB.

[cgchinicz]

Thanks for the feed-back and explanation about OPAL. I hope we'll be able to test it soon on ppc.

[hanetzer]

hello folks. neophyte here, maybe looking to toss my hat into the ring. As far as the thread knows, is there any public xen repo which is making progress towards this or is that still largely untouched? That being the case, are there any xen specific api/abi calls which need to be implemented per arch I should be aware of, or is most of that in the 'generic' code?

[llebout]

@hanetzer Xen is not being pursued, too big effort. Better to focus on getting Qubes OS with KVM working.

Progress happening slowly, we would really make use of a Qubes OS ppc64le development environment. For that I suggest building ppc64le ISOs for Qubes OS where not much will work but at least where you can hack on. I was intended to do that, but not getting anywhere also not very motivated for it right now. This is the repo one could start with: https://github.com/QubesOS/qubes-builder

[llebout]

It seems we have something for Qubes KVM dev env here: https://github.com/nrgaway/qubes-kvm-dev It is probably more x86_64 focused but should not be huge to adapt.

[hanetzer]

@hanetzer Xen is not being pursued, too big effort. Better to focus on getting Qubes OS with KVM working.

I may disagree on that. but thanks for letting me know I'm not going to be repeating/duplicating work.

[tlaurion]

Xen port is organizing slowly.

[tlaurion]

2 bitcoins = 63626.88 USD.