Support for ARM

[ghost]

Could you make a distribution for ARM?

[olivierlambert]

Hi @Kashcheyeu

To be able able to run on ARM, we'll need (non-exhaustive and non-ordered list):

• Our own Xen build on ARM (I think that wouldn't be a problem, just need some time to cross-compile it or have access to some ARM hardware)
• Build XAPI on ARM (should work but with a lot of weird stuff to fix)
• Build all the glue around (some features aren't available on ARM I suppose), eg xenops and all the x86 specifics
• Having a dom0 CentOS working on ARM
• And also the target. ARM is a very fragmented ecosystem. What system we should target?

Obviously, your help assessing those questions is more than welcome.

[ghost]

Perhaps Fedora will be better than CentOS? It's newer. As for hardware, I have A20-OLinuXino-LIME2 with Allwinner A20/T2 dual core Cortex-A7 processor, it's worth focusing on it.

[olivierlambert]

XCP-ng is based on CentOS, not Fedora. Using another dom0 OS will require far more work.

Feel free to experiment and report :)

[nagilum99]

Since XCP-ng ist based on CH - it can't be an option, since compatibility and manpower would be... Also: This kind of solution is not made to support cutting edge technology, it's made to run reliable/stable on solid platforms. Everything that is "very new" contains much bigger risks of problems and for reliability you want mature systems that (usually) alredy got a bunch of fixes and are prooved to be stable.

Edit: And an ARM A7 DualCore platform is very likely a very small niche for this kind of solution. Ratio won't be good.

[Josua-SR]

Throwing in my thoughts here since I was looking to experiment with xen on arm:

While armv7 platforms were somewhat limited in virtualization and ram - requiring explicit presence of optional virtualization extensions andor lpae - armv8 (aarch64) likely solves those --> So I would highly suggest targeting explicitly armv8-a.

It can currently be observed that some vendors are pushing for standardization in this space - uefi and acpi solutions are either in work or already available from some vendors of arm64 platforms in order to reduce the individual workload for every single arm board to be supported by every single distribution.

CentOS afaik runs on some armv8 devices - so imo that would be the best place to start efforts - and hope for the wider community to make sure interesting / powerful devices will eventually be supported by centos.

[nagilum99]

As you write ARMv8 could be the startpoint where it's interesting: Several Raspberry Pi models are ARMv8 based and Version 4 is even available with 8 GB of RAM for < 100 bucks, which could be interesting for someone trying to dev/test it? I know some things like media decoders are special but that would be stuff that won't bother here anyways - at least not in the first row to have a test platform for virtualization.

Edit: The mentioned A20-OLinuXino-LIME2 is only v7 and much weaker, IMHO not worth focussing on it.

[kmodexc]

I want to install xcp-ng on Raspberry. There is a version of Xen for the Raspberry but they had some problems(https://xenproject.org/2020/09/29/xen-on-raspberry-pi-4-adventures/) Does the changes they made on xen need to be applied on this project seperatly? Do you think this could be done with a reasonable amount of work?

[nagilum99]

You're mixing up XCP-ng (which is a distribution/an OS) and Xen, which is just the hypervisor.

The Image can't boot on Raspberry and you need to recompile (and probably adjust/patch) the whole sources.

[olivierlambert]

We did an initial estimation of the amount of work needed for one developer. It's around 36 months. But it's a rough estimate, might be bigger than this (or if we are lucky a bit lower but I'm not confident about this).

[woutervddn]

I saw the post of Xen being available on RPI. And although I haven't tried it, it did trigger me. Today I also spotted a video on youtube showing ESXI running on a Raspberry Pi. I see potential for this, but also see this as a huge amount of work. Anyway, here are some of my thoughts.

For real world (production) usage I don't suppose ARM will take over (a majority of deployments) from x86 anytime soon. That being said, datacenters like OVH are offering ARM based servers these days. And while they are mainly focused for single purpose deployments, I suppose it might not be that far fetched to expect some worthy ARM servers in a couple of years. Apple's own silicon being based on ARM, while not directly related to servers, is a strong sign that more companies might be developing arm based solutions (the open compute project being one of them).

For educational purposes on the other hand, this could be major today. With the release of the Raspberry Pi Compute 4 module we now have fairly decent specs on Single Board Computers for cheap. The people over at Turing Pi are already developing new hardware to support the new compute module and people like Jeff Geerling are making it really easy to set up tools like K8s.

The price point of the Turing Pi cluster makes it so that colleges and universities suddenly can buy multiple computing nodes for the price of a single desktop. In the formfactor of a single desktop even. A classroom where every student has it's own set of multiple computing nodes is suddenly possible. I can see schools showing of the difference between traditional VMs and things like K8s by actually letting every student set up its own system. This to me looks like a great opportunity to increase adoption of XCP-NG. Same goes for industry veterans: Contemplating a switch to XCP-NG, for about 1K $/€ you get a testsystem where you can experiment on multiple nodes to verify that XCP-NG has all the features to replace your existing insert_competitor_here installation.

[olivierlambert]

No need to convince me. However, it's all about deciding the priority. Money doesn't grow on trees, and you have to pay people to do that work. Obviously, bigger the success will continue with XCP-ng, bigger the chances to actually have the resources to make it real.

If we want to target server virt, having a PowerPC port might be more interesting on "short term" (still years of work).

[nagilum99]

Currently I see 2 ways: 1st: Vates can develop this, but industry needs to have wide interest on (paying for) such a product - which I don't see big enough for many years (I suppose the support, paying the developers, is mostly paid by companys with powerful X86 servers, which won't be replaced by ARM, in big enough numbers, by the next decade - if ever) 2nd: Commuity mainly needs to do it on their own and free of charge, but as Oliver said: It needs a lot of manpower to develop and test things until a prodiction ready release (and continuing support).

[olivierlambert]

Having a real traction from potential customer will be great indeed. However, I consent to spend some time to study "what's left to do" more in depth so we can at least identify pain points and how long we estimate. Probably an issue with a bullet list of each problem and estimated difficulty. This way, when we want to start (or someone else), there's already a plan :+1:

[nodesocket]

I'd love support for Rasberry Pi 4B's. I'd buy the 8 GB models to maximize memory.

[brainchild0]

Raspberry would be a sensible target due to widespread adoption, but the space is occupied largely by hobbyists, who are unlikely offer much commercial incentive.

For my own interest, I would run a XCP distribution on a RP4, especially if it has the same turnkey provisioning simplicity as current distributions.

As many OS distributions run on the Pi Zero, which has only 512MB of RAM, and on similarly limited earlier Raspberry models, it would seem feasible to run multiple guests on a hypervisor having as little as 4GB.

Would the new Windows distributions for ARM64 possibly run under a Xen hypervisor? It would be a big development not only to run Windows on a Pi, but also concurrent with other operating systems.

[Rexo81]

I have a mac mini M1 and I want to install xcp-ng with bootable disc but it doesn't boot , somebody can help

[nagilum99]

How about getting the proper hardware achitecture? ;-) M1 are not X86, it's ARM. That's why you also can't install Windows via bootcamp anymore.

[Rexo81]

is there another way to install or do we have an xcp iso for arm version ? thx in advance @nagilum99

[nagilum99]

There might be a chance through MacOS and VMware (or such), but not sure how good that would be running and absolutely won't recommend for production use. tl;dr: You need to wait for a proper ARM ISO and hope it'll work with Macs M1 architecture as it depends on drivers etc. It's a server OS, so not as 'well' equipped as Ubuntu etc. are.

[Rexo81]

can I install proxmox in place of ? yeah it's for production use . My plan was install ubuntu arm server and then install xcp on it , is that a good idea, what you think about ? thx @nagilum99

[nagilum99]

I'd say you can't instally anything except for MacOS or some Linux distro on it. In any way: You need to look for ARM ISOs. XCP is an OS, not an addon or software, so... don't. You coul install the Xen or KVM things and however manage them. That mac mini is IMHO the wrong decision for a server. No RAS and no wide-spread hardware that is being used for such.

[brainchild0]

I think the discussion has become off topic. Perhaps further comments following the same course may be placed elsewhere.

[olivierlambert]

Likely on https://xcp-ng.org/forum if you want to discuss about XCP-ng and Arm support in the future ;)

Closing the discussion here, for the most relevant comment, see https://github.com/xcp-ng/xcp/issues/413#issuecomment-713761130