[Devices] Offer support for hardware-accelerated inference in Firecracker

[raduweiss]

Doing hardware-accelerated inference in a serverless environment is compelling use case.

However, adding straight up GPU passthrough means that microVM can't oversubscribe memory, and we need to add PCI emulation to Firecracker, which comes with a lot of extra complexity/attack surface.

The first step here will be to research the options and alternatives (e.g., GPU passthrough, or something else), and figure out the path forward.

Related issues: #849, #776.

[nlauchande]

I am very interested on this usecase.

[richardliaw]

+1, very interested in this use case. Any update on this? (I understand it's still in the research phase)

[zaharidichev]

@raduweiss is this something that anyone is working on atm? Is it still on the roadmap?

[ananos]

Hi @zaharidichev,

we have some thoughts on this [1], shared them earlier this year in the slack wοrkspace [2], but a chat is still pending I'm afraid. We have a rough proof-of-concept implementation on firecracker, based on the design principles of [1], which exhibits negligible overhead for image inference (jetson-inference backend, using tensorRT, tested on an NVIDIA jetson nano & a generic x86_64 machine with an RTX 2060 SUPER & another machine with a T4). We should be able to open-source the whole stack pretty soon. Feel free to drop us a line if you're interested in our early PoC.

Essentially, the idea is that we abstract away the hardware-specific operations via a slim runtime library/system, that supports any kind of backend (ranging from a simple CUDA/OpenCL function to a TensorFlow operation/app). Combined with a simple virtio frontend/backend implementation we are able to forward operations from a guest to the host/monitor, which in turn executes the actual "acceleratable" function on the hardware accelerator.

Another option (if latency is not critical to you) could be to use rCUDA, which we plan to try but haven't had the time yet...

BTW, @raduweiss we should plan to have that chat [2] at some point -- give us a shout when you are available!

cheers, Tassos

[1] https://blog.cloudkernels.net/posts/vaccel/ [2] https://firecracker-microvm.slack.com/archives/CDL3FUR8B/p1591093992140800

[raduweiss]

@ananos , yeah our bad, we totally dropped the ball here. Our apologies! I'll reply directly so we can talk.

[ananos]

Hi @zaharidichev, all

just wanted to share our blog post about our approach on the above: https://blog.cloudkernels.net/posts/vaccel_v2/

using nvidia-container-runtime & a docker image we've put together, you are able to run the jetson-inference image classification example from a Firecracker VM. You can find more info in the above post or @ https://vaccel.org. Of course, you can ping us, we will be more than happy to share how to try out vAccel on Firecracker.

cheers, Tassos

[amrragab8080]

Any update on the GPU support in Firecracker?

[raduweiss]

We’ve been thinking about / experimenting in this space in the last months, and we'll keep at it this year, but there’s no ETA for this feature right now. For maximum utility in a serverless platform paradigm [a], a single GPU hardware resource needs to be safely used by multiple microVMs, without trading off the other capabilities that Firecracker users like (e.g., CPU/memory oversubscription, fast snapshot-restore, or high mutation rate of the host’s microVMs). This is a pretty complex problem, and we’re still exploring our options.

As with the other larger features, as we approach what we think is a good design here, we'll post some form of RFC to get community feedabck.

We’d be happy to hear of any use cases to so we can factor them in – feel free to update this thread, or share them directly on our Slack [b]!

[a] https://github.com/firecracker-microvm/firecracker/blob/master/CHARTER.md [b] firecracker-microvm Slack workspace link

[pdames]

Any updates? My team is interested in running Ray on Firecracker, but the current lack of GPU support would erode the value of doing so.

[raduweiss]

Any updates? My team is interested in running Ray on Firecracker, but the current lack of GPU support would erode the value of doing so.

Sorry for not getting back here sooner, we were still working through our options. We've settled on implementing plain PCIe GPU passthrough, which comes at the cost of requiring micoVMs to start with the their full memory mapped, will probably negate the advantages of using snapshot-restore, and requires the full GPU to be attached to a microVM - all things we wanted to see if we could improve upon, but we didn't find way that upholds all our tenets.

We will want to get broad feedback from the community here on how to actually present this as a feature (we'll start a discussion in the following weeks). Given the trade-offs above, we will consider building a separate Firecracker mode or Firecracker variant, or something along those lines.

[zvonkok]

@raduweiss I am leading the enablement of GPUs and other NV accelerators on Kata containers. I was trying to use the Slack Invite in the README.md but it is invalid.

What would be the best way to get into the loop on the PCIe implementation in firecracker? I fixed and I'm currently fixing several other issues (BAR sizes, MDEV support, ...) in Kata's PCIe (QEMU) implementation.

Would be nice if I could get hands-on with some pre-released artifacts to start testing on our side.

[raduweiss]

Hi @zvonkok . We've re-prioritized our roadmap, and for 2022 we're not pursuing the Firecracker PCIe implementation / GPU passthrough work anymore.

[DemiMarie]

@raduweiss: what would be needed for a “good” solution? Could https://libvf.io be helpful?

[mmcclean-aws]

Any plans to support Inferentia and Trainium based instances ? They expose the accelerators via PCI to the OS but I see PCI support is not planned for firecracker. See docs for more details on the devices exposed.

[kalyazin]

Hi @mmcclean-aws . Like discussed offline, an immediate obstacle for supporting Inferentia and Trainium instances is that they are virtualised (as opposed to bare metal), so Firecracker can't run on them, because AWS doesn't support nested virtualisation. Besides that, since Inf2 has 12 accelerators, and each accelerator can only be used in a single-tenant manner, the instance can carry up to 12 microVMs at the same time, which does not allow to extract oversubscription that is a key Firecracker's benefit. The only potential benefit (if/when bare metal Inf2* instances are available) could be shorter VM startup time if an instance needs to be partitioned dynamically.

[kalyazin]

Hi @peterdelevoryas . What is your specific motivation for moving off Qemu? Is that merely because of the Rust safety features? Firecracker was developed with CPU workloads in mind, and design decisions have been often driven by that (eg using MMIO virtio transport vs PCI). Being a live product, we may find it possible to reconsider those if sufficiently compelling reasons for doing so arise.

[peterdelevoryas]

Hi @peterdelevoryas . What is your specific motivation for moving off Qemu? Is that merely because of the Rust safety features? Firecracker was developed with CPU workloads in mind, and design decisions have been often driven by that (eg using MMIO virtio transport vs PCI). Being a live product, we may find it possible to reconsider those if sufficiently compelling reasons for doing so arise.

I don’t have any super strong reasons to migrate off QEMU, I just like the idea of something stripped down and written in Rust, and the fact it’s completely open source, free, and run in production for real aws workloads. I just don’t want to live with QEMU forever, even if just for the fact that I don’t enjoy mailing list development.

Edit: I noticed cloud-hypervisor, and realize that resolves this for me! nvm. I actually agree, firecracker should keep doing non-passthrough stuff, cloud-hypervisor makes more sense for passthrough use cases unless you can manage to integrate PCI passthrough into the microvm environment somehow.

[DemiMarie]

A few comments:

• https://libvf.io provides support for GPU virtualization. However, multi-tenant GPU virtualization requires trusting the proprietary vendor hardware & firmware to do its job. @raduweiss: Does Amazon consider this sufficent protection?
• With PCI passthrough, Amazon can avoid most of the security risks by using a custom board design where the GPU’s SPI flash is write-protected by hardware the GPU has no control of. Passthrough with stock hardware is much riskier.
• Memory oversubscription is possible by emulating a nested IOMMU. I’m not sure if Firecracker’s developers are interested in doing so given the performance penalties.

[mmcclean-aws]

Thanks. Does that mean that PCI passthrough should work for alternative devices (e.g. Trainium and Inferentia) that expose themselves in /dev ?

[DemiMarie]

It should work for almost any PCI device. Whether it is secure is another matter. That depends entirely on choosing a safe device and your ability to prevent early boot DMA attacks and unintended persistence via e.g. on-device flash storage.

[jayavanth]

Any updates on GPU support in your roadmap for 2023/2024?

[xmarcalx]

Hi @jayavanth ,

Thanks for your question. No we are not planning any GPU support in Firecracker at the moment. Once we will consider again this task we will add in our GitHub roadmap, which we are in the following weeks we are going to update and bring up to speed soon.

[fighterhit]

Hi @jayavanth ,

Thanks for your question. No we are not planning any GPU support in Firecracker at the moment. Once we will consider again this task we will add in our GitHub roadmap, which we are in the following weeks we are going to update and bring up to speed soon.

Hi @xmarcalx , in the current era of rapid AI development, GPU support is very important. I hope the team can seriously consider this feature. Thanks!