AMD IOMMU validation using PCILeech/PCIScreamer

[pietrushnic]

Hi @ufrisk, I'm enabling IOMMU in firmware (coreboot) for PC Engines apu2 router and would like to perform sanity checks if IOMMU really protects against DMA attacks. The device has 3 mPCIe ports and USB XHCI ports. I wonder what would be the best hardware to target this device. Typically users run Linux distros (Debian) or FreeBSD (pfSense) on those devices. There is no UEFI support right now.

Have you got any ideas about what kind of effort this project would require? Maybe it would be easier to use modified kernel drivers to validated IOMMU isolation?

[ufrisk]

Hi,

I have one myself running pfSense.

Regardless of other techniques to validate that it is working I believe actually validating it with real hardware would be a good thing. As far as the APU2 goes I believe some of the slots are mSATA only, but one or two of the slots should be mPCIe.

I would recommend FPGA hardware to do this. Possibly the SP605 or AC701 hardware setup together with an adapter from mPCIe to PCIe. I would not recommend the USB3380 - which is less capable and the PCIeScreamer which won't be 100% stable for your requirements.

[pietrushnic]

@ufrisk , that's great you are happy apu2 user :) Yes, this board have 1x mSATA and 2x mPCIe.

AC701 looks much better in terms of transfer speed, but pricing is twice of SP605. By adapter you mean something like this or there are some solutions that you already tested?

Do you think above setup would work fine with other solutions - I'm thinking also about i.MX6 based platforms with mPCIe like HumminbBoard Edge ?

[ufrisk]

main advantage for the AC701 over the SP605 are:

• pcie gen2 (sp605: gen1)
• more lanes x4 (only x2 in my verilog design) (sp605: x1)
• more modern development environment (vivado vs ISE)

For your test cases I don't know if it will make a difference - probably not?

I use these adapters: https://www.ebay.co.uk/itm/Mini-PCI-E-PCI-express-to-PCI-E-1X-Card-adapter-Riser-Extender-USB-port/271602546763 https://www.ebay.co.uk/itm/USB-3-0-PCI-E-1x-to-16x-Extender-Riser-Card-Adapter-Cable-Bitcoin-GPU-Mining/222791532380

If you wish I can check it out to see if I get it to work both with existing firmware (old coreboot?) and FreeBSD/pfSense with my HW. I'm off to DEF CON for a week now though, but if you so should wish I could take a quick look at it next week.

---

About the hummingbird - I have no clue about it, it's not x86/x64? I have no clue about whether it has an IOMMU or not. If it has it could be interesting to take a look at as well I guess. I myself haven't looked at non Intel x64 hardware yet.

[pietrushnic]

@ufrisk it would be great if you can take a look at this before I invest in hardware on my side. I can also craft some firmware if you would need some custom IOMMU configuration. We will have release v4.8.0.3 this week here: https://pcengines.github.io/ which will contain IOMMU enabled. I also prepare blog post which is WIP here.

Hummingboard Edge is NXP i.MX6 ARM Cortex-A9. According to what I read ARM TrustZone does memory mapping and interrupt handling. There are companies selling products based on the virtualization capabilities of that SoC for automotive. Please check this - I may be wrong here, but there is definitely need for isolation and if it is missing this would be an issue for the automotive industry.

We also believe that exploring non-Intel platforms would be interesting - maybe we can find something interesting there.

[ufrisk]

Interesting and awesome work! I'll definitely take a look once I'm back home from DEF CON. It's nice that I already have the hardware :)

About the HummingBoard Edge and other similar products, it would be nice if someone could look into them, but that person would have to be someone else than me unfortunately. PCIleech is just a hobby project of mine. And while I already put a lot of time into PCILeech for free I'm not too keen on also starting purchasing a lot of different types of specific hardware with my own taxed money just so that I can pentest them for free.

[pietrushnic]

@ufrisk we have HummingBoard Edge in our office so we can test that and see what is going on, but first we want to learn how we can use it and maybe if we will gain enough expertise help spread the word about PCILeech or even contribute.

So I'm waiting for feedback and if there will be some decision about which hardware we would like jump in and report back about discoveries.

[ufrisk]

I finally got around to do some testing.

The PCIeScreamer is mostly fine if you power it on with external power (via the barell jack) before you power on the APU2 if you only use an adapter like this: https://www.ebay.co.uk/itm/Mini-PCI-E-PCI-express-to-PCI-E-1X-Card-adapter-Riser-Extender-USB-port/271602546763 It may still be a bit unstable to get the initial connection, but when it got it it's quite stable.

I did not manage to get the SP605 to work at all. I don't know if it has to be with it only being PCIe gen1, or if there was some other error (haven't used it in a while). The AC701 FPGA board I have worked flawlessly.

My coreboot (PCEngines apu2 coreboot build 20160307) was totally vulnerable to all things DMA, but that was only expected. I did not find the v4.8.0.3 binaries so I have not tested it.

Please let me know if you need any additional help. Like in doing a quick check-up on v4.8.0.3. Otherwise I think the PCIeScreamer should be fine for your use case.

[pietrushnic]

Hi @ufrisk, thanks for checking out PC Engines apu2.

I already received adapter. If I understand correctly I should order this version of PCeScreamer, please confirm.

v4.8.0.3 was not yet published on PC Engines servers, but you can download it from 3mdeb cloud here - this link may expire in future.

IIUC to get correct support not only firmware should be configured but also OS have to utilize IOMMU? Or do you think IOMMU should be configured correctly out of the box? I'm trying to learn best practice here since right now we probably operate on AMD defaults, I also know that some combinations of IOMMU flags are not exactly stable. If you have any references worth diving in please let me know.

[ufrisk]

Thanks,

I'll try to check it out tomorrow. File downloaded.

That version of the PCIeScreamer should be fine, or order it without the JTAG programming cable if you already have one...

As with regards to the IOMMU, coreboot itself should ideally configure it to shield itself from DMA attacks very early boot. Ideally before PCIe is initialized.

Also you are correct in that the OS needs to use it as well in order to be fully protected. But that is really the responsibility of the OS. Leaving the IOMMU active when handing over to the OS, if the OS is unaware or doesn't expect it, is probably a very bad idea.

I haven't really into actually using the IOMMU, or any best practices around it. I've pretty much only focused on the attacking side of things so I doubt I'll be of much help here.