ufrisk / pcileech

Direct Memory Access (DMA) Attack Software
GNU Affero General Public License v3.0
4.87k stars 718 forks source link

Cannot read past first 3 gigabytes of memory (Screamer M.2 vs. Windows 10 18363 Target) #126

Closed michaelweber closed 4 years ago

michaelweber commented 4 years ago

Hi @ufrisk!

Apologies if this is a duplicate issue (I've been reading through the closed issues and it seems like this occasionally happens, but most of the threads seem to end with the users just seeing it magically work again).

I'm using a Screamer M.2 with the latest official firmware from pcileech-fpga to try to read from a Windows 10 Build 18363 machine. It's an older AMD machine (AMD FX 6300, nothing Ryzen tier), which I know isn't what your test hardware is, but I've disabled IOMMU + SVM and windows is telling me that all device security measures are unavailable so I think any hardware protections are out of the way.

What's weird is that I can read the first 3 gigs of memory fine using pcileech - but the moment I got past that point - everything stops working. If I try to use MemProcFS or invoke the kmdload functionality this also breaks everything.

An example successful run if I shut everything down, boot up, and then run from the lock screen:

C:\projects\hardware-dma>pcileech.exe -device fpga -v -vv -max 0xbfffffff probe

DEVICE: FPGA: ScreamerM2 PCIe gen2 x1 [300,0,500] [v4.3,0500]

----- FPGA DEVICE CONFIG REGISTERS: CORE-READ-ONLY     SIZE: 34 BYTES -----
0000    89 ab 00 00 22 00 00 00  04 03 04 00 00 00 00 00   ...."...........
0010    4b dc b2 24 01 00 00 00  4d dc b2 24 01 00 00 00   K..$....M..$....
0020    00 00                                              ..

----- FPGA DEVICE CONFIG REGISTERS: CORE-READ-WRITE    SIZE: 30 BYTES -----
0000    cd ef 04 00 1e 00 00 00  a0 86 01 00 00 00 00 00   ................
0010    ee 00 07 00 ee 00 66 06  02 1c 00 00 7f 00         ......f..... .

----- FPGA DEVICE CONFIG REGISTERS: PCIE-READ-ONLY     SIZE: 48 BYTES -----
0000    01 23 00 00 30 00 00 00  05 00 16 08 7c 00 00 00   .#..0.......|...
0010    00 00 00 00 00 00 00 00  00 28 00 00 00 00 40 00   .........(....@.
0020    12 10 00 00 1e 7f 00 00  00 00 ff f9 00 00 00 00   ..... ..........

----- FPGA DEVICE CONFIG REGISTERS: PCIE-READ-WRITE    SIZE: 84 BYTES -----
0000    45 67 00 f0 54 00 00 00  35 0a 00 01 01 00 00 00   Eg..T...5.......
0010    00 00 00 00 ff fd 48 00  00 00 00 0e 00 00 00 00   ......H.........
0020    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0030    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0040    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0050    00 00 00 00                                        ....

----- PCIe CORE Dynamic Reconfiguration Port (DRP)  SIZE: 0x100 BYTES -----
0000    00 00 00 01 00 02 00 00  00 00 00 00 00 00 f0 00   ................
0010    ff ff 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0020    00 00 00 00 00 00 00 00  00 00 00 40 00 00 00 00   ...........@....
0030    00 00 09 02 30 00 22 7f  02 00 01 00 00 03 11 0c   ....0." ........
0040    02 a1 00 43 01 00 1f fd  7f ff 09 ff 01 20 01 48   ...C.... .... .H
0050    00 05 01 60 11 9c 00 00  00 00 00 00 00 00 00 00   ...`............
0060    00 00 10 60 00 02 40 21  00 40 3d 48 00 23 00 00   ...`..@!.@=H.#..
0070    00 00 00 00 00 00 00 00  00 00 00 00 00 15 00 01   ................
0080    00 01 00 00 00 01 00 00  00 01 00 00 00 01 00 00   ................
0090    00 01 00 00 00 01 00 00  00 00 00 00 00 00 00 00   ................
00a0    00 00 00 00 00 00 00 00  00 02 00 00 12 34 10 18   .............4..
00b0    00 0b 00 01 00 11 00 00  00 00 00 00 00 01 00 00   ................
00c0    00 28 00 41 ff ff ff ff  00 e0 00 00 80 08 00 22   .(.A..........."
00d0    07 ff 03 52 02 48 00 08  00 40 0e 84 fa ac 00 00   ...R.H...@......
00e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................

----- PCIe CONFIGURATION SPACE (no user set values) SIZE: 0x200 BYTES -----
0000    ee 10 66 06 00 00 10 00  02 00 00 02 10 00 00 00   ..f.............
0010    00 00 10 fe 00 00 00 00  00 00 00 00 00 00 00 00   ................
0020    00 00 00 00 00 00 00 00  00 00 00 00 ee 10 07 00   ................
0030    00 00 00 00 40 00 00 00  00 00 00 00 0a 01 00 00   ....@...........
0040    01 48 03 78 08 00 00 00  05 60 80 00 00 00 00 00   .H.x.....`......
0050    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0060    10 00 02 00 e2 8f 00 00  00 28 00 00 12 f4 03 00   .........(......
0070    40 00 12 10 00 00 00 00  00 00 00 00 00 00 00 00   @...............
0080    00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00   ................
0090    02 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00a0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00b0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00c0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00d0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00f0    00 00 00 00 00 00 00 00  00 00 00 00 03 00 c1 10   ................
0100    00 00 00 00 35 0a 00 01  01 00 00 00 00 00 00 00   ....5...........
0110    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0120    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0130    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0140    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0150    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0160    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0170    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0180    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0190    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01a0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01b0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01c0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01d0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................

Successfully loaded LeechCore v1.7.0 Device 3
 Memory Map:
 START              END               #PAGES
 0000000000000000 - 00000000bfffffff  000c0000

 Current Action: Probing Memory
 Access Mode:    Normal
 Progress:       3072 / 3072 (100%)
 Speed:          384 MB/s
 Address:        0x00000000C0000000
 Pages read:     786432 / 786432 (100%)
 Pages failed:   0 (0%)
Memory Probe: Completed.

All the reads succeed. This can be run over and over again and it won't have any problems. But if I try to read a single page beyond the 3GB threshold:

C:\projects\hardware-dma>pcileech.exe -device fpga -v -vv -min 0xc0000000 -max 0xc0001000 probe

DEVICE: FPGA: ScreamerM2 PCIe gen2 x1 [300,0,500] [v4.3,0500]

----- FPGA DEVICE CONFIG REGISTERS: CORE-READ-ONLY     SIZE: 34 BYTES -----
0000    89 ab 00 00 22 00 00 00  04 03 04 00 00 00 00 00   ...."...........
0010    b0 81 8f b3 01 00 00 00  b2 81 8f b3 01 00 00 00   ................
0020    00 00                                              ..

----- FPGA DEVICE CONFIG REGISTERS: CORE-READ-WRITE    SIZE: 30 BYTES -----
0000    cd ef 04 00 1e 00 00 00  a0 86 01 00 00 00 00 00   ................
0010    ee 00 07 00 ee 00 66 06  02 1c 00 00 7f 00         ......f..... .

----- FPGA DEVICE CONFIG REGISTERS: PCIE-READ-ONLY     SIZE: 48 BYTES -----
0000    01 23 00 00 30 00 00 00  05 00 16 08 7c 00 00 00   .#..0.......|...
0010    00 00 00 00 00 00 00 00  00 28 00 00 00 00 40 00   .........(....@.
0020    12 10 00 00 1e 7f 00 00  00 00 ff f9 00 00 00 00   ..... ..........

----- FPGA DEVICE CONFIG REGISTERS: PCIE-READ-WRITE    SIZE: 84 BYTES -----
0000    45 67 00 f0 54 00 00 00  35 0a 00 01 01 00 00 00   Eg..T...5.......
0010    00 00 00 00 ff fd 48 00  00 00 00 0e 00 00 00 00   ......H.........
0020    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0030    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0040    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0050    00 00 00 00                                        ....

----- PCIe CORE Dynamic Reconfiguration Port (DRP)  SIZE: 0x100 BYTES -----
0000    00 00 00 01 00 02 00 00  00 00 00 00 00 00 f0 00   ................
0010    ff ff 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0020    00 00 00 00 00 00 00 00  00 00 00 40 00 00 00 00   ...........@....
0030    00 00 09 02 30 00 22 7f  02 00 01 00 00 03 11 0c   ....0." ........
0040    02 a1 00 43 01 00 1f fd  7f ff 09 ff 01 20 01 48   ...C.... .... .H
0050    00 05 01 60 11 9c 00 00  00 00 00 00 00 00 00 00   ...`............
0060    00 00 10 60 00 02 40 21  00 40 3d 48 00 23 00 00   ...`..@!.@=H.#..
0070    00 00 00 00 00 00 00 00  00 00 00 00 00 15 00 01   ................
0080    00 01 00 00 00 01 00 00  00 01 00 00 00 01 00 00   ................
0090    00 01 00 00 00 01 00 00  00 00 00 00 00 00 00 00   ................
00a0    00 00 00 00 00 00 00 00  00 02 00 00 12 34 10 18   .............4..
00b0    00 0b 00 01 00 11 00 00  00 00 00 00 00 01 00 00   ................
00c0    00 28 00 41 ff ff ff ff  00 e0 00 00 80 08 00 22   .(.A..........."
00d0    07 ff 03 52 02 48 00 08  00 40 0e 84 fa ac 00 00   ...R.H...@......
00e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................

----- PCIe CONFIGURATION SPACE (no user set values) SIZE: 0x200 BYTES -----
0000    00 00 00 00 00 00 10 00  02 00 00 02 10 00 00 00   ................
0010    00 00 10 fe 00 00 00 00  00 00 00 00 00 00 00 00   ................
0020    00 00 00 00 00 00 00 00  00 00 00 00 ee 10 07 00   ................
0030    00 00 00 00 40 00 00 00  00 00 00 00 0a 01 00 00   ....@...........
0040    01 48 03 78 08 00 00 00  05 60 80 00 00 00 00 00   .H.x.....`......
0050    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0060    10 00 02 00 e2 8f 00 00  00 28 00 00 12 f4 03 00   .........(......
0070    40 00 12 10 00 00 00 00  00 00 00 00 00 00 00 00   @...............
0080    00 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00   ................
0090    02 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00a0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00b0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00c0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00d0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
00f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0100    03 00 c1 10 35 0a 00 01  01 00 00 00 00 00 00 00   ....5...........
0110    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0120    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0130    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0140    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0150    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0160    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0170    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0180    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
0190    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01a0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01b0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01c0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01d0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01e0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
01f0    00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................

Successfully loaded LeechCore v1.7.0 Device 3
 Memory Map:
 START              END               #PAGES

 Current Action: Probing Memory
 Access Mode:    Normal
 Progress:       0 / 0 (100%)
 Speed:          4 kB/s
 Address:        0x00000000C0001000
 Pages read:     0 / 1 (0%)
 Pages failed:   1 (100%)
Memory Probe: Completed.

No dice. And now if I try to re-run the previous probe all the reads that had previously succeeded will now fail. I see similar failures if I try running the kmdload functions or just running MemProcFS.

C:\projects\hardware-dma>pcileech.exe -device fpga -v -vv -max 0xbfffffff probe

DEVICE: FPGA: ScreamerM2 PCIe gen2 x1 [300,0,500] [v4.3,0500]

...<snip>...

Successfully loaded LeechCore v1.7.0 Device 3
 Memory Map:
 START              END               #PAGES

 Current Action: Probing Memory
 Access Mode:    Normal
 Progress:       3072 / 3072 (100%)
 Speed:          341 MB/s
 Address:        0x00000000C0000000
 Pages read:     0 / 786432 (0%)
 Pages failed:   786432 (100%)
Memory Probe: Completed.

I've tried swapping between PCI lanes and rebooting the machine a few times as well as swapping to slower read speeds but this issue seems to keep popping up. Is there some other obvious step I've missed in order to get access to the remaining memory / is there anything else I should be trying?

Thanks for your patience and apologies if this is some RTFM material - I've tried to go through all the documentation and I'm just not sure what next steps to try here. Do you have any suggestions / is there any other information I can provide that would be helpful?

ufrisk commented 4 years ago

Address space in a PC isn't exclusively backed by RAM.

Between 3-4 GB there is a space which is reserved for devices; reading these addresses usually generate a read towards a memory mapped device. This is usually not a problem; but some computers on some hardware really dislikes this. Behavior can range from nothing, to freezes, to reboots, to device stops working. I think this is the most likely cause of your problem.

Right now you would need to use pcileech with the -min and -max flags to skip those addresses; like pcileech.exe dump -max 0xc0000000 -out 0-3gb.raw and pcileech.exe dump -min 0x100000000 -out 4gb.raw.

In next version which I have ready for release, which may be released soon I hope (it's dependent on legal issues right now which is out of my control, but which I hope will be resolved soon) I will have an option to manually specify memory map.

MemProcFS shouldn't really have these problems though; since it usually never read anything in those regions.

This is what I think is the most likely issue; There might be more issues at play as well (due to the MemProcFS error); but it's really hard for me to tell. Lets wait for that memmap feature first.

michaelweber commented 4 years ago

I tried reading from higher addresses to skip further ahead and kept running into the same issue (mins of 0x100000000 and a bunch of other higher values also failed). Definitely makes sense to wait for the memmap feature.

Sorry to hear that you're running into legal issues with this project - like it wasn't enough that you're dedicating your free time to a passion project, there needs to be legal shenanigans too. Do you have a Patreon or https://github.com/sponsors account that the willing can throw some money at if it would be helpful?

ufrisk commented 4 years ago

PCILeech also tries to auto-detect memory above 4GB by doing some very small reads. I never encountered a system which this caused problems on. Can you try to do a pcieech dump -min 0x100000000 -max 0x120000000 setting the -max will disable the auto-detect of the maximum memory. Are you able to read with this? I guess it should also be theoretically possible that your system does not like 64-bit PCIe TLPs; but I also have never seen this so I don't think it's a problem. If that does not work I'll ask you to send some raw TLPs.

I got enrolled in the Github sponsors a while ago, but they require you to sign a tax document. For that you need a US tax number from IRS. To get the IRS tax number I have to book an appointment at the US embassy and they'll need to sign papers. And the embassy is closed for that kind if business due to the corona. I do have an old paypal account - paypal@ulffrisk.com ; I've gotten 2-3 contributions before all from very nice individuals (corporations don't sponsor), but that's about it. In total I've made such a loss on this project both financially (hardware purchases) and with regards to time. I can't promise anything in return though.

whatsgoingon1177111 commented 4 years ago

You should add bitcoin. @ufrisk

michaelweber commented 4 years ago

Apologies for the delayed response here but specifying the max seems to be what does it.

Ex:

C:\projects\hardware-dma>pcileech -device fpga dump -min 0x100000000 -max 0x200000000 -out 4g.raw

 Current Action: Dumping memory
 Access Mode:    Normal
 Progress:       4096 / 4096 (100%)
 Speed:          93 MB/s
 Address:        0x0000000200000000
 Pages read:     1048576 / 1048576 (100%)
 Pages failed:   0 (0%)
Memory Dump: Successful.

If I don't specify the max and try the same command though...

C:\projects\hardware-dma>pcileech -device fpga dump -min 0x100000000 -out 4gb.raw

 Current Action: Dumping memory
 Access Mode:    Normal
 Progress:       848 / 20480 (4%)
 Speed:          106 MB/s
 Address:        0x0000000135000000
 Pages read:     0 / 5242881 (0%)
 Pages failed:   217088 (4%)

I guess I'll just manually dump and then feed address info into MemProcFS instead of letting it scan?

Thanks for the assist here @ufrisk!

ufrisk commented 4 years ago

Thank you for the very nice donation. I hope to be able to commit bunch of updates as soon as possible which I already have ready in my development copy, but I have to resolve the legal issues first unfortunately so it's most likely some time away.

With regards to DMA stops working on your computer due to the auto-detect algoritm I think I'll keep things as-is since specifying the correct -max value will resolve your issue. Also the issue is rare, it's the first time I had it reported, in combination with the auto-detect being super useful for the absolute majority.

ufrisk commented 4 years ago

@whatsgoingon1177111

Finally added bitcoin now: bc1q9kur5pym8wmh5yxkf65792rdqm0guncd2gl4tu

Also, good news is that I'll be able to keep this project going :) Plan is to release a major update in August/September some time.

ufrisk commented 4 years ago

Please try the release just released. I hope this should now be fixed. For best stability I would suggest dumping with: pcileech.exe dump -v -device fpga -memmap auto on your Ryzen system. Please let me know if the error should still be around.

ufrisk commented 4 years ago

I assume this issue was resolved. If not, please let me know otherwise (and re-open the issue).

michaelweber commented 4 years ago

Still haven't been able to test it on my end to be honest, but from all of the fixes you've pushed I have to imagine this won't be a problem. If I do see the issue I'll post here though for sure. Cheers!