ScreamerM2 hangs while persistent reading of small amount of data

[xNarkon]

Hey buddy,

I would like to ask you because you may have already encountered this problem or, what's more, you know what could be the cause.

The project I'm working on requires persistent reading of a small amount of data from memory. In general, everything works fine, but from time to time reading from memory - completely irregular, if I can call it that, hangs for about 1 second. The data is simply not read, and after this time, everything returns to normal. It is quite annoying but does not interfere in the context of the whole. Currently, I have firmware version 4.3 I haven't checked the latest changes you made. Do you know what is caused by this?

I would like to mention that I had played already a bit with a delay in reading, and it does not matter, regardless of whether I set it to 0 uS or, e.g., to 300 uS. The best way is not to have any delay at all because I need to get always the current data as soon as possible. I am reading memory with the NO_CACHE flag.

Specification of the computer being attacked:

CPU: Intel i7-9700K (not overclocked) RAM: 32GB DDR4 3600mhz with XMP profile enabled Motherboard: MSI Tomahawk Z390 (MS-7B18) GPU: MSI GTX 1080Ti

Settings in the BIOS also checked and everything that I read in other problems and documentation, making sure that it is turned off.

FPGA is inserted by the PCI adapter provided by LambdaConcept; I don't have any free M.2 slot

DSN is faked, Device-Id is faked to a non-existent device, and Vendor-Id is set as Realtek.

From what I remember, even on unmodified firmware, there is a similar situation.

Many thanks in advance for any ideas and help.

[ufrisk]

Apologies, but I cannot really give support for custom built bitstreams; there are way too many unknowns for me to be able to work with it.

I can think of two reasons though: 1) The vmm library does background refreshes of internal state, if it's in the middle of one of those your read will be delayed (but not failed). Disable this with -norefresh flag to VMMDLL_Initialize or set option for it after startup.

2) I have it locking up completely for me in very rare cases at random (I need to unplug usb cable and replug it again); resetting the device itself does not work. I'm trying to look into it, but since I don't see any pattern at all it's very hard. Also this does not exactly fit your issue since it's working for you after this.

Anyhow, I'm closing this issue now. You are welcome to re-open it if you can confirm you have the same issue on the stock bitstream I publish of most recent version.

[xNarkon]

Apologies, but I cannot really give support for custom built bitstreams; there are way too many unknowns for me to be able to work with it.

I can think of two reasons though:

1. The vmm library does background refreshes of internal state, if it's in the middle of one of those your read will be delayed (but not failed). Disable this with -norefresh flag to VMMDLL_Initialize or set option for it after startup.
2. I have it locking up completely for me in very rare cases at random (I need to unplug usb cable and replug it again); resetting the device itself does not work. I'm trying to look into it, but since I don't see any pattern at all it's very hard. Also this does not exactly fit your issue since it's working for you after this.

Anyhow, I'm closing this issue now. You are welcome to re-open it if you can confirm you have the same issue on the stock bitstream I publish of most recent version.

hey,

A quick summary. Unfortunately, but I also checked the latest version 4.5 without any changes and the situation is identical. At some point, everything related to reading memory by vmmdll suddenly hangs for a while and after 1-3 seconds returns to normal operation. This is definitely not a problem for my program, because when I change the adapter to Native Win32 where I use ReadProcessMemory and run it directly on the victim machine, then there is no problem.

I tried to set the -norefresh flag but unfortunately this is not a solution for me because from what I noticed I get false results very quickly - phys2virt transference is a problem here?

Would you be able to try to verify this problem at home? Maybe the issue here is the setup of the attacked device?

[ufrisk]

I'll be able to check this out yes, but I'm away for the next few weeks to it will take some time unfortunately. Thanks for reporting :)

[N0x61r0x6Bo0x6E]

@ufrisk I also noticed this kind of problem. I'm using 4.5 version of FPGA (also ScreamerM2). In my case, the problem starting occurred after a while when I call VMMDLL_Init and VMMDLL_Close connection a few times. The longer the device is working the performance is going to be worse. Is it possible that there is some kind of software problem or maybe hardware like device temperature is too high or something?

[ufrisk]

I haven't really envisioned this to be used by calling VMMDLL_Init / VMMDLL_Close lots of times. I suspect there may be all kinds of predominantly memory leaks in the Close function. How many times are you reconnecting before you notice this becoming a problem?

Proper way to do it is to call VMMDLL_Init once and then keep it open. If you would need to refresh and clear caches there are options you can "set" to achieve that.

[xNarkon]

I also noticed the same problem. For me, frequent connection/disconnection is because of the testing the program that I am writing. Unfortunately, every 1 hour or so (during this time, I connect and disconnects from the device 3-4 times) I have to restart the target PC because the efficiency drops by about 40%

[ufrisk]

Thanks. I'll put this on my todo list to look into. I'm quite certain it may have to do with various memory and handle leaks.

[aDioS786]

HI Ufrisk, I am doing somethjing similar reading large data and noticed sometimes the performances drops massively. I am opening connection only once, and going to try -norefresh as u mentioned above. Not sure if you had a chance to look into this, as your last post mentioned that ? Thanks

[ufrisk]

This was quite some time ago. I looked into it as in disconnecting and reconnecting and fixed a few memory leaks and such. Nothing major. I've also changed a lot of internals since then.

I haven't really looked into the performance part of running it a very long time. Things like that may be a bit tricky to replicate; but obviously I need to do some long term testing later this summer (I'm quite busy as always this month). But soon I have some vacation and some time to look into this.

A thought about this, it may be that the Screamer is getting very hot after prolonged use. A small heat spreader on top of it may help slightly. I don't know if this is your issue though; but it may be worth a try.

Also, if you would be able to reliably replicate this it would make things much easier for me to look into; random slowdowns after 1 hour+ use is a bit hard for me to pinpoint. But I have the full respect for that this may not be easier than this and this is something I'll have to look into regardless...

[aDioS786]

Thanks for getting back, i'll do more testing. Meanwhile one more difference I've noticed. If i connect USB to same pc where screamer is, the same code works flawless, everything gets processed pretty quick (reading data from different location usually 1 6-32 bytes off each entity (and then each entity (offsets I'm reading). When I connect screamer USB to other pc/laptop (tried different pc's) the same code takes around 5-10 seconds more to process. Not sure why it is taking so long, when other pc is used to run same code. I've tried USB2.0 and 3.0 on difference PC's the result is pretty much same. Not sure if 've managed to explain the issue...

[ufrisk]

About the issue with your other PC, which I suspect is completely separate, are you sure it's running at usb3 speeds? Sometimes it's connected to a usb2 only port; some times there have been cable glitches downgrading it to usb2 which is much slower.

you could try to do: pcileech.exe dump -out none -device fpga -v -vv and see at which speed it dumps at the different PCs. If it's in the 40MB/s region or below I'm almost 100% certain your issue is downgrade to USB2.

[aDioS786]

Thanks, I've tried the cmd on different ports on pc screamer is in.. and getting roughly around 140MB/s, whereas on laptop i5 getting 18MB/s on all 3 ports i've tried. But if I use command pcileech.exe -v -vv -device fpga -min 0x100000000 probe on laptop, gettinga round 128MB/s. That's a lot higher speed I'm getting when connected to same pc, I've tried different ports and getting similar like 7x higher speed. I've tested on friends pc as well, and he's got a new laptop and he's getting around 31MB/s on one port and the other one 28MB/s, although his laptop spec does says USB3.0.

[ufrisk]

Probe speeds are not relevant here; it's not reading complete pages. Memory dump speeds around 30MB/s indicates this is running on USB2 speed, not USB3 speed. Check your ports and possibly try another cable on your slow computer.

[aDioS786]

sorry for late response, I tested on other laptops and different usb's sadly achieving max around 28-31MB/s. Doesn't make sense as soon i connect to host pc itself (where screamer is) then the reading speeds goes up to around 135MB/s .. Not sure why it's so fast when connected to itself? Also is there any possibility for these screamer to absorb multi threading? may be 2 threads at least? Thanks.

[ufrisk]

My guess is that still sounds like an USB issue. It's fast on your target system since it's a USB3 connection. On your other laptops I'm guessing it's downgrading to USB2. Maybe try a different cable or USB port.

PCIe itself should be able to push way more than USB3 regardless; even if it's on a slow system. This is not a threading issue.

[aDioS786]

Thanks for ur response, with threading question, it's not related to this issue, was just asking if multi-threading can be applied and if screamer are capable to handle multi-threading (c++)?

[ufrisk]

My libraries are multi-threaded and there are some benefits around it. If looking for a performance boost when reading you could read multiple scattered memory locations in parallel in one single thread though; have a look at the ReadScatter function.

The actual DMA access towards the Screamer is serialized (if needed) behind a lock which makes it single threaded though. This is completely transparent for the end user. It's the current implementation; I have some thoughts around trying a different approach; but I don't expect it to give any big gains especially if running at USB3 speeds; but it may help a bit. This would be some time away though.

[ufrisk]

I'm closing this issue due to old age.

Recently there are a lot of smaller bug fixes that may fix this issue. I don't experience this issue at all on PCIe Squirrel style of hardware nowdays and if something are to remain on the ScreamerM2 it may be due to heat issues or other hardware related issues. I don't think it would be related to pcileech-fpga project code any more.