Scatter read does not read consistently

[qskateboard]

I have a case where I need to queue a lot of read and execute them at the same time for better performance. The problem is that when I am using scatter read for queing reads (the number or items I queue is around 300ish), I am not getting all the data. The exact number of lost data is unknown, but I would say is around 30-40%. Anyway, first I thought there were some kind of problem with my code, but after hours of debugging, I replaced scatter read with basic read operations (very unperformant) and they worked as expected and read all the data I needed. So my question is if there is some kind of scatter read limit or is it even working? I read an old github issue where there were problem with page alignment crossing but I am using new functions that should resolve it, or am I missing something?

I am using scatter read like that:

const auto flags = VMMDLL_FLAG_NOCACHE | VMMDLL_FLAG_NOPAGING | VMMDLL_FLAG_ZEROPAD_ON_FAIL | VMMDLL_FLAG_NOPAGING_IO;
const VMMDLL_SCATTER_HANDLE scatterHandle = VMMDLL_Scatter_Initialize(handle, processInfo.pid, flags);
// address is of type const std::uintptr_t&
VMMDLL_Scatter_PrepareEx(scatterHandle , addr, size, static_cast<PBYTE>(buffer), &memoryPrepared);

VMMDLL_Scatter_Clear(scatterHandle , processInfo.pid, NULL);

[ufrisk]

are you using the most recent versions of the vmm.dll and leechcore.dll binaries as downloaded today?

[qskateboard]

I have tried several version including the new ones, but still miss most of the data

[salat-23]

Same issue as well. I am not excactly sure why that happens, is there a wiki article on howtouse scatter read coz I cant find it

[ufrisk]

There is an example in the example project about how to use the scatter.

But there might be an issue as well, and it's worrying if multiple persons are having this issue. Thing is that I haven't done anything with this for quite a while so I'm a bit perplexed as to why I'm hearing about it just about now. But there may very well be some issue...

@qskateboard just for testing, could you try to use leechcore.dll version 2.15 together with the most recent vmm.dll to see if the error persists. This is not recommended for "production use" since there are bug fixes and performance boosts in the newer leechcore versions, but I'm trying to rule out if one of the performance fixes is causing this bug. I don't think so but it would be great if you could run this test and let me know about the result.

If it's not resolving anything would it be possible for you to trigger this issue reliably with some piece of test code. If that's the case it would be totally awesome if I could receive a small piece of test code that triggers this issue against a common process, such as explorer.exe. It would make it so much easier for me to track down this issue (if there is an issue at all, which I'm not completely confident about).

[qskateboard]

Here is an example of the code where scatter read fails: https://gist.github.com/qskateboard/94c619445016bfaa6271f6ce8cf4e6f1

The main data loss is at the 4th level of reading data, almost all of the pointers are read as 0x0's, although when observing pointers by hand, they do exist and are valid I tried to make something out of explorer to provide simillar example, but did not succeed due to inability to read deep memory structures I also tried launching lost of threads which used scatter read all at once and it worked flawlessly The most interesting part is that when I change scatter queue read to default read - it works okay, slow but it reads everything, I am thinking that the problem might be with how I am opening and closing scatter handles but I wasnt able to replicate it so I am not sure...

Here is also my wrapper functions that are used: https://gist.github.com/qskateboard/67981d5e004ae16b27f215e5f7c06e55

[ufrisk]

I'm afraid this won't help me pinpoint the error you're receiving.

can't it just be that the scatter reads just take a bit of longer time if you queue a lot of items in it and your pointer chain is very fast flux so that the target application changed things around when you read at a deeper level?

if your read fails and you immediately try to re-read the same memory address and length with normal read (as the failed read, not the failed pointer chain) will it fail or succeed?

[qskateboard]

Yes, so I thought in the beginning as well, I did a test where I read the memory with normal read func immediately after, and it was doing it perfectly. The addresses themselves do not change when the read happens, I checked them manually when debugging. Is there anything else I could provide? All the non-read values become 0x0

[ufrisk]

ok, so there is a high possibility there may be an issue, but I cannot track it down like this.

if you on your 3rd, 4th level experience like 20%+ failed reads and you then verify that a failed read indeed should work if you use normal read can you printf all reads in the order they were added if they are "only" about 300-ish. Maybe there are some pattern.

like something similar as below:

added     address     size      fail
====================================
0001   7fff30002000   1000      N
...
00f0   7fff30004008      8      Y
00f0   7fff30005008      4      N

If we're lucky it may help tracking this down or make it easier to construct a test case in which its possible to replicate the issue.

[ufrisk]

Since I haven't heard back on this issue I'm assuming it was resolved. I'm closing this issue.