Pymem and memprocfs

[matheusbranhan]

Is there any way to make pymem read a vmm process?

self.handle = vmm.process("explorer.exe") ` def get_pointer(self, address: int) -> int:

pointer in 64

    try:
        return self.handle.read_ulonglong(address)
    except pymem.exception.MemoryReadError:
        return 0`

[ufrisk]

There is no read_ulonglong function in the API currently.

I guess you could do something like

data = self.handle.process.memory.read(address, 8);

and then convert it into a number. I guess it might be nice to add an API function for it though.

[matheusbranhan]

Could you also add byte reading functions without using the struct function?

Like for example: def read_memory(process, address, size): return struct.unpack(f"<{size}B", process.memory.read(address, size)) def read_float_memory(process, address): return struct.unpack("<f", process.memory.read(address, 4))[0] def read_int_memory(process, address): return struct.unpack("<I", process.memory.read(address, 4))[0]

[matheusbranhan]

the fact of using struct makes my code very slow

[matheusbranhan]

@ufrisk Could you do it? I think it would help a lot of people with these new features in the API

[ufrisk]

I closed your other issue. Please don't open multiple issues for the same thing.

There is no way of doing this in the API currently, but I could easily add it should it be required. But what data types do you need besides float in that case?

but have you tried:

b = b'\x00\x00\x80\x3f'
f = memoryview(b).cast('f')[0]
print(f)

is the memory view very slow as well, or is it faster?

[matheusbranhan]

I'm sorry, I didn't mean to be inconvenient, I just thought it would be more organized.

It would help me a lot if you put this together, really. Currently, the functions I use for readings are:

def read_int_memory(process, address):
    return struct.unpack("<I", process.memory.read(address, 4))[0]

def read_int64_memory(process, address):
    return struct.unpack("<Q", process.memory.read(address, 8))[0]

def read_string_memory(process, address):
    data = b""
    try:
        while True:
            byte = process.memory.read(address, 1)
            if byte == b'\0':  # Verifica se é o terminador nulo
                break
            data += byte
            address += 1
        decoded_data = data.decode('utf-8')
        return decoded_data
    except UnicodeDecodeError:
        return data

def read_float_memory(process, address):
    return struct.unpack("<f", process.memory.read(address, 4))[0]

I believe these are

[ufrisk]

did you test the memory view solution I linked above? if it's fast enough there would be no need for me to look into this so I'd rather see if this would work or not first before implementing functionality.

can you check if it's fast enough?

[matheusbranhan]

unfortunately both returned practically the same value, in fact, the struct still managed to perform faster

on first run i used the code you sent me

First run: Average execution time = (0.00044630 + 0.00004090 + 0.00004190 + 0.00000870 + 0.00000660 + 0.00000670 + 0.00000680 + 0.00000750 + 0.00000840 + 0.00001070) / 10 = 0.00005854 seconds

Second run: Average execution time = (0.00041350 + 0.00003150 + 0.00002300 + 0.00000650 + 0.00000590 + 0.00000570 + 0.00000560 + 0.00000560 + 0.00001400 + 0.00002670) / 10 = 0.00003200 seconds

unfortunately, I need this to be faster, mainly to get the vec3 positions

[ufrisk]

thanks for confirming,

do you only need to a read single float per call, or also arrays of floats, i.e. where float1 starts at offset 0, float2 at offset 4, float 3 at offset 8 and so on?

[matheusbranhan]

I only need one per call for now

[ufrisk]

I'll be adding this tomorrow. I have come up with a solution that will fit all needs.

When the memory reading fails, do you prefer that I'd return 0 or None object?

[matheusbranhan]

tomorrow? faster than I expected, thank you so much ulf return 0, please

[ufrisk]

I've updated it with the requested functionality. I've also done some general performance optimizations.

If just doing single reads you'll get almost no performance improvements. (compare read single 4M with read struct 4M below).

If doing multiple reads at the same time you'll get decent performance improvements, especially if you're doing it with the Scatter API.

Examples:

# read single type
process.memory.read_type(address, 'u16', flags)

# read multiple types
process.memory.read_type([[address1, 'u64'], [address2, 'f32']], flags)

# read multiple types with scatter api
scatter_memory.read_type([[address1, 'u64'], [address2, 'f32']])

Please let me know if you have any comments around this. I know the performance impact wasn't what you'd wished for, but reading multiple types at the same time should at least give a decent performance boost...

[matheusbranhan]

I will test them. What would be the flag options?

[ufrisk]

Same as the other read function flags. i.e. memprocfs.FLAG_*

[matheusbranhan]

unfortunately, there was an increase, but not very noticeable. I didn't understand very well how memoryscatter works, could you demonstrate it in the line below? procd.memory.read_type(modedll + Offset.pos, 'u64')

[ufrisk]

I now realize I don't have a working example in the example file for the new Scatter API. I'll try to update in the coming week.

it's basically.

1. initialize via process memory
2. call prepare function multiple times with ranges to prepare
3. call execute function
4. call read functions to read from prepared ranges

[ConnorMAD]

I liked it, I'm also having these same problems, I'd also like a working example of the scatter api, I'll be waiting. I've been trying for days to get the performance to be good, but I can't

[ConnorMAD]

At least without using the struct, it's one less library in the code

[matheusbranhan]

I now realize I don't have a working example in the example file for the new Scatter API. I'll try to update in the coming week.

it's basically.

1. initialize via process memory
2. call prepare function multiple times with ranges to prepare
3. call execute function
4. call read functions to read from prepared ranges

ok

[ironman97]

不幸的是，有增加，但不是很明显。 我不太了解内存散射的工作原理，你能在下面的行中演示一下吗？ procd.memory.read_type(modedll + Offset.pos, 'u64')

hs = procd.memory.scatter_initialize(memprocfs.FLAG_NOCACHE) hs.prepare(modedll + Offset.pos, 4*3) hs.execute() pos_x=truct.unpack("<f", hs.read(modedll + Offset.pos, 4))[0] pos_y=truct.unpack("<f", hs.read(modedll + Offset.pos+4, 4))[0] pos_z=truct.unpack("<f", hs.read(modedll + Offset.pos+8, 4))[0] hs.close()

[ConnorMAD]

不幸的是，有增加，但不是很明显。 我不太了解内存散射的工作原理，你能在下面的行中演示一下吗？ procd.memory.read_type(modedll + Offset.pos, 'u64')

hs = procd.memory.scatter_initialize(memprocfs.FLAG_NOCACHE) hs.prepare(modedll + Offset.pos, 4*3) hs.execute() pos_x=truct.unpack("<f", hs.read(modedll + Offset.pos, 4))[0] pos_y=truct.unpack("<f", hs.read(modedll + Offset.pos+4, 4))[0] pos_z=truct.unpack("<f", hs.read(modedll + Offset.pos+8, 4))[0] hs.close()

Thank you boss, I did it. I will carry out tests with memory scatter

[ufrisk]

Also, I'll look into making some additional performance improvements to the scatter reads shortly.

Primarily it's about not releasing the Python GIL on the fastest function calls in the scatter API. This will save up quite some time.

[ConnorMAD]

Guys, thank you very much for your help, for me using scatter did the trick, I hope it also helps the owner of the issue

[ConnorMAD]

def read_single_float_memory(process, address):
    hs = process.memory.scatter_initialize(memprocfs.FLAG_NOCACHE)
    hs.prepare(address, 4)
    hs.execute()
    value = hs.read_type(address, 'f32')
    hs.close()
    return value

I hope it helps

[ufrisk]

Just published 5.8.12 which contains additional performance improvements as well as some new functionality.

Performance improvements are mainly from abstaining from releasing/re-acquiring the Python GIL on some fast functions, such as scatter read/prepare (but not execute).

New functionality: being able to do convenient scatter reads using normal read, but with a list of ranges, i.e.

list_of_ranges_to_read = [[module_explorer.base, 8], [module_explorer.base+0x3000, 0x10]]
result = process_explorer.memory.read(list_of_ranges_to_read, memprocfs.FLAG_NOCACHE)

I added examples to the Python examples.

Please let me know if you see any performance gains using this. For the new multi-read to be more performant than the old once I think you'd realistically have to do 3 reads or so, but I'm uncertain.

[ConnorMAD]

I can't use it for multiple addresses

                scatter = procd.memory.scatter_initialize(memprocfs.FLAG_NOCACHE)
                scatter.prepare([modedll + Offsets.posb, 12], [modell + Offsets.posc + 4, 12])
                scatter.execute()
                value = scatter.read_type([[modedll + Offsets.posb, 'f32'], [modell + Offsets.posc + 4, 'f32']])
                print(f'scatter --> : {value}')
                scatter.close()

am I doing something wrong? I think so

[ufrisk]

@ConnorMAD prepare call is wrong. Argument is list of ranges, not varar ranges as you did. Add outer [ ] and it will work.

[matheusbranhan]

@ufrisk Will you make new optimizations? I believe I've already gotten to where I wanted to be

[ufrisk]

I think I've optimized things as much as possible already in the critical code paths, i.e. reading memory. I don't plan to look into doing more optimizations right now than the ones that already exists in 5.8.12.

Awesome to see things are looking good @matheusbranhan - I'm closing this issue then 👍 Some very good improvements came out of this issue, please let me know if you should run into anything else you feel like it's missing.

Also if you find my project useful down the road please consider sponsoring with a small amount on my Github Sponsors. Sponsorships go for as little as $2/mo and even the smaller sponsorships are very much appreciated :) Thank You 💖