search

lks9 / src-tracer

Other
0 stars 0 forks source link

Memory Error in print_trace #28

Open lks9 opened 1 year ago

lks9 commented 1 year ago

When trying to open a large trace file (30 GB) I am getting a memory error:

$ python print_trace.py 2023-05-04-000230-appletlib.c.trace
Traceback (most recent call last):
  File "print_trace.py", line 28, in <module>
    trace = Trace.from_file(trace_file)
  File "src_tracer/trace.py", line 51, in from_file
    trace_bytes = f.read()
MemoryError
lks9 commented 1 year ago

Now I know how to solve the error by using mmap like in find_in_trace.py. However, it's not yet implemented in src_tracer/trace.py.

ducorduck commented 1 year ago

Should i just replace read with the appropriate one from mmap?

Also I don't have a big trace so I can't recreate the bug.

ducorduck commented 1 year ago

i've done just that in the mmap_trace branch i've just created. Could you try it out?

lks9 commented 1 year ago

Sorry, this week I am on vacation and I don't have that 30 G file with me.

Maybe you can just create such a file yourself, by recording a loop? The C program can be very simple and the recording should be streight forward.

ducorduck commented 11 months ago

I've added bigTrace.c to test this issue. This give a 10gb compact trace or 14gb text trace, because i only have 8gb of ram so it's enough to encounter the error for me. I've tested my implementation and it works. If you want a bigger trace you could change the variable numGb to your like (10 is the default value). Just a note, the runtime for the text trace is a lot longer (about 10x) than the compact trace on this program.