Random freeze using "Find out what accesses this address"

[damajor]

I think I ran into another freeze issue. It is pretty annoying :)

My steps to reproduce:

0. Open target process
1. Open Pince
2. Attach process
3. Find location
4. Add location to address table
5. Find out what accesses this address
6. Wait a bit and stop then use close button
7. If not frozen, repeat steps (5) and (6) more times

Freeze is almost always coming after 3rd or 4rth tries, and it happens even more faster if you kill & restart PINCE without restarting the target process.

Here is the kind of error I have in console output:

Exception in thread Thread-12 (state_observe_thread):
Traceback (most recent call last):
  File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/home/.../libpince/GDB_Engine.py", line 301, in state_observe_thread
    check_inferior_status()
  File "/home/.../PINCE/libpince/GDB_Engine.py", line 283, in check_inferior_status
    if bp_num and breakpoint_on_hit_dict[bp_num.group(1)] != type_defs.BREAKPOINT_ON_HIT.BREAK:
                  ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: '1'

The KeyError is always matching the number of time I use "Find out what accesses this address" feature.

Unfortunately I was NOT able to recover using the workaround you mentioned in the previous post.

Edit: After some console output reading it looks like some kind of race condition between thread, but I am not that familiar with your code to pinpoint the root cause.

Working output:

*running,thread-id="all"
=breakpoint-modified,bkpt={number="1",type="acc watchpoint",disp="keep",enabled="y",what="* (char[4] *) 0x8403f38",thread-groups=["i1"],times="508",script=["pince-get-track-watchpoint-info ['1']","c&"],original-location="* (char[4] *) 0x8403f38"}
~"\n"
~"Thread 1 \"TARGETPROCESS\" hit Hardware access (read/write) watchpoint 1: * (char[4] *) 0x8403f38\n"
~"\nValue = "
~"\"c\\000\\000\"\n"
~"0x0000000141a849b8 in ?? ()\n"
*stopped,hw-awpt={number="1",exp="* (char[4] *) 0x8403f38"},reason="access-watchpoint-trigger",value={new="\"c\\000\\000\""},frame={addr="0x0000000141a849b8",func="??",args=[],arch="i386:x86-64"},thread-id="1",stopped-threads="all",core="25"
Last command: pince-examine-expressions
*running,thread-id="all"
^done
0.004693508148193359
Last command: interpreter-exec mi "-break-list"
^done,BreakpointTable={nr_rows="1",nr_cols="6",hdr=[{width="7",alignment="-1",col_name="number",colhdr="Num"},{width="14",alignment="-1",col_name="type",colhdr="Type"},{width="4",alignment="-1",col_name="disp",colhdr="Disp"},{width="3",alignment="-1",col_name="enabled",colhdr="Enb"},{width="18",alignment="-1",col_name="addr",colhdr="Address"},{width="40",alignment="2",col_name="what",colhdr="What"}],body=[bkpt={number="1",type="acc watchpoint",disp="keep",enabled="y",what="* (char[4] *) 0x8403f38",thread-groups=["i1"],times="508",script=["pince-get-track-watchpoint-info ['1']","c&"],original-location="* (char[4] *) 0x8403f38"}]}
^done
0.0006825923919677734
Last command: delete 1
=breakpoint-deleted,id="1"
^done
0.0006196498870849609

Freezing output:

*running,thread-id="all"
=breakpoint-modified,bkpt={number="2",type="acc watchpoint",disp="keep",enabled="y",what="* (char[4] *) 0x8403f38",thread-groups=["i1"],times="702",script=["pince-get-track-watchpoint-info ['2']","c&"],original-location="* (char[4] *) 0x8403f38"}
~"\n"
~"Thread 1 \"TARGETPROCESS\" hit Hardware access (read/write) watchpoint 2: * (char[4] *) 0x8403f38\n"
~"\nValue = "
~"\"c\\000\\000\"\n"
~"0x0000000141a849b8 in ?? ()\n"
*stopped,hw-awpt={number="2",exp="* (char[4] *) 0x8403f38"},reason="access-watchpoint-trigger",value={new="\"c\\000\\000\""},frame={addr="0x0000000141a849b8",func="??",args=[],arch="i386:x86-64"},thread-id="1",stopped-threads="all",core="0"
*running,thread-id="all"
^done,BreakpointTable={nr_rows="1",nr_cols="6",hdr=[{width="7",alignment="-1",col_name="number",colhdr="Num"},{width="14",alignment="-1",col_name="type",colhdr="Type"},{width="4",alignment="-1",col_name="disp",colhdr="Disp"},{width="3",alignment="-1",col_name="enabled",colhdr="Enb"},{width="18",alignment="-1",col_name="addr",colhdr="Address"},{width="40",alignment="2",col_name="what",colhdr="What"}],body=[bkpt={number="2",type="acc watchpoint",disp="keep",enabled="y",what="* (char[4] *) 0x8403f38",thread-groups=["i1"],times="702",script=["pince-get-track-watchpoint-info ['2']","c&"],original-location="* (char[4] *) 0x8403f38"}]}
^done
0.003161907196044922
Last command: delete 2
=breakpoint-modified,bkpt={number="2",type="acc watchpoint",disp="keep",enabled="y",what="* (char[4] *) 0x8403f38",thread-groups=["i1"],times="703",script=["pince-get-track-watchpoint-info ['2']","c&"],original-location="* (char[4] *) 0x8403f38"}
~"\n"
~"Thread 1 \"TARGETPROCESS\" hit Hardware access (read/write) watchpoint 2: * (char[4] *) 0x8403f38\n"
~"\nValue = "
~"\"c\\000\\000\"\n"
~"0x0000000141a849b8 in ?? ()\n"
Exception in thread Thread-12 (state_observe_thread):
Traceback (most recent call last):
  File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/home/.../PINCE/libpince/GDB_Engine.py", line 304, in state_observe_thread
    check_inferior_status()
  File "/home/.../PINCE/libpince/GDB_Engine.py", line 286, in check_inferior_status
    if bp_num and breakpoint_on_hit_dict[bp_num.group(1)] != type_defs.BREAKPOINT_ON_HIT.BREAK:
                  ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: '2'

Also in the freezing output we can see that after *stopped line there is no Last command: pince-examine-expressions before *running,thread-id="all" .

I tested more with a single (or maybe low) threaded app like galculator (Gnome calculator) and I am not able to reproduce the issue.

So if multiples target process threads try to access the memory location watched by PINCE, that seems to lead to some issues in PINCE processing.

I hope this will help.

[korcankaraokcu]

I've tested this on KMines and Firefox but was unable to reproduce the issue. Which program did you try this on? Also, what distro do you use?

[korcankaraokcu]

A second glance at the log tells me that 2nd breakpoint was deleted while PINCE was trying to use it. Is your breakpoint triggered a lot? For this to happen, it should be triggered at least 10 times a second. But you are right about the racing condition

[korcankaraokcu]

https://github.com/korcankaraokcu/PINCE/commit/f9f5b37663985f1569bcaa73d2ecf2dd8e3354db should fix the racing softlock. But of course it might not be a definitive fix for breakpoint racing conditions. More stress tests will be done

[korcankaraokcu]

Upon further testing, there seems to be no critical bugs left with this issue. For minor effects, another issue could be opened with reproduction steps. Closing this issue

[Mind78]

Consider opening this issue again. I just came upon it. It happens when attaching to Stellaris. There is no need to do anything special other than just finding some variable, adding it to the list and then doing an "What access this address" It freezes without fail after a couple of seconds.

I am using Linux Mint 21.3 Virginia 64-bit Kernel Linux 5.15.0-119-generic x86_64 MATE 1.26.0 32 Gig ram

[brkzlr]

Try doing the same thing in Cheat Engine, either by using CEServer (if the game is Linux native) or by running it under the same Proton prefix.

From our experience, some games tend to crash CE or freeze PINCE using this feature so it's nothing we can do from our side.

[korcankaraokcu]

This issue was created for the "What accesses this address" feature for addresses that has been accessed very frequently, around 1000 times a second. Try using the same feature on other addresses that doesn't get accessed frequently and see if the issue persists. If it does, your issue might not related to this one.

[Mind78]

Hmm, yeah it was accessed quite frequently from one place, something like 360 times in about a second (or less) before if froze. It is still somewhat usable though, The game doesn't crash so I can write down the adresses and restart Pince and set up a different circumstance and run again, weeding out the one I'm looking for.

[korcankaraokcu]

I'll reopen the issue for you. Please write down the reproduction steps in detail. Also, make sure that you are using the native build on Steam. I'll try to reproduce this on my end as well.

[Mind78]

Native build. Yes, check.

1. Start stellaris as a Rogue defence system, determined exterminator machine empire.
2. Make sure Ironman is OFF
3. let the game be paused the entire time
4. Click on homeplanet
5. Press the console key to get up debug console in Stellaris
6. Start Pince
7. Attach to Stellaris
8. Search for the Planet size (Int64)
9. type planet_size 35 in stellaris console
10. Search for planet size 35
11. type planet_size 20 in stellaris console
12. repeat process until planet size is found.
13. add the address to the address list
14. Close the console in stellaris
15. Close the planet view in Stellaris, so you are just looking at the home system.
16. right click the address and choose "Find out what reads this address"

On my end this will read hits from 3 different addresses, one of them will reach a couple of hundreds and then Pince freezes.

[korcankaraokcu]

I couldn't find Rogue Defence System in the campaign start menu, maybe I didn't have the required DLC. However, I've followed the rest of the steps and found the planet size and successfully executed the "Find out what reads this address" feature multiple times without anything crashing. PINCE sometimes becomes unable to stop the process but that's a bug on GDB's side and it's an entirely different issue

Maybe you are using an outdated version of PINCE. There has been several patches regarding GDB racing issues. What's your PINCE version?

Edit: I've now also tried using the "Find out what accesses this address" feature and I get the 3 accessing instructions as you mentioned. This tanks my frame rate and occasionally pops up "Can't access memory at expression $pc" message box but that's another minor racing issue that doesn't result in crashing

[korcankaraokcu]

@Mind78 A reminder to continue the conversation so we don't leave issues hanging