search

microsoft / PTVS

Python Tools for Visual Studio
https://aka.ms/PTVS
Apache License 2.0
2.53k stars 676 forks source link

How to debug problem within Python. Strange behavior of debugger. #2384

Closed mrolle45 closed 7 years ago

mrolle45 commented 7 years ago

I have some Py code that makes calls into a COM server, via the comtypes package. Running the code gives me a stack overflow exception, with or without debugging. If I debug the code and break the debugger at a certain point, I get either some COM error or the stack overflow, and sometimes (varying with what I previously did during debugging) an access violation. By the way, with the access violation, I can then run the module without debugging and now I get the access violation (but with a different address).

Now I can step through the COM Item () call at line 37 in mixed mode debugging to see where the COM error came from. But I can't seem to break the debugger to trap either the access violation or the stack overflow at line 36. If I do a Step Into at line 36, I go to tagVARIANT.init. Then I do Step Out, and another Step Into at line 36, then I go straight into the except statement.

Here's my Python module (bad_example.py) and other things that it needs:

bad_example.zip

Now it would appear that the call to Item () is corrupting memory somewhere, since this method invokes an in-process COM server written in C++. As I mentioned above, I can step through the COM implementation of the Item () call. However, I can't see the C code (in _ctypes.pyd) that called it.

I seem to recall in the past being able to debug some PyC code that I wrote. But it doesn't seem to be happening with the PyC code in _ctypes. Do I need to provide a symbol file for _ctypes?

What I am looking for in the way of help is either

  1. Tell me how to debug _ctypes.pyd, or
  2. Maybe you can run the example yourself and find out where the accvio and stack overflow exceptions are coming from.

To run my example, you need to install the comtypes package, generate the comtypes.gen.Dia2Lib module, and register the msdia120.dll type library. You need to register either the 32 bit (in bin) or the 64 bit (in bin/amd64) version, corresponding to the build of Python 2.7 you have. Use the regsvr32 command on the command line to do this. I have the comtypes tree which you can add to your site-packages folder, and the VS 2013 version of DIA SDK in the above zip file.

mrolle45 commented 7 years ago

It's possible that there's a bug either in the PyC code in various packages being called, or in the PTVS debugger itself. Thus, it may be important to more than just myself to solve this issue. Thanks.

zooba commented 7 years ago

@int19h may have some more tips, but I think we have special handling around _ctypes.

You need symbols to even start mixed-mode debugging, so you likely have them already (though it may be worth looking in the Modules window to see if they failed to load for some reason).

int19h commented 7 years ago

Yes, we do have special handling for ctypes. Basically, when you invoke a native function from Python via ctypes, stepping into it should "just work". Like the rest of mixed-mode debugging, this functionality requires symbols, in this case for _ctypes.pyd.

If I understand correctly, what you really want in this case is the opposite - i.e. for the magic to "go away", so to speak, so that you can see and debug the C code that implements the native call inside _ctypes.pyd. We do have an undocumented feature that lets you do this.

To enable it, download and import this .reg file: https://github.com/Microsoft/PTVS/blob/master/Python/Product/Debugger/DkmDebugger/PythonDeveloper.reg

After that, you will see a couple new items show up in the context menu of debugger windows (in the same place where you normally see items for "Python View" and "C++ View"). The "Show Native Python Frames" option will show all the stack frames that belong to Python implementation that are normally hidden - basically everything from python.dll and ctypes.pyd. "Use Python Stepping" option, when unchecked, disables Python stepper completely, meaning that native debugger takes over the stepping (i.e. if you're in Python code, you'll be stepping through the Python interpreter itself).

So, to debug the way you want, you can hit the breakpoint on the line with the call, then enable native frames and disable Python stepping, and do "Step in" until you hit the code in ctypes.pyd that you want to debug.

huguesv commented 7 years ago

Is this something we want to document (PythonDeveloper mode)? Or shall we close this?