Breakpoints only hit first time when remote debugging

[joneal]

I have scripts hosted in a C# application, using IronPython 2.7. When running in debug mode, the scripts are loaded and encapsulated in the Python code that is an embedded resource (text file).

       private void executeScript(string scriptFileName, bool debugScript = false)
        {
            string strExec = String.Empty;

            ScriptScope scope = !debugScript ? m_scope : m_debug_scope;

            PythonCompilerOptions pco = (PythonCompilerOptions)m_pythonEngine.GetCompilerOptions(scope);
            pco.ModuleName = "__main__";
            pco.Module |= IronPython.Runtime.ModuleOptions.Initialize;

            if (!debugScript)
            {
                strExec = string.Format(Properties.Resources.PythonScriptContainer, scriptFileName);
            }
            else
            {
                pco.Optimized = false;
                strExec = string.Format(Properties.Resources.PythonDebugScriptContainer, scriptFileName);
            }

            ScriptSource scriptSource = m_pythonEngine.CreateScriptSourceFromString(strExec);

            CompiledCode compiled = scriptSource.Compile(pco);
            compiled.Execute(scope);
        }

Properties.Resources.PythonDebugScriptContainer:

try:
    print 'Configuring PTVSD...one moment please'
    try:
        import ptvsd
        ptvsd.enable_attach("Anduin", address=('localhost', 6677))
        print 'PTVSD configured'
    except ptvsd.AttachAlreadyEnabledError:
        print 'PTVSD already configured for this process...continuing'
    print 'Waiting for remote debugger to attach...'
    ptvsd.wait_for_attach()
    print 'Remote debugger attached!  Running script...'
    execfile(r'{0}')    
except Exception, e:
    if str.find(str(sys.exc_info()[1]),'Thread was being aborted.') == -1:
        print CommonTestUtilities.createExceptionOutputString(sys.exc_info())
    raise e
finally:
    ptvsd.visualstudio_py_debugger.detach_process_and_notify_debugger()

Once the Python engine and script with PTVSD initializes, it waits for a remote debugger to attach. I can attach with either VS2013 with PTVS or VSCode with Python extension (0.5.5). Breakpoints are preset. Everything works great the first time thru the script. When the remote debugger detaches, I run the script a second time. When the remote debugger attaches, none of the breakpoints are hit. This issue is seen with both VS2013 and VSCode.

Also, it doesn't seem to matter if the clean up in the finally block is there or not:

finally:
    ptvsd.visualstudio_py_debugger.detach_process_and_notify_debugger()

Any suggestions on how to retain the breakpoints? The only way to get them to work again is to kill the hosting C# process and restart, which is obviously not desired.

Thanks!

[int19h]

I am actually surprised this works at all. The problem is that you're loading the file as string, and by the time you pass it to IronPython, it has no way of knowing where that string originated from. So it can't really match breakpoints - which are just (filename, line_number) tuples - to the executing code.

If I remember correctly, CreateScriptSourceFromString has an overload that takes a filename as an argument - can you try that and see if it helps?

[joneal]

I tried the following:

ScriptSource scriptSource = m_pythonEngine.CreateScriptSourceFromString(strExec, SourceCodeKind.Statements);

ScriptSource scriptSource = m_pythonEngine.CreateScriptSourceFromString(strExec, SourceCodeKind.File );

ScriptSource scriptSource = m_pythonEngine.CreateScriptSourceFromString(strExec, @"<path_to_python_file>");

ScriptSource scriptSource = m_pythonEngine.CreateScriptSourceFromString(strExec, @"<path_to_python_file>\<python_file>.py");

ScriptSource scriptSource = m_pythonEngine.CreateScriptSourceFromString(strExec, @"<path_to_python_file>\<python_file>.py", SourceCodeKind.File);

Same behavior -- breakpoints hit with first run, but not in any subsequent runs.

I'm new to Python and PTVSD, but since breakpoints appear to be restored if I cycle the hosting process i.e. quit, start the C# host and then reload the Python script, could it be that breakpoint info is cached/cleared in the visualstudio_py_debugger.py that needs to be refreshed when the remote debugger connects?

[int19h]

I was wrong above, because it's only the helper script that you use to start ptvsd and then load your code that's not associated with any file. Since the script itself uses execfile inside, the actual source file that is executed by that (which, I assume, is where you have your breakpoints) does have source file info. Hence why breakpoints work the first time.

And yes, VS should clear the breakpoints when it detaches. But it will also re-apply them when it attaches.

I wonder if this has something to do with threads. Do you, by chance, run your scripts on a different thread each time? enable_attach installs settrace hooks (that it needs for breakpoints) on the thread on which it is called from, and it also tries to detect new threads and install them there too; but it only works for threads that are created via Python. I'm not that familiar with IronPython threading model, but if it maps 1:1 to CLR threads, and those threads are created outside of IronPython (not via threading) etc, then that might explain it.

Also, what happens if you do ptvsd.break_into_debugger?

[joneal]

My script where I have breakpoints is in the file loaded by execFile, so its reassuring that breakpoints are supposed to work, which they do the first time executed. The thread idea is something I can explore next week. This application, which I am learning, does a lot of InvokeRequired checks to marshal a lot of results back onto the UI thread, so it is very possible that the routine that launches the script is run on a different thread each time. I'll work my way back up the call stack to see, and also test ptvsd.break_into_debugger. Thanks for your help.

[joneal]

I researched my application and found that a new .NET thread is created for each run that encapsulates a run of the script (an actual new thread...not from the thread pool!). I did a quick modification so that only one background thread is created to ensure script thread affinity and then pulsed with an AutoResetEvent. Now breakpoints are retained with each connection of the remote debugger.

I'll do more testing, but I think I am going to tune this new design since it seems to resolve the issue, at least for my scenario.

Thanks for you help!

[zooba]

We have plans to significantly refactor the debugger (#750), and I think as part of that we should make it so that calling enable_attach multiple times will activate tracing on the current thread. Right now, I believe it doesn't do anything because the global state says it's already active.