PythonContext leak

[deneuxj]

It seems that PythonContext objects created under the hood in hosted scripting can leak in some circumstances. From my observations, it seems normal and acceptable that the first instance of PythonContext remains live, and the same applies to the latest one.

Other instances, however, should be collected. This is important because PythonContext holds references to AST data that can get pretty large (I have observed 70 MB leaks at work).

The example below shows how to create leaks. Importing some modules triggers these leaks, but not all. For instance, sys is OK, but os leaks.

using System;
using System.Collections.Generic;
using System.Threading;
using Microsoft.Scripting.Runtime;
using IronPython.Compiler;
using IronPython.Runtime;
using Microsoft.Scripting.Hosting.Providers;
using Microsoft.Scripting.Hosting;

namespace IronPythonLeak
{
    class Program
    {
        static readonly string code = @"
import os
";
        static WeakReference wr = null;

        static void AddSearchPath(ScriptEngine engine) {
            var sp = engine.GetSearchPaths();
// Set to your liking
            sp.Add(@"....\main\External.LCA_RESTRICTED\Languages\IronPython\27\Lib");
            engine.SetSearchPaths(sp);
        }

        static void RunOnce(bool storeLanguageContext)
        {
            var options = new Dictionary<string, object>();
            options["Frames"] = ScriptingRuntimeHelpers.True;
            options["FullFrames"] = ScriptingRuntimeHelpers.True;
            options["LightweightScopes"] = ScriptingRuntimeHelpers.True;
            var setup = IronPython.Hosting.Python.CreateLanguageSetup(options);
            setup.ExceptionDetail = true;
            var runtime = IronPython.Hosting.Python.CreateRuntime();            
            var engine = runtime.GetEngineByFileExtension("py");
            if (storeLanguageContext)
                wr = new WeakReference(HostingHelpers.GetLanguageContext(engine));
            var compilerOptions = (PythonCompilerOptions)engine.GetCompilerOptions();
            compilerOptions.Module |= ModuleOptions.Interpret;
            compilerOptions.Module &= ~ModuleOptions.Optimized;
            AddSearchPath(engine); // Needed if using freshly compiled, non-installed IronPython DLLs.
            engine.Execute(code);
            runtime.Shutdown();
        }

        static void Main(string[] args)
        {
            Console.Write(code);
            RunOnce(false);
            RunOnce(true);
            RunOnce(false);
            System.GC.Collect();
            if (wr.IsAlive)
                Console.Write("Leaked");
            else
                Console.Write("Not leaked");
            Console.ReadKey(true);
        }
    }
}

[slide]

@jdhardy Another leak issue that may be solved by your updates.

[slide]

This still shows "leaked" on master

[slide]

@simplicbe Can you profile this one too?