Memory leak when using first ScriptEngine generated by python to compile global variables

[serasval]

Description

I've included a simple console app that demonstrates the leak and a screenshot of the memory retention graph after all the objects go out of scope and should have been garbage collected.

So if you have a Python script that has global variables defined in it a leak exists when a global variable in the python script is assigned a value. That global variable is never let go for the duration of the application and therefore whatever object its referencing can't be garbage collected either which, depending on what you're referencing, could be a lot of memory leaked. In the Console app its just a simple class with a name so you can tell which instance of the object is being held in memory by the global variable.

The most interesting part of this however is that using an instanced based approach for the ScriptEngines (originally it was a static reference and it leaked all the time) only the first ScriptEngine created in the application and used to compile scripts with a global variable leaks. If you uncomment out the first two lines of the main method (thereby making and then throwing away the first ScriptEngine) the leak goes away.

Console app files

IronPythonGlobalMemoryLeakConsole.zip

Steps to Reproduce

1. Open and build the project I included IronPythonGlobalMemoryLeakConsole
2. Let it run for a second until it hits the readline
3. Use whatever tool you're familiar with to investigate the current state of the memory in the application (I was using .Net memory profiler by Scitech)

Expected behavior: I expect the global variables to be released/cleaned up after the ScriptEngine that compiled them is out of scope whether or not its the first ScriptEngine created.

Actual behavior: The first ScriptEngine created leaks on python scripts that are compiled/called with global variables and leaks on the global variables

Versions

2.7.8

[slozier]

Here's a slightly simpler to follow example:

class Program
{
    class TestObject { }

    static void Main(string[] args)
    {
        TestLeak(10);

        // There is still one TestObject alive

        Console.WriteLine("Waiting for ReadLine");
        Console.ReadLine();
    }

    public static void TestLeak(int loops)
    {
        const string code = @"
policy = None
def testMethod(args):
    global policy
    policy = args
";

        for (int i = 0; i < loops; i++)
        {
            // Only leaks in the first engine that is instantiated
            var engine = Python.CreateEngine();

            // Doesn't leak with SourceCodeKind.AutoDetect
            var compiled = engine.CreateScriptSourceFromString(code, SourceCodeKind.File).Compile();
            var scope = engine.CreateScope();
            compiled.Execute(scope);

            var testMethod = scope.GetVariable("testMethod");
            testMethod(new object[] { new TestObject() });
        }
    }
}

[serasval]

Any idea why SourceCodeKind.AutoDetect doesn't leak whereas SourceCodeKind.File does? Cause I've been trying to resolve memory leaks for like a month and finally stamped them all out through other various fixes and that one parameter fixes literally all of them by itself.

[slozier]

I think what happens is something like the following:

You create a ScriptRuntime and get a ScriptEngine which initializes the LanguageContext (in this case a PythonContext). In my sample this is done by Python.CreateEngine. When PythonContext is initialized it also creates a CodeContext which which is tied to the PythonContext and captures the state of modules. It is important to note that when the first PythonContext is created, its CodeContext becomes the default context and is kept alive by a static variable. When you create a script using the SourceCodeKind.File attribute it's basically treated as a temporary module so its state is stored in the current CodeContext.

This means that there are a few possible workarounds for your issue:

• Create a dummy engine which will initialize the default PythonContext/CodeContext. This is something you had pointed out in your sample code.
• Don't create your ScriptSouce using SourceCodeKind.File since it creates a temporary module.
• Run an empty script which overrides the temporary module state.

I'm not sure what the proper fix is. Need to think about it some more.