pandas recursion error

[mikeymike94]

python:

import site
import pandas as pd

def mypandas():

    array = [1, 7, 2]
    myvar = pd.Series(array)

java: org.graalvm.polyglot.PolyglotException: RecursionError: maximum recursion depth exceeded

This error occurs even before the function is called, happens when the file is being loaded. Any ideas?

[msimacek]

Could you please provide more information about how you run the script? How did you create the virtualenv? What version of graalpython are you running?

[mikeymike94]

Hey @msimacek Thanks for the reply. The required information is as below :

1. Have installed Ubuntu on windows , since PYTHON is not provided by GRAALVM for Windows systems.
2. Have installed GRAALVM Version-22.1.0
3. Set the PATH variable for GRAALVM .
4. Then create Virtual environment using the command ----- >: graalpython -m venv "environmnt-name" source /bin/activate
5. Then installed packages using command -------- > : graalpython -m ginstall install "package-name" and also tried with command "environmnt-name" -m ginstall install "package-name"
6. Then ran the pytthon script using command python "file-name"
7. Python version installed in the virtual environment is python3.8

Observations : Python file was able to run successfully but while consuming it in JAVA we were having this error mentioned above . The code of Python file is as below :

import site import pandas as pd

def mypandas():

a = [1, 7, 2]
myvar = pd.Series(a)
print(myvar)

mypandas()

[msimacek]

Python file was able to run successfully but while consuming it in JAVA we were having this error mentioned above .

I don't understand this. You mean that CPython (the system python) was able to run it, but not graalpython? Or do you run it as a part of some Java application?

Then ran the pytthon script using command python "file-name" Make sure you don't name the script or the virtualenv "pandas" or anything that could conflict with it.

[mikeymike94]

@msimacek

"It ran with graalpython command also in the kernel , but while we are trying to consume the same python file in our JAVA code then it is not running and shwoing the above mentioned error ."

Below is the JAVA code which we have written for handling our Python file having Pandas library , and we have kept the Python file in the resource folder of our project :

public class GraalVmUtils {

public GraalVmUtils() {
}

public static Context getContext() {

    Context context = Context.newBuilder("python").allowNativeAccess(true).allowIO(true)
              .allowPolyglotAccess(PolyglotAccess.ALL)
              .allowHostAccess(HostAccess.ALL)
              .build();
    Source source;
    File file;
    try {
        FileInputStream fileInput = new FileInputStream(new File(GraalVmUtils.class.getClassLoader().getResource("SamplePanda.py").getFile()));
        System.out.println(fileInput);
        file = new File("temp.py");

        FileOutputStream fileOutPut = new FileOutputStream(file,false);
        int read;

        byte[] bytes = new byte[8192];
        while((read =fileInput.read(bytes))!= -1) {
            fileOutPut.write(bytes, 0, read);;
        }

        Value reference = context.eval(Source.newBuilder("python", file)
                  .build());

        Value addFunction =reference.getMember("mypandas");
         String[] arr = {"fruits", "vegetables", "food"};
         System.out.println(addFunction.execute(arr).asInt());

    }
    catch (Exception e) {
        System.out.println(e);
    }

    return context;
}

[mikeymike94]

Hey @msimacek any other information that I can provide?

[msimacek]

You need to point the Java program to the virtualenv where you installed pandas. Use .option("python.Executable", "[path-to-virtualenv]/bin/python") on the context builder, replace [path-to-virtualenv] with the actual path to the virtualenv where you installed pandas.

[mikeymike94]

@msimacek we did try with that as well, please note that we are able to execute the python script successfully from java with other libraries (for example numpy) instead of pandas. The issue happens when we try to run a pandas example, that's when it throws this error

java: org.graalvm.polyglot.PolyglotException: RecursionError: maximum recursion depth exceeded

[msimacek]

I cannot reproduce it with the example you provided (after adding the executable option). Please provide a complete example with all the options

[msimacek]

I wasn't able to reproduce the issue and you haven't provided more information. Feel free to reopen if you provide a complete reproducer.

[kai-zhu-sonatype]

Thanks @msimacek We also got this error RecursionError: maximum recursion depth exceeded when using Python in Java 21-graalce and 17.0.8-graalce, if you could try to dump AST for https://github.com/mgeitz/eqalert/blob/master/eqa/lib/parser.py like

ast.dump(ast.parse({parser.py file content}))

from the source seems the 1000 is fixed, not affected by sys.setrecursionlimit

• https://github.com/oracle/graalpython/blob/f936e8ffa29fc61879677b1d377c2551b00c8393/graalpython/com.oracle.graal.python.cext/src/ceval.c#L44

• https://github.com/oracle/graalpython/blob/f936e8ffa29fc61879677b1d377c2551b00c8393/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/objects/cext/capi/CApiContext.java#L135

• https://github.com/oracle/graalpython/blob/f936e8ffa29fc61879677b1d377c2551b00c8393/graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/cext/PythonCextCEvalBuiltins.java#L212

and seems option("python.Executable", ...) didn't affect context, tried to put an invalid path, code still ran.

@andrewstein also opened the issue https://github.com/oracle/graalpython/issues/362

[msimacek]

Hi @kai-zhu-sonatype, I replied in #362, let's continue the discussion there.

[timfel]

@kai-zhu-sonatype

and seems option("python.Executable", ...) didn't affect context, tried to put an invalid path, code still ran.

Most code will still run, this path is akin to setting argv[0] explicitly when using an exec function, argv[0] does not have to match the running executable. In Python, the value is used to resolve the venv configuration and to decide what to put in sys.executable which affects things like multiprocessing.

[kai-zhu-sonatype]

Thanks, @timfel for your reply. So there is no plan to support sys.setrecursionlimit in a native way in Java application yet?

[timfel]

Thanks, @timfel for your reply. So there is no plan to support sys.setrecursionlimit in a native way in Java application yet?

The JVM determines the stack size of a thread when the thread is created, so we don't have anything we can do for the main thread. As such, the only thing we could support is an artificially lower limit, but we cannot raise it