Support for robotframework

[vehovsky]

I would like to start this issue mainly to see interest for the support in robotframework community.

Ever since RoboCon 2021 GraalVM Python was on our radar as potential replacement for Jython. Seems the progress since then was very limited. After talking to @steve-s recently on GeeCon 2022 I understand GraalVM Python developers were not aware of such use-case and that might be the reason why there was minimal progress.

I've tried basically the same as robotframework-after-jython with latest GraalVM 22.3.0

$ python --version
GraalVM Python 3.8.5 (GraalVM CE Native 22.3.0)

$ python -m robot --help
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vehovsky/playground/graalvm-ce-java11-22.3.0/languages/python/lib-python/3/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/vehovsky/playground/graalvm-ce-java11-22.3.0/languages/python/lib-python/3/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/vehovsky/playground/graalvm-ce-java11-22.3.0/languages/python/lib-python/3/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/vehovsky/rf/lib/python3.8/site-packages/robot/__init__.py", line 43, in <module>
    from robot.rebot import rebot, rebot_cli
  File "/home/vehovsky/rf/lib/python3.8/site-packages/robot/rebot.py", line 45, in <module>
    from robot.run import RobotFramework
  File "/home/vehovsky/rf/lib/python3.8/site-packages/robot/run.py", line 44, in <module>
    from robot.running.builder import TestSuiteBuilder
  File "/home/vehovsky/rf/lib/python3.8/site-packages/robot/running/__init__.py", line 104, in <module>
    from .userkeyword import UserLibrary
  File "/home/vehovsky/rf/lib/python3.8/site-packages/robot/running/userkeyword.py", line 24, in <module>
    from .userkeywordrunner import UserKeywordRunner, EmbeddedArgumentsRunner
  File "/home/vehovsky/rf/lib/python3.8/site-packages/robot/running/userkeywordrunner.py", line 28, in <module>
    from .timeouts import KeywordTimeout
  File "/home/vehovsky/rf/lib/python3.8/site-packages/robot/running/timeouts/__init__.py", line 24, in <module>
    from .posix import Timeout
  File "/home/vehovsky/rf/lib/python3.8/site-packages/robot/running/timeouts/posix.py", line 16, in <module>
    from signal import setitimer, signal, SIGALRM, ITIMER_REAL
ImportError: cannot import name 'setitimer' from 'signal' (/home/vehovsky/playground/graalvm-ce-java11-22.3.0/languages/python/lib-python/3/signal.py)

Which I tracked to https://github.com/robotframework/robotframework/issues/4100:

The problem occurs when importing robot.running.timeouts.posix module which shouldn't be imported at all when using Jython.

[msimacek]

Hi @vehovsky, thank you for your interest in GraalPy. I tried to run robotframework tests with GraalPy and there were two things I needed to change on our side to get it working:

• Add signal.setitimer (as you noted). Unfortunately, the Jython-compatibility code that used to disable the import for Jython has already been removed from robotframework's master, so I think we just have to add the builtin. I'm not sure how difficult it would be, signal handling on JVM is tricky, but we will try. For my testing I added just an empty dummy.
• It parses the output of python --version and we don't match the format it expects. I will fix that.

I'll keep you updated on the fixes.

[msimacek]

We have fixed both of those issues. Could you please try your use cases with a snapshot build?

[vehovsky]

Latest snapshot I see now is [GraalVM CE 23.0.0-dev-20221103_2324] released 12 days ago. Tried it and still see the same issue..

[msimacek]

Oh, you're right, I'm sorry. The builds are supposed to be done daily, but there are apparently some failures. I notified our release engineering

[msimacek]

A new build is there and I checked that it contains the fixes. Please try that one

[vehovsky]

Hi,

must say, looks pretty promising! 👍

I was able to run successfully quite a lot of the Robot Framework acceptance tests.

Quite a lof of those failures were caused by:

16:55:22.079    INFO    Processing output '/tmp/robottests/GraalPy-3.10.7-Linux/output/output.xml'. 
16:55:22.118    INFO    ${SUITE} = None 
16:55:22.125    INFO    Traceback (most recent call last):
  File "/home/martin/playground/robotframework-4.1-maintenance/atest/resources/TestCheckerLibrary.py", line 85, in process_output
    ExecutionResultBuilder(path).build(result)
  File "/home/martin/playground/robotframework-4.1-maintenance/src/robot/result/resultbuilder.py", line 108, in build
    self._parse(source, handler.start, handler.end)
  File "/home/martin/playground/robotframework-4.1-maintenance/src/robot/result/resultbuilder.py", line 120, in _parse
    for event, elem in context:
  File "/home/martin/playground/graalvm-ce-java17-23.0.0-dev/languages/python/lib-python/3/xml/etree/ElementTree.py", line 1259, in iterator
    root = pullparser._close_and_return_root()
  File "/home/martin/playground/graalvm-ce-java17-23.0.0-dev/languages/python/lib-python/3/xml/etree/ElementTree.py", line 1302, in _close_and_return_root
    root = self._parser.close()
  File "/home/martin/playground/graalvm-ce-java17-23.0.0-dev/languages/python/lib-python/3/xml/etree/ElementTree.py", line 1722, in close
    self._raiseerror(v)
  File "/home/martin/playground/graalvm-ce-java17-23.0.0-dev/languages/python/lib-python/3/xml/etree/ElementTree.py", line 1622, in _raiseerror
    raise err   
16:55:22.129    FAIL    Processing output failed: ParseError: no element found: line 964, column 5

I did not see such issues in the Jython execution (plenty of different issues there), so unclear to me if that is something that can be fixed in Graal?

I've also tried executing the RF tests from Java, that worked. Including Java library and using keywords written in Java is another story. It would be great if someone like @pekkaklarck could have a look at it.

Here is what I tried, very basic:

System.out.println("Creating Python context..");
try (Context context = Context.newBuilder("python").allowAllAccess(true).build()) {
    System.out.println("Make installed Python packages available..");
    context.eval("python","import site");
    System.out.println("Import robot..");
    context.eval("python", "from robot import run_cli");
    System.out.println("Run tests..");
    context.eval("python", "run_cli(['--log', 'none', '--report', 'none', 'Simple.robot'], exit=False)");
}

Performance wide, far from native Python, but quite faster than Jython (not exactly a fair comparison.. but anyway)

Complete RF acceptance tests results: GraalPy-3.10.7-Linux.zip Python-3.10.6-Linux.zip StandaloneJAR-2.7.2-Linux.zip

[msimacek]

Thank you for the nice summary :slightly_smiling_face:

I did not see such issues in the Jython execution (plenty of different issues there), so unclear to me if that is something that can be fixed in Graal?

It looks like something that should be fixed on our side, but I cannot say for sure just from the traceback. Could you please post a command how to run one such test individually? I'll try to debug the problem

[vehovsky]

Thanks for looking into this.

What I've done is to grab the 4.1-maintenance sources. Then if you want to execute just single test, use this option

--test <name>

Example:

atest/run.py python --test "Robot.Keywords.Type Conversion.Annotations.Bytearray" atest/robot

Output:

martin@EU-603F1F3-L:~/playground/robotframework-4.1-maintenance$ atest/run.py python --test "Robot.Keywords.Type Conversion.Annotations.Bytearray" atest/robot
GraalPy 3.10.7 on Linux
-----------------------

Running command:
/home/martin/playground/graalvm-ce-java17-23.0.0-dev/bin/python3 /home/martin/playground/robotframework-4.1-maintenance/src/robot/run.py --doc Robot Framework acceptance tests --metadata interpreter:GraalPy 3.10.7 on Linux --variablefile /home/martin/playground/robotframework-4.1-maintenance/atest/interpreter.py;python;GraalPy;3.10.7 --pythonpath /home/martin/playground/robotframework-4.1-maintenance/atest/resources --outputdir /home/martin/playground/robotframework-4.1-maintenance/atest/results/GraalPy-3.10.7-Linux --splitlog --console dotted --consolewidth 100 --SuiteStatLevel 3 --TagStatExclude no-* --exclude require-lxml --exclude require-jython --exclude require-ipy --exclude require-py2 --exclude require-windows --test Robot.Keywords.Type Conversion.Annotations.Bytearray atest/robot

Running suite 'Robot' with 1 test.
====================================================================================================
F
----------------------------------------------------------------------------------------------------
FAIL: Robot.Keywords.Type Conversion.Annotations.Bytearray
Parent suite setup failed:
Processing output failed: ParseError: no element found: line 964, column 5
====================================================================================================
Run suite 'Robot' with 1 test in 20 seconds 498 milliseconds.

FAILED
1 test, 0 passed, 1 failed

Output:  /home/martin/playground/robotframework-4.1-maintenance/atest/results/GraalPy-3.10.7-Linux/output.xml
Log:     /home/martin/playground/robotframework-4.1-maintenance/atest/results/GraalPy-3.10.7-Linux/log.html
Report:  /home/martin/playground/robotframework-4.1-maintenance/atest/results/GraalPy-3.10.7-Linux/report.html

[msimacek]

Just to provide an update - we've been actively looking at the failures and fixing the most common ones. I think next week we should have a new snapshot for testing.

[pekkaklarck]

This looks really interesting! Some quick comments:

• Robot Framework supporting GraalPython would be awesome! Unfortunately Robot Framework Foundation that sponsors our development doesn't have that much resources right now, but I believe I could spend some time for this myself as well at least next year. If we would get some external sponsorship, that would obviously help a lot.
• Getting GraalPython running on RF 4.1 which still has Jython support would be a great step forward and I'm sure would help with GraalPython's Jython compatibility. I would hope we could make the current Robot Framework code GraalPython compatible as well, though.
• If GraalPython is compatible with CPython, then Robot should run on it pretty much directly. We don't, for example, have that much PyPy related code. Unless your are able to make Java classes look exactly like Python classes, that wouldn't make implementing libraries with Java possible, though.
• If we were to add Java library support back, it would probably make sense to rewrite it instead of using the old code that worked with Jython. That means GraalPython wouldn't need to support all same low level introspection APIs as Jython.
• Related the above, we had separate argument conversion logic for Java based libraries. If Java libraries are to be supported again, I believe the argument conversion logic should be integrated with the newer (and better) argument conversion system that works based on Python type hints.
• Also related to the above, Robot's internal library inspection code isn't great. It probably would make sense to rewrite (big) parts of it if Java library support is added.
• Timeouts can be considered an optional Robot feature. We could easily change our code so that if importing stuff timeouts needs from signal fails, timeouts are just disabled.
• If we are doing something strange on Robot side that isn't compatible with GraalPython, we can consider changing our logic as well. Changes are possible both in the 4.1-maintenance branch and in master, but in general I hope we could concentrate on the latter.

[msimacek]

Thank you for the comments.

Getting GraalPython running on RF 4.1 which still has Jython support would be a great step forward and I'm sure would help with GraalPython's Jython compatibility. I would hope we could make the current Robot Framework code GraalPython compatible as well, though.

I tried to look at the 4.1 branch and the Java detection there is too Jython-specific - it outright expects the name in python --version to be Jython and we cannot claim to be Jython. We would need a more generic interpreter definition.

If GraalPython is compatible with CPython, then Robot should run on it pretty much directly. We don't, for example, have that much PyPy related code. Unless your are able to make Java classes look exactly like Python classes, that wouldn't make implementing libraries with Java possible, though.

It seems to run quite fine already. I'll have to play a bit with how the Jython integration worked, I haven't really tried it much so far. We can make Java classes look like Python's to some extent, but there will always be limitations, like for example you wouldn't be able to add type annotations on a Java method. I expect we will always need at least a thin python wrapper around the actual Java library object.

Timeouts can be considered an optional Robot feature. We could easily change our code so that if importing stuff timeouts needs from signal fails, timeouts are just disabled.

We have already added setitimer, so timeouts work, although they don't yet trigger in all cases (signal handling on Java is tricky). Interrupting a python loop or a subprocess works.

[msimacek]

@vehovsky A new snapshot with fixes should be available. Could you please give it a try again?

[pekkaklarck]

It definitely sounds like we should forget the old Jython specific library API and write a new one for GraalPython. The more the Java classes look like normal Python classes the easier that is. Having some Java/Graal specific code isn't a problem.

How is Java type information available currently? I assume it cannot be in __annotations__ as it maps argument names to types and argument names aren't, to my knowledge, available at the execution time with Java. Having types available as a list in the same order as in the signature would be enough for us. Not sure how that should work with overloaded methods, though.

[msimacek]

I think the way to go about obtaining type information would be to use the Java reflection APIs from python, i.e. to call getDeclaredMethods on the class object, iterate those and then call getParameterTypes on the methods you find.

Example:

def get_method_types(clazz, method_name):
    overloads = []
    for method in getattr(clazz, 'class').getDeclaredMethods():
        # will return just the first overload
        if method.getName() == method_name:
            overloads.append(method.getParameterTypes())
    if not overloads:
        raise AttributeError(method_name)
    return overloads

import java
types = get_method_types(java.type('java.lang.StringBuilder'), 'append')
for overload in types:
    print([t.getSimpleName() for t in overload])

It prints:

['StringBuffer']
['CharSequence']
['CharSequence', 'int', 'int']
['char[]']
['CharSequence']
['CharSequence', 'int', 'int']
['Object']
['String']
['int']
['long']
['float']
['double']
['char']
['boolean']
['char[]', 'int', 'int']
['CharSequence', 'int', 'int']
['char[]']
['char[]', 'int', 'int']
['boolean']
['CharSequence']
['StringBuffer']
['String']
['float']
['double']
['Object']
['char']
['char']
['int']
['long']

[vehovsky]

Hi @msimacek,

I've tried running again all the the Robot Framework acceptance tests on 4.1-maintenance branch. And the results are pretty impressive!

GraalPy-3.10.8-Linux.zip

I'll try to run the acceptance tests now also on the latest stable version of RobotFramework 6.0.1 and get back to you with the results.

[msimacek]

Thank you for the results. I think the lxml tests could run too if you install lxml into the virtualenv. The requirements file has this:

lxml; platform_python_implementation == 'CPython'

so it doesn't get installed on graalpy by default.

[vehovsky]

Thanks for the tip.

Here are my results for acceptance tests on RobotFramework 6.0.1

GraalPy-3.10.8-Linux.zip Python-3.10.6-Linux.zip

Looks very very good to me. What do you think @pekkaklarck?

Thank you @msimacek!

[pekkaklarck]

Do I get it right that with GraalPython you get 75 failures and with CPython you get 160? The former number is really promising. There likely are tests that would need to be updated to take GraalPython into account and tests for optional dependencies (e.g. docutils) can be ignored. You can actually exclude such tests altogether by using tests, for example, with --exclude require-docutils. For more details about dependencies see atest/README.rst.

160 tests failing with CPython is strange because all tests should pass. Have you looked at why they are failing? If there's some system level dependency missing, that could also affect GraalPython execution.

Anyway, there clearly has been awesome progress. I need to install latest GraalPython version myself and run test myself so that it's easier to debug them. Unfortunately I'm super busy right now, but I hope I'd have time for that after Christmas or latest early next year. If things look good enough, we could add official GraalPython support into RF 6.1 scope. It wouldn't contain the Java library API, but we can start discussing about that as well.

Update: I took a quick look at the provided CPython results and majority of the failures are due to python not being executable in the system. There are, for example, Process library tests that run something like python -c print('Hello') and they all fail. Most likely you only have python3. Could you configure the system to also have python? This probably should be mentioned in the aforementioned atest/README.rst.

[vehovsky]

@pekkaklarck you are right, sorry about that, still 34 failing though.. Python-3.10.6-Linux.zip