robotframework-sikulixlibrary

The all new, modern, SikuliX Robot Framework library for Python 3.x, based on JPype or Py4J Python modules.

It can be used with e.g. Robot Framework projects (https://robotframework.org), Robocorp projects (https://robocorp.com) - the easiest approach for beginners or with pure Python projects. Either for Test Automation type of projects, or for open source RPA (Robot Process Automation) projects.

JPype is a Python module to provide full access to Java from within Python.

Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a JVM.

This library is a wrapper to SikuliX that is exposing Java functions as Robot Framework keywords, and it can be enabled to use by choice any of the JPype or Py4J modules. This is done by creating SIKULI_PY4J environment variable and setting to 1. When not defined or set to 0, JPype is used instead. Please note that on MacOS, only Py4J can be used, while on Windows or Ubuntu, any of them is working.

While in the past the only approach to use Sikuli functionality within Robot Framework was through Remote Server with XML RPC interface (SikuliLibrary), the aim of this library is to replace that approach and make it a lot easier to use SikuliX within Robot Framework projects with a simple Library statement (i.e. no need to start remote server and so on).

Also with this implementation is very easy to extend the library with new custom keywords, for example with the purpose to create migration classes to help migrate from current Sikuli libraries or other image recognition alternatives. For practical examples check migrate folder.

See keyword documentation.

Installation instructions (Windows)

1. Python 3.8 or newer, as supported by JPype or Py4J
2. JPype 1.4.1 or newer and JPype project dependencies as explained on project page: https://github.com/jpype-project/jpype
   • Install Java 8 or newer
   • While not mentioned on JPype page, on a new Windows 10 machine also Visual C++ Redistributable 2015 and newer are needed (e.g. vc_redist.x64.exe)
3. or Py4J 0.10.9.7 or newer
4. SikuliX as a standalone jar from project page: https://raiman.github.io/SikuliX1/downloads.html
   • Put jar file in any local directory (e.g. C:\sikulix\sikulix.jar)
   • Install Tesseract 4 for using OCR functionality
   • Py4J server is enabled from SikuliX 2.0.5 onward and currently advertised as experimental. However, this library is working as expected with Py4J.
   • Recommended to use environment variable SIKULI_HOME that point to sikulix local directory
5. pip install robotframework-sikulixlibrary
6. Check Known issues section below if libspec is not generated within your environment (RF IDE tool) and thus library keywords are not recognised

While JPype JVM is always started automatically, Py4J JVM can be started manually or automatically. To start manually, use the command:

java -jar sikulix.jar -p (to start Py4J server) or java -jar -DsikuliDebug=3 sikulixide.jar -p (useful e.g. for checking sikulix debug info)

Installation instructions (MacOS)

Mainly the same steps as for Windows above. However, installing Tesseract 4 seems to be challenging since e.g. brew install tesseract will install Tesseract 5. But at least the following method from stackoverflow for installing Tesseract 4.1.3 is working.

Examples

Testing with Robot Framework

*** Settings ***
Library   SikuliXLibrary    sikuli_path=sikulixide-2.0.5.jar

*** Test Cases ***
Example Test
    imagePath add   ${my_path}
    settings set    MinSimilarity  ${0.9}
    app open        C:/Windows/System32/notepad.exe
    region wait     iNotepad.PNG
    region paste    Welcome!

Testing with Python.

from SikuliXLibrary import SikuliXLibrary
sikuli_path = 'sikulixide-2.0.5.jar'
lib = SikuliXLibrary(sikuli_path)
lib.imagePath_add('my_path')
lib.settings_set('MinSimilarity', float(0.9))
lib.app_open("C:\\Windows\\System32\\notepad.exe")
lib.region_wait('iNotepad')
lib.region_paste('Welcome!)

Testing

Git clone, and if not using pip install for this library, then just point PYTHONPATH to local robotframework-sikulixlibrary folder and execute:

python testlibrary_win.py (or any .py file from under test directory and for OS of choice

robot --outputdir results/default test_defaultlibrary_win.robot (or any .robot file from under test directory and for OS of choice)

or maybe on MacOS

python3 -m robot --outputdir results/default test_defaultlibrary_osx.robot

Obviously, image files from test/img/MacOS, Ubuntu or Windows might not work on specific environment and would need to be regenerated. Also for these tests SIKULI_PATH is defined and the name of SikuliX is sikulixide-2.0.5.jar

Additionally, debugging with some RF supported tools is also possible with this library, for both Robot Framework and Pyton code. Python library debugging was tested with Visual Studio Code with Robot Framework Language Server by Robocorp, by using debug test.py file. Also Robot Framework test code from within test directory was tested with debugging, with the same tool, by creating a specific configuration within launch.json file (VSCode specific file).

• Tested on Windows 10 with 1920x1080, Windows Dark mode, Light app mode. Also limited testing with 4k.
• Tested on MacOS Big Sur with 1440x900, Dark Appearance. Also limited testing with 4k. Text Edit should be put in Plain Text mode.
• Tested on Ubuntu with 1920x1080 and with Leafpad application.

Supported Operating Systems

1. Windows 10

   • supported, tested with both JPype and Py4J

2. OSX

   • SikuliX works under OSX, however currently there are issues with JPype generally working under OSX: https://github.com/jpype-project/jpype/issues/911
   • Py4J tested and working under MacOS and always enabled without definding SIKULI_PY4J environment variable. However, forcing JPype for experimental purpose is possible with environment variable SIKULI_PY4J=0.

3. Linux

   • supported, tested with Ubuntu 20.04 and Leafpad application. Tested with both JPype and Py4J.
   • due to https://github.com/RaiMan/SikuliX1/issues/438, openApp is not currently working with SikuliX 2.0.5, thus it is disabled in the test .py and .robot code for Ubuntu. This means you have to start the test and open manually Leafpad app in order for tests to succeed.
   • tested with: python3.8, default-jre (openjdk-11-jre), libopencv4.2-java as explained on SikuliX support page, gnome-panel, pip install robotframework-sikulixlibrary
   • start the tests with e.g. python -m robot --outputdir results/ubuntu test_defaultlibrary_ubuntu.robot or python testlibrary_ubuntu.py

Known Issues

• When using Py4J, libdoc will not generate library documentation within IDE, e.g. Visual Studio Code with Robocorp extension (Robot Framework Language Server). The workaround is to start manually SikuliX Py4J server (see above or run java -jar sikulix.jar -p), then reload the library to generate the keyword documentation. If not using pip install for this library, then the extension settings.json need to have pythonpath set to local robotframework-sikulixlibrary folder.

• When generating library documentation within IDE, the library is instantiated with no arguments and it will look for sikulix.jar file within SIKULI_HOME environment variable defined directory, so make sure this file is there even if it is a duplicate of e.g. sikulixide-2.0.5.jar. Otherwise, it is also possible to manually generate the libspec with the following command, and copy it under project folder or PYTHONPATH:

python -m robot.libdoc SikuliXLibrary::sikuli_path=path\to\sikulixide-2.0.5.jar SikuliXLibrary.libspec

See this documentation for reference: https://github.com/robocorp/robotframework-lsp/blob/master/robotframework-ls/docs/faq.md