Importing Sikuli is causing a Jenkins Failure!

[mzbedat]

Hi, A strange behavior that I'm facing when running my sikuli tests from jenkins... The test cases themselves are passing, but at the end of the job jenkins is loosing the connection with the client:

Process leaked file descriptors. See (http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information) FATAL: Remote call on AU_9 failed java.io.IOException: Remote call on AU_9 failed at hudson.remoting.Channel.call(Channel.java:789) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:953) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:540) at hudson.model.Run.execute(Run.java:1738) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410)

I followed the link in the error and there they are explaining about this problem and how it can be avoided (by starting jenkins with processTree disabled or by creating a deamon\service that is runing all the time and preventing from killing the child job).

However, my concern is why that is happening only when I do import for the sikuli in my test suite (even without running the TCs)?

Regards.

[rainmanwy]

Could you provide a simple example of your test suite.

For this library, when RF import this library, a java process will be started, and this java process need be stopped by keyword "stop remote server".

From you description, i am not sure whether it is because you did not stop the java process in your suite.

[mzbedat]

Hi, Nothing special in my test suite, and I have "Stop Remote Server" in my TearDown! Anyway, I solved it by adding this Environment variable to the job: BUILD_ID=dontKillMe

[rainmanwy]

Ok.

When RF with SikuliLibrary is started, there are python process and java process. Maybe you could check the existed processes when test suites are done in jenkins build.

Thank you very much!

[mzbedat]

One more thing please... how can I prevent sikuli from generating all the sikuli_java_stdout \ sikuli_java_stderr txt files?

[rainmanwy]

Sorry, it could not be yet. You may clean workspace of jenkins job each build. Or these logs are not in your workspace folder?

[jpeltonen]

There is a warning about not using stdout and stderr files also in Robot Manual:

Standard output and error streams

By default processes are run so that their standard output and standard error streams are kept in the memory. This works fine normally, but if there is a lot of output, the output buffers may get full and the program can hang. Additionally on Jython, everything written to these in-memory buffers can be lost if the process is terminated.

You could remove logging by modifying start_process in sikuli.py by removing stdout and stderr arguments (it could also be parameterized by constructor variable to choose whether to log or not):

if(self.logToFile == True):
     process.start_process(command, shell=True, stdout=self._output_file(), stderr=self._err_file())
else: 
    process.start_process(command, shell=True)

Also noticed that shell=True argument in start_process is not recommended by Robot manual:

Giving the shell argument any non-false value, such as shell=True, changes the program to be executed in a shell. It allows using the shell capabilities, but can also make the process invocation operating system dependent. Having a shell between the actually started process and this library can also interfere communication with the process such as stopping it and reading its outputs. Because of these problems, it is recommended to use the shell only when absolutely necessary.

Is it possible to remove the shell=True argument?

[rainmanwy]

@jpeltonen, thanks for these information.

For stdout and stderr, i have met hang problem if process's stdout and stderr are configured to sys.stdout and sys.stderr. But for file, it should be a problem, as the messages will be written to file at runtime. And this is also described in Process lib.

To avoid the above mentioned problems, it is possible to use ``stdout``
and ``stderr`` arguments to specify files on the file system where to
redirect the outputs. This can also be useful if other processes or
other keywords need to read or manipulate the outputs somehow.

@mzbedat , whether you have several test suites, and each test suites will create their own sikuli log file. If so, i may add a parameter to disable it as @jpeltonen suggested.

About "shell=True", i will check this later. Thanks again, @jpeltonen

[jpeltonen]

I would like generate sikuli log files other folder not the folder sikuli library is started from, because I have to delete sikuli logs manually after some time when there are too many of them. It would be easier if they were in other directory than my code. Could it be configurable where to generate sikuli logs?

Maybe it can be done with start_process argument cwd?

Current working directory

By default the child process will be executed in the same directory as the parent process, the process running tests, is executed. This can be changed by giving an alternative location using the cwd argument. Forward slashes in the given path are automatically converted to backslashes on Windows.

Standard output and error streams, when redirected to files, are also relative to the current working directory possibly set using the cwd argument.

Example: Run Process prog.exe cwd=${ROOT}/directory stdout=stdout.txt

[rainmanwy]

There is a bug in code, these log file should be in "OUTPUTDIR" of robot, that mean it should be in same folder as robot log, report and output file. But actually it is in curdir.

def _output_file(self):
    outputDir = self._get_output_folder()
    outputFile = 'Sikuli_java_stdout_'+str(time.time())+'.txt'
    return outputFile

def _err_file(self):
    outputDir = self._get_output_folder()
    errFile = 'Sikuli_java_stderr_'+str(time.time())+'.txt'
    return errFile

def _get_output_folder(self):
    outputDir = os.path.abspath(os.curdir)
    try:
        outputDir = BuiltIn().get_variable_value('${OUTPUTDIR}')
    except Exception, err:
        pass
    return outputDir

whether this folder is ok for you?

btw, i have plan to add a keyword to start sikuli java process, so that it is not needed to start it in "init". Right now, if there are two suites, this library will only be init once as its' scope is "GLOBAL". I think this is a problem.

[mzbedat]

Please note that in addition, for each run there is a cmd.exe & conhost.exe & java.exe processes that is opened in windows and not killed... this cause a memory\performance leak after a while.

[rainmanwy]

@mzbedat, how many suites in your project, and how SikuliLibrary is imported in RF(in suite file, or _init file). As this library is "GLOBAL" scope, RF only init it once(for same init parameters), so should only one java process. And java process could be stopped by call keyword "stop remote server". If this keyword is called in teardown, i think java process should not be there as expected.

[jpeltonen]

@rainmanwy I have all my logs in some directory which is created for every RF test suite run. I would like to put Sikulijava... logs there also. I also had problems how to redirect Sikuli captured images to RF OUTPUTDIR.

I think this will fix it.

[rainmanwy]

@jpeltonen, how do you execute thest test suites? In separate pybot, or just one pybot to execute all test suites?

def _start_sikuli_java_process(self):
    libFolder = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'lib')
    jarList = glob.glob(libFolder+os.sep+'*.jar')
    if len(jarList) != 1:
        raise Exception('Sikuli jar package should be exist in lib folder')
    sikuliJar = jarList[0]
    command = 'java -jar '+sikuliJar+' %s ' % str(self.port)+self._get_output_folder()
    process = Process()
    process.start_process(command, shell=True, stdout=self._output_file(), stderr=self._err_file())
    self.logger.info('Start sikuli java process on port %s' % str(self.port))
    self._wait_process_started()
    self.logger.info('Sikuli java process is started')

def _get_output_folder(self):
    outputDir = os.path.abspath(os.curdir)
    try:
        outputDir = BuiltIn().get_variable_value('${OUTPUTDIR}')
    except Exception, err:
        pass
    return outputDir

As expect, captured images should be in "OUTPUTDIR" folder. If in you environment, it is not in OUTPUTDIR, there is really a problem. Could you provide more information? Thanks!

[jpeltonen]

outputdir is provided by parameter for separate pybot call for every test suite.

[rainmanwy]

So captured images are not in outputdir?

[jpeltonen]

Haven't tested with your fix yet.

[mzbedat]

Hi, Regarding your question: "how many suites in your project, and how SikuliLibrary is imported in RF(in suite file, or _init file). As this library is "GLOBAL" scope, RF only init it once(for same init parameters), so should only one java process. And java process could be stopped by call keyword "stop remote server". If this keyword is called in teardown, i think java process should not be there as expected."

I have only one suite, and SikuliLibrary is imported from inside a Keywords.txt file: Library SikuliLibrary

I tried simulating it when running from eclipse and when running from Jenkins and here are my findings:

From Eclipse:

a cmd.exe and conhost.exe processes are started, and they are not killed after the suite is finished to run. They are killed only when I press the "Terminate" red square button of stopping the test run. Once I click it both processes are gone.

From Jenkins:

cmd.exe and conhost.exe are not killed (despite having the "Stope Remote Server" in Teardown) In order to handle it I'm running a cmd command for doing it:

wmic Path win32_process Where "CommandLine Like '%SikuliLibrary.jar%'" Call Terminate Note: In this case, I need to disable the "Stop Remote Server" from teardown, cause process is already kileed

The problem is that I cant do somthing similar for the conhost, cause there is a conhost process that is needed for jenkins and I cant identify which one is it!

[rainmanwy]

So you create a windows batch script to run RF test suite, and conhost&cmd processes are started by jenkins job, am i correct? I will install a jenkins server to have a try this.