Jenkins does fails on bot runs

[andrawaag]

Since bot #22 of GeneBot_microbes jenkins does not successfully complete.

• An initial test on an unrelated bot (Wikipathways) showed the exact same behaviour (Fail), suggesting a system issue and not a bot issue
• First thought of memory issues, since some of the generated output/logfiles can be substantial. Manually removing some of the bot runs from before 2018 did not lead to success
• I have set jenkins to only keep 10 builds Clearing disk space by removing old builds did not solve the issue

Since the issue seems to surface "naturally", i.e. no specific updates I am aware of. I suspected an automatic update issue similar to the pyshex downstream update earlier this month.

• I updated jenkins using "yum update jenkins" through ssh in the CLI (jenkins.sulab.org)
• I upgraded the underlying system

The issue persisted. I was able to reproduce the error outside jenkins, running part of the script on the CLI. The error message is a bit more elaborate than within jenkins, yet remains unhelpful in finding sollution on google or stackoverflow.

After installing the first required python library pip install wikidataintegrator the following error message appears

Installing collected packages: six, python-dateutil, oauthlib, simplejson, chardet, urllib3, idna, certifi, requests, jsonasobj, isodate, pyparsing, rdflib, antlr4-python3-runtime, dataclasses, pyjsg, rdflib-jsonld, pyshexc, ShExJSG, sparqlwrapper, sparql-slurper, CFGraph, pyshex, requests-oauthlib, PyJWT, mwoauth, wikidataintegrator
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/var/lib/jenkins/workspace/venv/lib/python3.6/site-packages/six.py'
Consider using the `--user` option or check the permissions.

The issue appears to be with pip.

[andrawaag]

I was able to reproduce the issue outside jenkins and the system jenkins runs on. This bug has nothing to do with jenkins, but is a wdi pypi issue.

[andrawaag]

Closing this issue was premature. The issue was reproduced due to not using sudo.

[andrawaag]

The issue might have to do with virtualenv Jenkins seems to choke on

New python executable in /home/ec2-user/venv/bin/python36
Not overwriting existing python script /home/ec2-user/venv/bin/python (you must use /home/ec2-user/venv/bin/python36)
Installing setuptools, pip, wheel...done.

Running the bot steps out of jenkins, runs the bot as expected.

#!/bin/bash
pwd
echo $WDUSER
echo "0"
virtualenv -p python36 venv
echo "1"
source venv/bin/activate
echo "2"
pip install --upgrade pip
echo "3"
pip install wikidataintegrator

git clone https://github.com/sulab/scheduled-bots.git
cd scheduled-bots
pip install -r requirements.txt
python3 setup.py install

cd ..

cd scheduled-bots/scheduled_bots/geneprotein/

python3 GeneBot.py --taxon 243161,microbe
python3 ../logger/bot_log_parser.py logs

I added echo "1", "2", "3" as markers to identify where the script chokes in jenkins. Only the message "0" is printed suggesting an issue with virtualenv

Started by user Jenkins Admin
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/GeneBot_microbes
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Done
[GeneBot_microbes] $ /bin/bash /tmp/jenkins2304417844414179954.sh
/var/lib/jenkins/workspace/GeneBot_microbes
MicrobeBot
0
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

[andrawaag]

I tried to move all the tasks to a clean install of Jenkins on an Ubuntu system. This inspired by this blog. The effort failed because it turns out that in /var/lib/jenkins system-specific parameters are stored as well. Since I installed a fresh Jenkins on Ubuntu and sulab.jenkins.org is running on an amazon linux instance this effort didn't go well. . I am running Jenkins on a fuga.cloud, where I can't instantiate an amazon Linux instance.

[stuppie]

I created a Dockerfile that sets up Jenkins, the required plugins, and example jobs. Here: https://github.com/SuLab/wikibase-jenkins/tree/master/genewiki-jenkins Maybe this will be of some help ...

[andrawaag]

On March 22nd without any major fixes, Jenkins started running our bots again, successfully. No idea, why it did and also what it caused