get_unit_test_report.py: no longer running correctly

[geissonator]

https://jenkins.openbmc.org/job/latest-unit-test-coverage/ is still running but the data is not valid anymore. Everything is either ERROR or SKIPPED for the most part.

The unit test and coverage works fine when run-unit-test-docker.sh is run so my guess is something changed in the output of the unit test framework that is not handled by get_unit_test_report.py

Here's the output under a build/meson-logs/ dir:

build/meson-logs/
coverage.info          coverage.info.raw  coveragereport   meson-log.txt  testlog.junit.xml  testlog-ubasan.json       testlog-ubasan.txt     testlog-valgrind.junit.xml
coverage.info.initial  coverage.info.run  install-log.txt  testlog.json   testlog.txt        testlog-ubasan.junit.xml  testlog-valgrind.json  testlog-valgrind.txt

@lkammath is this something you could help look into?

[adathatri]

@geissonator get_unit_test_report.py is unable to generate the UT report due to the below error. Hence all the repo's which are not skipped or archived are reported as "ERROR".

+ echo 'Building docker image with build-unit-test-docker'
Building docker image with build-unit-test-docker
+ export BRANCH
++ ./scripts/build-unit-test-docker
  File "./scripts/build-unit-test-docker", line 52
    rev: str
       ^
SyntaxError: invalid syntax
+ DOCKER_IMG_NAME=
DEBUG - WORKSPACE=$(pwd) UNIT_TEST_PKG=hiomapd ./openbmc-build-scripts/run-unit-test-docker.sh
DEBUG - UT BUILD EXITED FOR: hiomapd

[geissonator]

Thanks for taking a look @adathatri . That script, build-unit-test-docker, is utilized by our CI job out at https://jenkins.openbmc.org/job/ci-repository/ and in a number of other areas so I think your error may be more your system setup. Maybe a downlevel docker version?

[williamspatrick]

It sounds like a version of Python that is old enough that it doesn't support type identifiers? You probably need 3.6 or higher if I had to guess for that script to work.

[adathatri]

@geissonator I assume that the existing docker image on Jenkins node has an older version of Python and this is preventing the execution of build-unit-test-docker in order to build a new image. get_unit_test_report.py uses run-unit-test-docker.sh to build docker image and run unit tests which in turn uses build-unit-test-docker to build the image.

[williamspatrick]

@geissonator Oh. This job is restricted to builder5 which has an old version of Python. We need to move it to a different system (why is it restricted at all?).

[williamspatrick]

(I would think we would use the same Jenkins tag we use for CI jobs.)

[geissonator]

@geissonator Oh. This job is restricted to builder5 which has an old version of Python. We need to move it to a different system (why is it restricted at all?).

Some of the build machines didn't even have python installed (or had even older versions). We should probably move this script to a docker container. I updated the job to use one of the new nodes, lets see if that helps. https://jenkins.openbmc.org/job/latest-unit-test-coverage/247/console

[geissonator]

I had to update the jenkins job to call the script using "python3" since our new nodes only have python3. We hit this though:

+ python3 ./get_unit_test_report.py ./unit-test-meta-data
Number of repositories (Including archived): 164
1 in 164 completed
2 in 164 completed
Traceback (most recent call last):
  File "./get_unit_test_report.py", line 283, in <module>
    if result.__contains__("coveragereport"):
TypeError: a bytes-like object is required, not 'str'
Build step 'Execute shell' marked build as failure

Probably a python3 nuance that needs updating in the script?

[lkammath]

Yes. This would need to updated as per python3 coding guidelines

[williamspatrick]

Some of the build machines didn't even have python installed (or had even older versions). We should probably move this script to a docker container.

The script itself turns around and runs Docker. I don't think we support Docker-in-Docker do we? (That is hard to set up and introduces security issues, so most people don't do it).

[geissonator]

The script itself turns around and runs Docker. I don't think we support Docker-in-Docker do we? (That is hard to set up and introduces security issues, so most people don't do it).

Yeah, good point. Probably best to just get this script working with python3 and I'll make a new "python3" label in jenkins that we can put on the nodes that support it.

[geissonator]

Got this one working by just hard coding the job to use builder_fb_c0. Keep this issue open to make this a bit more dynamic.

[williamspatrick]

Did adding the tag not work?