Tests are failing upon compiling and docker image building (2.6.0)

[andrese02]

Main goal is to run version 2.6.0 inside docker.

Environment:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

java --version
java 11.0.17 2022-10-18 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.17+10-LTS-269)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.17+10-LTS-269, mixed mode)

docker --version
Docker version 20.10.17, build 100c70180f

---

Code checkout and init

> mkdir siga_2.6.0
> cd siga_2.6.0
> git clone git@github.com:open-eid/SiGa.git .
Cloning into '.'...
remote: Enumerating objects: 14118, done.
remote: Counting objects: 100% (916/916), done.
remote: Compressing objects: 100% (328/328), done.
remote: Total 14118 (delta 509), reused 811 (delta 465), pack-reused 13202
Receiving objects: 100% (14118/14118), 31.65 MiB | 4.83 MiB/s, done.
Resolving deltas: 100% (5508/5508), done.

> git status
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean

> git fetch --all --tags
Fetching origin

> git checkout tags/v2.6.0 -b v2.6.0-branch
Switched to a new branch 'v2.6.0-branch'

> git status
On branch v2.6.0-branch
nothing to commit, working tree clean

> git log -n 1
commit 4858a25ae4062a4cacad0edbc3f2e48bf04e8ae6 (HEAD -> v2.6.0-branch, tag: v2.6.0, origin/master)
Author: Mart Aarma <mart.aarma@nortal.com>
Date:   Fri Jan 20 17:49:15 2023 +0200

    SIGA-452 Fix statistics logging by adding security context to async MID/SID polling tasks

---

Compiling

./mvnw clean install

[INFO]
[ERROR] Tests run: 196, Failures: 2, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Signature Gateway 2.6.0:
[INFO]
[INFO] Signature Gateway .................................. SUCCESS [  4.729 s]
[INFO] siga-common ........................................ SUCCESS [ 53.770 s]
[INFO] siga-auth .......................................... SUCCESS [02:02 min]
[INFO] siga-session-storage ............................... SUCCESS [  2.607 s]
[INFO] siga-monitoring .................................... SUCCESS [ 19.281 s]
[INFO] siga-signature-service ............................. FAILURE [05:41 min]
[INFO] siga-webapp ........................................ SKIPPED
[INFO] siga-test .......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  09:07 min
[INFO] Finished at: 2023-01-31T14:56:50Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project siga-signature-service: There are test failures.

Failed tests reports: siga_2.6.0\siga-signature-service\target\surefire-reports\ee.openeid.siga.service.signature.container.status.SessionStatusReprocessingServiceTest.txt 20230201_01_00_SessionStatusReprocessingServiceTest.txt siga_2.6.0\siga-signature-service\target\surefire-reports\ee.openeid.siga.service.signature.SignatureFinalizingTest.txt 20230201_01_00_SignatureFinalizingTest.txt

---

Simply disabled the said tests (commented out from the Java code) and recompiled. All other tests succeeded and compilation was finished.

---

Build docker image

./mvnw spring-boot:build-image -pl siga-webapp

[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   SmartIdApplicationTests.smartIdFlowWihCertificateChoice:104->BaseTestLoggingAssertion.assertInfoIsLoggedOnce:43->BaseTestLoggingAssertion.assertMessagesAreLoggedOnceInRelativeOrder:61 » ConcurrentModification
[INFO]
[ERROR] Tests run: 91, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  05:39 min
[INFO] Finished at: 2023-01-31T15:56:23Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project siga-webapp: There are test failures.

Failed tests report: siga_2.6.0\siga-webapp\target\surefire-reports\ee.openeid.siga.SmartIdApplicationTests.txt 20230201_01_01_SmartIdApplicationTests.txt

[heititobi]

Thank you for the extensive report and we are taking a deeper look in these as we are having hard time reproduce the errors. SignatureFinalizingTest we were able to repeat and looking into fixing it. Also would it be possible for you to try in SessionStatusReprocessingServiceTest class increase assertMaxReprocessingAttempts timeout values (e.g. from 60 yo 180) and see if that´s any help?

PS! There is a paremeter skipTests=True what can be used for compiling and image building without running tests, e.g. ./mvnw clean install -DskipTests=True.

[andrese02]

I tested the timeout "fix" and it worked. This leads me to believe that SessionStatusReprocessingServiceTest results are actually timeout related. As i was testing and trying to get the docker up and running, i recompiled and ran the tests multiple times. Sometimes the same tests would fail, other times not, sometimes no tests failed. There were couple of other tests that behaved similary, but unfortunately i wasn't able to reproduce the same result multiple times. Same goes to the SmartIdApplicationTests that seemed to fail at random intervals

So i assume that most of the failures are related to network speed or external servers load.