search

RotherOSS / otobo

OTOBO is one of the most flexible web-based ticketing systems used for Customer Service, Help Desk, IT Service Management. https://otobo.io/
GNU General Public License v3.0
247 stars 71 forks source link

Resolve unit test failures when syncing with S3 is enabled #1777

Closed bschmalhofer closed 1 year ago

bschmalhofer commented 2 years ago

Currently there are still 23 test scripts that are failing when syncing with MinIO is activated. These need to be acivated.

One case is scripts/test/Console/Command/Maint/Ticket/InvalidUserCleanup.t . In this the following activities take place.

  1. A ZZZZUnitTest*.pm file. that sets the temporary article dir, is written to S3
  2. The system time is set to '2019-06-15 00:00:00'
  3. A ZZZZUnitTest*.pm file, that activates Ticket::Watcher, is written to S3

Activity 3. fails with the not very helpful status code 403 and the message "Forbidden". That the activity fails is fine as the S3 interface implements a timestamp sanity check. See "Time stamp requirement" in https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html.

bschmalhofer commented 2 years ago

See also #769.

bschmalhofer commented 2 years ago

Fixed the first batch of test failures. Let's do a new run before proceeding. ... Looks good so far, down to 14 failing scripts

bschmalhofer commented 2 years ago

Same of the tests are failing because of expired test certificates. See #1779. So, I'm ignoring the failures in

for now,

@svenoe : Do you have insight whether certificates need to be synced via S3?

bschmalhofer commented 2 years ago

There are failures with the test test script scripts/test/Selenium/Agent/Admin/AdminLog.t . These failures are kind of random, as errors appear at different lines and sometimes the test scripts succeeds. This is not surprising. Kernel/Modules/AdminLog.pm uses `Kernel::System::Log::GetLog()_ which reads a copy of the log from shared memory. The web containers do not share their shared memory per default. Thus it depends on the container what the content in IPC shared memory is.

There is a setup where containers share their shared memory, but IMHO this is against the spirit of containerisation, https://docs.docker.com/engine/reference/run/#ipc-settings---ipc . Thus we simply skip the test when OTOBO_SYNC_WITH_S3 is active.

bschmalhofer commented 2 years ago

Investigating the failure in scripts/test/Selenium/Output/Dashboard/News.t I found that files Kernel::System::UnitTest::Helper::CustomCodeActivate() is not synced via S3. Reopening #1769.

bschmalhofer commented 2 years ago

Looks better now. One remaining failure is in scripts/test/Console/Command/Admin/Package/Upgrade.t . Maybe keys containing colons, like Kernel/Config/Files/ZZZZUnitTestAdmin::Package::Upgradetest1522904334200000.pm can't be used in MinIO.

bschmalhofer commented 2 years ago

Looked at the failure in scripts/test/SupportDataCollector.t and did some cleanup. But now the failure is no longer reproducible. Let's see whether the failure occurs again.

bschmalhofer commented 2 years ago

Looked at the failure in scripts/test/SupportBundleGenerator.t . There the failure is that the file ZZZZUnitTestMaskPasswords.pm can't be unlinked because it has been removed before. This is fine. Kernel::System::SysConfig::new() explicitly calls Kernel::Config::new() which does syncing from S3. This means that ZZZZUnitTestMaskPasswords.pm is already removed there, as it hasn't been entered in S3.

bschmalhofer commented 2 years ago

Things look better after resolving the issues with SMIME #1779 .


Test Summary Report
-------------------
/opt/otobo/scripts/test/Console/Command/Admin/Package/Upgrade.t                                           (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/Storage/S3.t                                                                      (Wstat: 512 Tests: 9 Failed: 2)
  Failed tests:  4, 9
  Non-zero exit status: 2
/opt/otobo/scripts/test/SupportDataCollector.t                                                            (Wstat: 512 Tests: 328 Failed: 2)
  Failed tests:  323-324
  Non-zero exit status: 2
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=964, Tests=80976, 6254 wallclock secs (30.73 usr  3.13 sys + 1913.71 cusr 229.73 csys = 2177.30 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host 4905d56fb8c6 .
bschmalhofer commented 2 years ago

Still some test failures:


Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/Admin/AdminPGP.t                                                   (Wstat: 256 Tests: 21 Failed: 1)
  Failed test:  21
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Agent/Admin/AdminSMIME.t                                                 (Wstat: 256 Tests: 19 Failed: 1)
  Failed test:  19
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Agent/Admin/AgentTicketEmailSMIME.t                                      (Wstat: 256 Tests: 4 Failed: 1)
  Failed test:  4
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SupportDataCollector.t                                                            (Wstat: 512 Tests: 326 Failed: 2)
  Failed tests:  321-322
  Non-zero exit status: 2
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=964, Tests=80790, 6152 wallclock secs (30.50 usr  3.29 sys + 1901.47 cusr 229.16 csys = 2164.42 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host b995945d6786 .
bschmalhofer commented 2 years ago

The sporadic failures in scripts/test/Selenium/Agent/Admin/AdminPGP.t, scripts/test/Selenium/Agent/Admin/AdminSMIME.t , and scripts/test/Selenium/Agent/Admin/AgentTicketEmailSMIME.t all have a common cause. The relevant dirs are not shared across the containers running the OBOTO web server. The files could be shared via S3, just like the article data, but it is not obvious whether this is the best approach. So for now, let's disable the tests.

TODO:

bschmalhofer commented 2 years ago

There is still some work to do:

Test Summary Report
-------------------
/opt/otobo/scripts/test/Package/ConfigurationCleanup.t                                                    (Wstat: 1792 Tests: 12 Failed: 7)
  Failed tests:  6-12
  Non-zero exit status: 7
/opt/otobo/scripts/test/Selenium/Agent/Admin/AgentTicketEmailPGP.t                                        (Wstat: 256 Tests: 10 Failed: 1)
  Failed test:  10
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Output/Dashboard/CmdOutput.t                                             (Wstat: 256 Tests: 5 Failed: 1)
  Failed test:  4
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SupportDataCollector.t                                                            (Wstat: 512 Tests: 326 Failed: 2)
  Failed tests:  321-322
  Non-zero exit status: 2
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=964, Tests=80662, 6183 wallclock secs (31.34 usr  3.04 sys + 1918.00 cusr 231.88 csys = 2184.26 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host 6e84cf5ad80c .
bschmalhofer commented 2 years ago

Another run:


Test Summary Report
-------------------
/opt/otobo/scripts/test/Package/ConfigurationCleanup.t                                                    (Wstat: 2304 Tests: 12 Failed: 9)
  Failed tests:  4-12
  Non-zero exit status: 9
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Customer/DynamicFields/DynamicFieldDatabase.t                            (Wstat: 256 Tests: 148 Failed: 1)
  Failed test:  148
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Output/Dashboard/CmdOutput.t                                             (Wstat: 256 Tests: 5 Failed: 1)
  Failed test:  4
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Output/Preferences/Customer/SMIME.t                                      (Wstat: 512 Tests: 10 Failed: 2)
  Failed tests:  9-10
  Non-zero exit status: 2
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SupportDataCollector.t                                                            (Wstat: 512 Tests: 326 Failed: 2)
  Failed tests:  321-322
  Non-zero exit status: 2
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=964, Tests=80610, 6251 wallclock secs (30.39 usr  3.24 sys + 1899.29 cusr 231.26 csys = 2164.18 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host f3a1eec58ce1 .
bschmalhofer commented 2 years ago

Not really sure how the test script scripts/test/Package/ConfigurationCleanup.t works and what it acutally tests. I tried several times running the script abain and no errors were reported. A hunch is that the removed files had been recreated for some reason. I can*t be Kernel::Config::SyncWithS3() as that method does not consider the files in Kernel/Config/Files/XML . It might be that packages are reinstalled, but it is not obvious who would trigger this.

bschmalhofer commented 2 years ago

Latest run of the test suite with two web containers:

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 (exited 3) Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Customer/DynamicFields/DynamicFieldDatabase.t                            (Wstat: 256 (exited 1) Tests: 148 Failed: 1)
  Failed test:  148
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SupportDataCollector.t                                                            (Wstat: 512 (exited 2) Tests: 328 Failed: 2)
  Failed tests:  323-324
  Non-zero exit status: 2
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=964, Tests=80605, 6443 wallclock secs (30.62 usr  3.29 sys + 1975.38 cusr 239.42 csys = 2248.71 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host 375e5f83cdcb .
bschmalhofer commented 1 year ago

Still some work to do:

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 (exited 3) Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SupportDataCollector.t                                                            (Wstat: 512 (exited 2) Tests: 324 Failed: 2)
  Failed tests:  319-320
  Non-zero exit status: 2
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=964, Tests=80655, 6169 wallclock secs (27.21 usr  3.53 sys + 1811.26 cusr 244.41 csys = 2086.41 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host cb78dcc91524 .
stefanhaerter commented 1 year ago

See also #2029

bschmalhofer commented 1 year ago

Still not sure what the problem is with scripts/test/SupportDataCollector.t . When run as prove -I . -I Kernel/cpan-lib/ -v scripts/test/SupportDataCollector.t then there are usually no failures. The unit test config overrides look fine in S3 and in Kernel/Config/Files . The support data is collected via fetching from http://web:5000. No idea why the webserver does not see the config overrides when called as batch.

bschmalhofer commented 1 year ago

prove -I . -I Kernel/cpan-lib/ -v scripts/test/SysConfig/GlobalEffectiveValueGet.t works fine as well.

bschmalhofer commented 1 year ago

The next run showed some more errors:

Test Summary Report
-------------------
/opt/otobo/scripts/test/DynamicField/DynamicFieldDB.t                                                     (Wstat: 768 (exited 3) Tests: 23 Failed: 3)
  Failed tests:  9-11
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 (exited 3) Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SupportDataCollector.t                                                            (Wstat: 512 (exited 2) Tests: 324 Failed: 2)
  Failed tests:  319-320
  Non-zero exit status: 2
/opt/otobo/scripts/test/SysConfig/ConfigurationDeploy.t                                                   (Wstat: 2048 (exited 8) Tests: 24 Failed: 8)
  Failed tests:  4-9, 19-20
  Non-zero exit status: 8
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=964, Tests=80655, 6932 wallclock secs (32.18 usr  4.29 sys + 2021.01 cusr 272.98 csys = 2330.46 CPU)
Result: FAIL
ran tests for product OTOBO 11.2.x on host 77a5e4e34583 .
bschmalhofer commented 1 year ago

Another run:

Test Summary Report
-------------------
/opt/otobo/scripts/test/DynamicField/DynamicFieldDB.t                                                     (Wstat: 768 (exited 3) Tests: 23 Failed: 3)
  Failed tests:  9-11
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 (exited 3) Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t                              (Wstat: 256 (exited 1) Tests: 38 Failed: 1)
  Failed test:  38
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=964, Tests=80489, 6111 wallclock secs (27.57 usr  3.51 sys + 1800.12 cusr 241.59 csys = 2072.79 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host f22978da885f .
bschmalhofer commented 1 year ago

This is a bit involved. First I fiddled until I could find a reproducible behavior. The identified case is:

The test /ConfigurationCleanup.t installs an OTOBO package. The test GlobalEffectiveValueGet.t verifies that a modified setting is actually modified when the config is reloaded. Without sleeping the latter test fails in most cases. With a sleep of 10 s the latter always succeeds. This behavior indicates that there is a connection with installing packages.

In the general case OTOBO can be served by different web servers that have independent /opt/otobo directories. Changed config and newly installed packages are synchronized via the shared S3 backend and via the shared database. Changed config is synchronised by writing the master copy to S3 first. Information about new packages is distributed by writing the file _eventpackage.json into S3. An updated timestamp on the S3 object indicates that the webserver should reinstall packages and restart the webserver. When installing a package, there often are new or changed files and new or changed configuration. Updating the configuration but not the files could create havoc. Therefore there is check in Kernel::Config::SyncWithS3 that config is only synced when the packages are up to date. The webserver has an event loop that checks every 10 s for new packages. This explains the time dependency. When running the two tests without sleep, then the config is not synced from S3 as the webserver hasn't reinstalled the packages yet.

There are several possibilities for redeeming this situation;

The last option might be the least fragile.

bschmalhofer commented 1 year ago

Another thing that should be investigated. After running the test suite there are a lot of defunct processes:

otobo@871c0487f4a6:~$ ps waux | grep perl | more
otobo          1  0.0  0.1  49156 12748 ?        Ss   12:07   0:07 /usr/local/bin/perl /opt/otobo_install/local/bin/plackup --server Gazelle --env deployment --por
t 5000 -I /opt/otobo -I /opt/otobo/Kernel/cpan-lib --loader SyncWithS3 bin/psgi-bin/otobo.psgi
otobo         81  0.0  0.0      0     0 ?        Z    12:08   0:00 [perl] <defunct>
otobo         82  0.0  0.0      0     0 ?        Z    12:08   0:00 [perl] <defunct>
otobo       4588  0.0  0.0      0     0 ?        Z    12:18   0:00 [perl] <defunct>
otobo       4589  0.0  0.0      0     0 ?        Z    12:18   0:00 [perl] <defunct>
otobo       4590  0.0  0.0      0     0 ?        Z    12:18   0:00 [perl] <defunct>
otobo       4591  0.0  0.0      0     0 ?        Z    12:18   0:00 [perl] <defunct>
otobo       4592  0.0  0.0      0     0 ?        Z    12:18   0:00 [perl] <defunct>
otobo      11536  0.0  0.0      0     0 ?        Z    13:07   0:00 [perl] <defunct>
bschmalhofer commented 1 year ago

Run the test suite with rel-11_0 and S3 activated. It still does not look fine. The errors in DynamicFieldDatabase.t and CreateScreens.t seem to be unrelated to S3, some sort of timing issue. See #1067. The failure in AdminQueueTemplates.t is sporadic.

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/Admin/AdminQueueTemplates.t                                        (Wstat: 512 (exited 2) Tests: 38 Failed: 2)
  Failed tests:  37-38
  Non-zero exit status: 2
/opt/otobo/scripts/test/Selenium/Agent/DynamicFields/DynamicFieldDatabase.t                               (Wstat: 256 (exited 1) Tests: 120 Failed: 1)
  Failed test:  120
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t                              (Wstat: 256 (exited 1) Tests: 133 Failed: 1)
  Failed test:  133
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=959, Tests=80300, 3767 wallclock secs (14.83 usr  2.56 sys + 763.67 cusr 104.04 csys = 885.10 CPU)
Result: FAIL
ran tests for product OTOBO 11.0.x on host de7d95d4475a .
bschmalhofer commented 1 year ago

The lates test run:

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/AgentTicketSplitInlineImage.t                                      (Wstat: 768 (exited 3) Tests: 33 Failed: 3)
  Failed tests:  30, 32-33
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t                              (Wstat: 256 (exited 1) Tests: 39 Failed: 1)
  Failed test:  39
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=959, Tests=80297, 3802 wallclock secs (14.89 usr  2.40 sys + 767.02 cusr 108.52 csys = 892.83 CPU)
Result: FAIL
ran tests for product OTOBO 11.0.x on host ad8923ced2d4 .
bschmalhofer commented 1 year ago

Another try:

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=959, Tests=80472, 6188 wallclock secs (15.00 usr  2.48 sys + 760.35 cusr 108.09 csys = 885.92 CPU)
Result: FAIL
ran tests for product OTOBO 11.0.x on host 775a1066f820 .
bschmalhofer commented 1 year ago

Another run:

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/Admin/Admin.t                                                      (Wstat: 512 (exited 2) Tests: 143 Failed: 2)
  Failed tests:  142-143
  Non-zero exit status: 2
/opt/otobo/scripts/test/Selenium/Agent/AgentTicketSplitInlineImage.t                                      (Wstat: 768 (exited 3) Tests: 33 Failed: 3)
  Failed tests:  30, 32-33
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/SysConfig/GlobalEffectiveValueGet.t                                               (Wstat: 256 (exited 1) Tests: 3 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=959, Tests=80369, 10852 wallclock secs (15.02 usr  2.20 sys + 771.17 cusr 108.59 csys = 896.98 CPU)
Result: FAIL
ran tests for product OTOBO 11.0.x on host e7b385b04e7b .
bschmalhofer commented 1 year ago

Another disappointing test run. There was high load on the local test machine. The high load might explain the failures, but the cause of the high load is unknown.

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/Admin/Admin.t                                                      (Wstat: 513 (exited 3) Tests: 144 Failed: 2)
  Failed tests:  142-143
  Non-zero exit status: 2
/opt/otobo/scripts/test/Selenium/Agent/AgentTicketSplitInlineImage.t                                      (Wstat: 768 (exited 3) Tests: 33 Failed: 3)
  Failed tests:  30, 32-33
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t                              (Wstat: 256 (exited 1) Tests: 136 Failed: 1)
  Failed test:  136
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Output/UserTitleTranslation.t                                            (Wstat: 257 (exited 1) Tests: 14 Failed: 1)
  Failed test:  14
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/SysConfig/AgentSetting.t                                                 (Wstat: 65280 (exited 255) Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 65280 (exited 255) Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=959, Tests=80235, 4274 wallclock secs (15.27 usr  2.30 sys + 781.05 cusr 119.39 csys = 918.01 CPU)
Result: FAIL
ran tests for product OTOBO 11.0.x on host 491fd0000a01 .
bschmalhofer commented 1 year ago

Without excessive load:

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/Admin/Admin.t                                                      (Wstat: 512 (exited 2) Tests: 143 Failed: 2)
  Failed tests:  142-143
  Non-zero exit status: 2
/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t                              (Wstat: 256 (exited 1) Tests: 29 Failed: 1)
  Failed test:  29
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
Files=958, Tests=80215, 3857 wallclock secs (15.34 usr  2.53 sys + 779.04 cusr 111.33 csys = 908.24 CPU)
Result: FAIL
ran tests for product OTOBO 11.0.x on host 7c21b7b69e80 .
bschmalhofer commented 1 year ago

I double checked by running the test suite without S3 in a single web container. No errors there. Memory usage of the plackup processes was comparable to the S3 case. One process grabbed more than 400 MB.

bschmalhofer commented 1 year ago

Still the failure in CreateScreens.t

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/Agent/MultiAttachmentUpload/CreateScreens.t                              (Wstat: 256 (exited 1) Tests: 136 Failed: 1)
  Failed test:  136
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
Files=958, Tests=80408, 3749 wallclock secs (14.91 usr  2.11 sys + 748.27 cusr 105.71 csys = 871.00 CPU)
Result: FAIL
ran tests for product OTOBO 11.0.x on host 71fec89bc092 .
bschmalhofer commented 1 year ago

Another test run with rel-11_0 and S3 support shows a new error. It could be that the failures were due to dynamic fields that were not cleaned up properly. For now I only converted the test script to using subtests. But this is only for making analysis easier, when the error occurs again.

Test Summary Report
-------------------
/opt/otobo/scripts/test/GenericInterface/Invoker/Ticket/WebserviceTicketInvoker.t                         (Wstat: 1280 (exited 5) Tests: 65 Failed: 5)
  Failed tests:  5, 15, 26, 48, 59
  Non-zero exit status: 5
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
Files=958, Tests=80469, 3651 wallclock secs (14.78 usr  2.58 sys + 749.15 cusr 106.98 csys = 873.49 CPU)
Result: FAIL
ran tests for product OTOBO 11.0.x on host 6b42b4d70971 .
bschmalhofer commented 1 year ago

The latest test run looks fine. There are some sporadic failures, but these are covered in separate issues:

Closing this issue.

Test Summary Report
-------------------
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
Files=958, Tests=80409, 7893 wallclock secs (15.36 usr  2.49 sys + 761.02 cusr 109.71 csys = 888.58 CPU)
Result: PASS
ran tests for product OTOBO 11.0.x on host fda1617ccdc6 .