search

os-autoinst / openQA

openQA web-frontend, scheduler and tools.
http://openqa.opensuse.org/
GNU General Public License v2.0
321 stars 208 forks source link

Test works, but live streaming return 504 #5384

Closed trdthg closed 4 months ago

trdthg commented 11 months ago

Hi, I'm trying to use openQA in container (I have used it in openSUSE VM before), after I solved unable to upgrade ws to command server, test started successful, I can see the current module xxx at xxx changes, and get all testresult after the end. But streaming (as well as livelog) just return 504 after a while. How should I debug? 😵

Martchus commented 11 months ago

Does the upload of the logs work at the end of the test run? To debug this, I'd have a look at the logs on the web UI services. I think openqa-webui.service is the most relevant one.

trdthg commented 11 months ago

Does the upload of the logs work at the end of the test run?

Yes, all logs, video and shortcuts

image image

logs on the web UI services

I run a mini test case, here is a screenshot(docker-compsoe, webui and worker) image

test case:

# Copyright 2014-2018 SUSE LLC
# SPDX-License-Identifier: GPL-2.0-or-later

use base 'basetest';
use strict;
use testapi;

sub run {
    # wait for bootloader to appear
    assert_screen 'bootloader';

    # conclude test early if there's no boot media
    return undef if match_has_tag 'no-boot-media';

    # press enter to boot right away
    send_key 'ret';

    # wait for the desktop to appear
    assert_screen 'desktop', 300;
}

1;

all logs:

Martchus commented 11 months ago

I guess the logs of openqa-webui.service and possible the other web UI services would have been more interesting than what's going on on the worker-side.

EDIT: Oh, that screenshot contains web UI logs. It also contains worker logs which where a bit distracting. Unfortunately I couldn't find anything useful in these log excerpts, likely everything that comes after the worker registration would be more useful. It is possible that this is a problem of the docker-compose setup (and it would work when setting things up via plain old RPMs.)

trdthg commented 11 months ago

Thank you for your attention.

everything that comes after the worker registration

I use docker-compose up --build start a new worker, rerun the test, here is the log:

although the livestreaming was canceled prematurely due to the short test run, instead of returning 504

worker-1  | TEST_DISTRI_DEPS: 
worker-1  | Loading repository data...
worker-1  | Reading installed packages...
worker-1  | Resolving package dependencies...
worker-1  | 
worker-1  | The following 60 NEW packages are going to be installed:
worker-1  |   lsof os-autoinst-distri-opensuse-deps perl-Archive-Zip perl-B-Hooks-EndOfScope perl-B-Keywords perl-Class-Inspector perl-Class-Singleton perl-Class-Tiny perl-Clone perl-Code-DRY perl-DateTime perl-DateTime-Locale perl-DateTime-TimeZone perl-Dist-CheckConflicts perl-Eval-Closure perl-Exception-Base perl-File-Path perl-File-ShareDir perl-IO-String perl-JSON perl-Lingua-EN-Inflect perl-List-SomeUtils perl-List-SomeUtils-XS perl-Module-Build perl-Module-Pluggable perl-NetAddr-IP perl-PPI perl-PPIx-QuoteLike perl-PPIx-Regexp perl-PPIx-Utilities perl-Package-Stash perl-Package-Stash-XS perl-Params-ValidationCompiler perl-Path-Tiny perl-Perl-Critic perl-Perl-Critic-Community perl-Perl-Critic-Policy-Variables-ProhibitLoopOnHash perl-Perl-Critic-Pulp perl-Perl-Tidy perl-Pod-MinimumVersion perl-Pod-Spell perl-Readonly perl-Selenium-Remote-Driver perl-SemVer perl-Specio perl-String-Format perl-Sub-Identify perl-Symbol-Util perl-Task-Weaken perl-Test-Assert perl-Test-Fatal perl-Test-LongString perl-Tie-IxHash perl-Variable-Magic perl-XML-SAX-Expat perl-XML-Simple perl-XML-Writer perl-constant-boolean perl-namespace-autoclean perl-namespace-clean
worker-1  | 
worker-1  | 60 new packages to install.
worker-1  | Overall download size: 8.9 MiB. Already cached: 0 B. After the operation, additional 36.2 MiB will be used.
worker-1  | Continue? [y/n/v/...? shows all options] (y): y
worker-1  | Retrieving: perl-Perl-Tidy-20230912.0.0-lp155.2.1.noarch (devel_openQA_Leap) (1/60), 768.9 KiB    
worker-1  | Retrieving: perl-Perl-Tidy-20230912.0.0-lp155.2.1.noarch.rpm [...................................done (497.4 KiB/s)]
worker-1  | Retrieving: lsof-4.91-1.11.x86_64 (Main Repository) (2/60), 330.9 KiB    
worker-1  | Retrieving: lsof-4.91-1.11.x86_64.rpm [....done (89.8 KiB/s)]
worker-1  | Retrieving: perl-Archive-Zip-1.60-3.3.1.noarch (Main Repository) (3/60), 103.4 KiB    
worker-1  | Retrieving: perl-Archive-Zip-1.60-3.3.1.noarch.rpm [..done (4.1 KiB/s)]
worker-1  | Retrieving: perl-B-Keywords-1.24-bp155.1.5.noarch (Main Repository) (4/60),  21.2 KiB    
worker-1  | Retrieving: perl-B-Keywords-1.24-bp155.1.5.noarch.rpm [..done (794 B/s)]
worker-1  | Retrieving: perl-Class-Inspector-1.32-1.22.noarch (Main Repository) (5/60),  32.0 KiB    
worker-1  | Retrieving: perl-Class-Inspector-1.32-1.22.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Class-Singleton-1.5-1.24.noarch (Main Repository) (6/60),  19.3 KiB    
worker-1  | Retrieving: perl-Class-Singleton-1.5-1.24.noarch.rpm [.....done (18.6 KiB/s)]
worker-1  | Retrieving: perl-Class-Tiny-1.006-1.24.noarch (Main Repository) (7/60),  30.8 KiB    
worker-1  | Retrieving: perl-Class-Tiny-1.006-1.24.noarch.rpm [.......done (30.1 KiB/s)]
worker-1  | Retrieving: perl-Clone-0.46-bp155.1.5.x86_64 (Main Repository) (8/60),  21.9 KiB    
worker-1  | Retrieving: perl-Clone-0.46-bp155.1.5.x86_64.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Dist-CheckConflicts-0.11-1.23.noarch (Main Repository) (9/60),  24.8 KiB    
worker-1  | Retrieving: perl-Dist-CheckConflicts-0.11-1.23.noarch.rpm [....done (4.3 KiB/s)]
worker-1  | Retrieving: perl-Exception-Base-0.2501-bp155.2.8.noarch (Main Repository) (10/60),  52.6 KiB    
worker-1  | Retrieving: perl-Exception-Base-0.2501-bp155.2.8.noarch.rpm [...done (4.2 KiB/s)]
worker-1  | Retrieving: perl-File-Path-2.150000-1.22.noarch (Main Repository) (11/60),  39.5 KiB    
worker-1  | Retrieving: perl-File-Path-2.150000-1.22.noarch.rpm [...done]
worker-1  | Retrieving: perl-IO-String-1.08-152.24.x86_64 (Main Repository) (12/60),  17.0 KiB    
worker-1  | Retrieving: perl-IO-String-1.08-152.24.x86_64.rpm [..done (798 B/s)]
worker-1  | Retrieving: perl-JSON-2.97001-1.18.noarch (Main Repository) (13/60),  92.4 KiB    
worker-1  | Retrieving: perl-JSON-2.97001-1.18.noarch.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Lingua-EN-Inflect-1.905-bp155.2.12.noarch (Main Repository) (14/60),  60.6 KiB    
worker-1  | Retrieving: perl-Lingua-EN-Inflect-1.905-bp155.2.12.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-List-SomeUtils-XS-0.56-1.17.x86_64 (Main Repository) (15/60),  41.3 KiB    
worker-1  | Retrieving: perl-List-SomeUtils-XS-0.56-1.17.x86_64.rpm [......done (32.3 KiB/s)]
worker-1  | Retrieving: perl-Module-Build-0.422400-1.24.noarch (Main Repository) (16/60), 259.4 KiB    
worker-1  | Retrieving: perl-Module-Build-0.422400-1.24.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Module-Pluggable-5.2-1.24.noarch (Main Repository) (17/60),  33.6 KiB    
worker-1  | Retrieving: perl-Module-Pluggable-5.2-1.24.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-NetAddr-IP-4.079-1.27.x86_64 (Main Repository) (18/60), 120.4 KiB    
worker-1  | Retrieving: perl-NetAddr-IP-4.079-1.27.x86_64.rpm [....done (3.5 KiB/s)]
worker-1  | Retrieving: perl-Package-Stash-XS-0.28-1.26.x86_64 (Main Repository) (19/60),  30.9 KiB    
worker-1  | Retrieving: perl-Package-Stash-XS-0.28-1.26.x86_64.rpm [...done (251.7 KiB/s)]
worker-1  | Retrieving: perl-Readonly-2.05-1.22.noarch (Main Repository) (20/60),  36.5 KiB    
worker-1  | Retrieving: perl-Readonly-2.05-1.22.noarch.rpm [...done]
worker-1  | Retrieving: perl-SemVer-0.10.0-bp155.1.4.noarch (Main Repository) (21/60),  20.4 KiB    
worker-1  | Retrieving: perl-SemVer-0.10.0-bp155.1.4.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-String-Format-1.18-bp155.2.12.noarch (Main Repository) (22/60),  23.1 KiB    
worker-1  | Retrieving: perl-String-Format-1.18-bp155.2.12.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Sub-Identify-0.14-1.26.x86_64 (Main Repository) (23/60),  20.2 KiB    
worker-1  | Retrieving: perl-Sub-Identify-0.14-1.26.x86_64.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Symbol-Util-0.0203-bp155.2.12.noarch (Main Repository) (24/60),  30.7 KiB    
worker-1  | Retrieving: perl-Symbol-Util-0.0203-bp155.2.12.noarch.rpm [.....done (30.7 KiB/s)]
worker-1  | Retrieving: perl-Task-Weaken-1.06-2.13.noarch (Main Repository) (25/60),  24.0 KiB    
worker-1  | Retrieving: perl-Task-Weaken-1.06-2.13.noarch.rpm [....done (24.0 KiB/s)]
worker-1  | Retrieving: perl-Test-Fatal-0.014-1.23.noarch (Main Repository) (26/60),  24.1 KiB    
worker-1  | Retrieving: perl-Test-Fatal-0.014-1.23.noarch.rpm [...done]
worker-1  | Retrieving: perl-Test-LongString-0.17-bp155.2.12.noarch (Main Repository) (27/60),  17.8 KiB    
worker-1  | Retrieving: perl-Test-LongString-0.17-bp155.2.12.noarch.rpm [..done (788 B/s)]
worker-1  | Retrieving: perl-Tie-IxHash-1.23-1.23.noarch (Main Repository) (28/60),  18.4 KiB    
worker-1  | Retrieving: perl-Tie-IxHash-1.23-1.23.noarch.rpm [..done (799 B/s)]
worker-1  | Retrieving: perl-Variable-Magic-0.62-1.19.x86_64 (Main Repository) (29/60),  62.2 KiB    
worker-1  | Retrieving: perl-Variable-Magic-0.62-1.19.x86_64.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-XML-SAX-Expat-0.51-1.22.noarch (Main Repository) (30/60),  16.1 KiB    
worker-1  | Retrieving: perl-XML-SAX-Expat-0.51-1.22.noarch.rpm [...done (4.1 KiB/s)]
worker-1  | Retrieving: perl-XML-Writer-0.625-1.24.noarch (Main Repository) (31/60),  34.6 KiB    
worker-1  | Retrieving: perl-XML-Writer-0.625-1.24.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Eval-Closure-0.14-1.22.noarch (Main Repository) (32/60),  25.2 KiB    
worker-1  | Retrieving: perl-Eval-Closure-0.14-1.22.noarch.rpm [...done]
worker-1  | Retrieving: perl-File-ShareDir-1.104-1.22.noarch (Main Repository) (33/60),  27.0 KiB    
worker-1  | Retrieving: perl-File-ShareDir-1.104-1.22.noarch.rpm [.....done (548.4 KiB/s)]
worker-1  | Retrieving: perl-Path-Tiny-0.104-1.22.noarch (Main Repository) (34/60),  77.7 KiB    
worker-1  | Retrieving: perl-Path-Tiny-0.104-1.22.noarch.rpm [....done (4.1 KiB/s)]
worker-1  | Retrieving: perl-Pod-MinimumVersion-50-bp155.3.10.noarch (Main Repository) (35/60),  35.0 KiB    
worker-1  | Retrieving: perl-Pod-MinimumVersion-50-bp155.3.10.noarch.rpm [...done]
worker-1  | Retrieving: perl-List-SomeUtils-0.56-1.20.noarch (Main Repository) (36/60),  46.9 KiB    
worker-1  | Retrieving: perl-List-SomeUtils-0.56-1.20.noarch.rpm [...done (1.7 MiB/s)]
worker-1  | Retrieving: perl-Code-DRY-0.10-bp155.2.10.x86_64 (Main Repository) (37/60),  44.0 KiB    
worker-1  | Retrieving: perl-Code-DRY-0.10-bp155.2.10.x86_64.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Package-Stash-0.37-1.23.noarch (Main Repository) (38/60),  35.7 KiB    
worker-1  | Retrieving: perl-Package-Stash-0.37-1.23.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-constant-boolean-0.02-bp155.2.8.noarch (Main Repository) (39/60),  23.6 KiB    
worker-1  | Retrieving: perl-constant-boolean-0.02-bp155.2.8.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-PPI-1.276-bp155.1.4.noarch (Main Repository) (40/60), 349.9 KiB    
worker-1  | Retrieving: perl-PPI-1.276-bp155.1.4.noarch.rpm [....done (3.9 KiB/s)]
worker-1  | Retrieving: perl-B-Hooks-EndOfScope-0.21-1.22.noarch (Main Repository) (41/60),  30.3 KiB    
worker-1  | Retrieving: perl-B-Hooks-EndOfScope-0.21-1.22.noarch.rpm [..done (791 B/s)]
worker-1  | Retrieving: perl-XML-Simple-2.24-1.22.noarch (Main Repository) (42/60),  81.4 KiB    
worker-1  | Retrieving: perl-XML-Simple-2.24-1.22.noarch.rpm [...done]
worker-1  | Retrieving: perl-Specio-0.42-1.17.noarch (Main Repository) (43/60), 160.8 KiB    
worker-1  | Retrieving: perl-Specio-0.42-1.17.noarch.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Params-ValidationCompiler-0.26-1.17.noarch (Main Repository) (44/60),  34.0 KiB    
worker-1  | Retrieving: perl-Params-ValidationCompiler-0.26-1.17.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Pod-Spell-1.25-bp155.1.5.noarch (Main Repository) (45/60),  44.2 KiB    
worker-1  | Retrieving: perl-Pod-Spell-1.25-bp155.1.5.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Test-Assert-0.0504-bp155.2.8.noarch (Main Repository) (46/60),  34.6 KiB    
worker-1  | Retrieving: perl-Test-Assert-0.0504-bp155.2.8.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-PPIx-Utilities-1.001000-bp155.3.9.noarch (Main Repository) (47/60),  36.2 KiB    
worker-1  | Retrieving: perl-PPIx-Utilities-1.001000-bp155.3.9.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-PPIx-Regexp-0.086-bp155.1.5.noarch (Main Repository) (48/60), 329.3 KiB    
worker-1  | Retrieving: perl-PPIx-Regexp-0.086-bp155.1.5.noarch.rpm [..done (792 B/s)]
worker-1  | Retrieving: perl-PPIx-QuoteLike-0.023-bp155.1.5.noarch (Main Repository) (49/60),  86.5 KiB    
worker-1  | Retrieving: perl-PPIx-QuoteLike-0.023-bp155.1.5.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-namespace-clean-0.27-1.22.noarch (Main Repository) (50/60),  24.0 KiB    
worker-1  | Retrieving: perl-namespace-clean-0.27-1.22.noarch.rpm [....done (3.5 KiB/s)]
worker-1  | Retrieving: perl-Perl-Critic-1.148-bp155.1.4.noarch (Main Repository) (51/60), 894.5 KiB    
worker-1  | Retrieving: perl-Perl-Critic-1.148-bp155.1.4.noarch.rpm [....done (5.3 MiB/s)]
worker-1  | Retrieving: perl-namespace-autoclean-0.28-1.22.noarch (Main Repository) (52/60),  27.7 KiB    
worker-1  | Retrieving: perl-namespace-autoclean-0.28-1.22.noarch.rpm [...done (165.2 KiB/s)]
worker-1  | Retrieving: perl-Perl-Critic-Pulp-99-bp155.2.8.x86_64 (Main Repository) (53/60), 238.3 KiB    
worker-1  | Retrieving: perl-Perl-Critic-Pulp-99-bp155.2.8.x86_64.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Perl-Critic-Policy-Variables-ProhibitLoopOnHash-0.008-bp155.2.8.noarch (Main Repository) (54/60),  21.4 KiB    
worker-1  | Retrieving: perl-Perl-Critic-Policy-Variables-ProhibitLoopOnHash-0.008-bp155.2.8.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-DateTime-Locale-1.170000-1.17.noarch (Main Repository) (55/60),   2.5 MiB    
worker-1  | Retrieving: perl-DateTime-Locale-1.170000-1.17.noarch.rpm [............done (2.3 MiB/s)]
worker-1  | Retrieving: perl-Perl-Critic-Community-1.0.3-bp155.1.4.noarch (Main Repository) (56/60), 160.0 KiB    
worker-1  | Retrieving: perl-Perl-Critic-Community-1.0.3-bp155.1.4.noarch.rpm [..done (782 B/s)]
worker-1  | Retrieving: perl-Selenium-Remote-Driver-1.49-bp155.2.3.1.x86_64 (Update repository of openSUSE Backports) (57/60), 816.3 KiB    
worker-1  | Retrieving: perl-Selenium-Remote-Driver-1.49-bp155.2.3.1.x86_64.rpm [.............done (2.9 MiB/s)]
worker-1  | Retrieving: perl-DateTime-TimeZone-2.60-150000.3.3.1.noarch (Update repository with updates from SUSE Linux Enterprise 15) (58/60), 354.3 KiB    
worker-1  | Retrieving: perl-DateTime-TimeZone-2.60-150000.3.3.1.noarch.rpm [......done (2.1 MiB/s)]
worker-1  | Retrieving: perl-DateTime-1.45-1.16.x86_64 (Main Repository) (59/60), 143.0 KiB    
worker-1  | Retrieving: perl-DateTime-1.45-1.16.x86_64.rpm [...done (3.9 KiB/s)]
worker-1  | Retrieving: os-autoinst-distri-opensuse-deps-1.1702033475.f69e77d2-lp155.13340.1.noarch (devel_openQA) (60/60),   8.2 KiB    
worker-1  | Retrieving: os-autoinst-distri-opensuse-deps-1.1702033475.f69e77d2-lp155.13340.1.noarch.rpm [........done (17.0 KiB/s)]
worker-1  | 
worker-1  | Checking for file conflicts: [..done]
worker-1  | ( 1/60) Installing: perl-Perl-Tidy-20230912.0.0-lp155.2.1.noarch [..done]
worker-1  | ( 2/60) Installing: lsof-4.91-1.11.x86_64 [..done]
worker-1  | ( 3/60) Installing: perl-Archive-Zip-1.60-3.3.1.noarch [..done]
worker-1  | ( 4/60) Installing: perl-B-Keywords-1.24-bp155.1.5.noarch [..done]
worker-1  | ( 5/60) Installing: perl-Class-Inspector-1.32-1.22.noarch [..done]
worker-1  | ( 6/60) Installing: perl-Class-Singleton-1.5-1.24.noarch [..done]
worker-1  | ( 7/60) Installing: perl-Class-Tiny-1.006-1.24.noarch [..done]
worker-1  | ( 8/60) Installing: perl-Clone-0.46-bp155.1.5.x86_64 [..done]
worker-1  | ( 9/60) Installing: perl-Dist-CheckConflicts-0.11-1.23.noarch [..done]
worker-1  | (10/60) Installing: perl-Exception-Base-0.2501-bp155.2.8.noarch [..done]
worker-1  | (11/60) Installing: perl-File-Path-2.150000-1.22.noarch [..done]
worker-1  | (12/60) Installing: perl-IO-String-1.08-152.24.x86_64 [..done]
worker-1  | (13/60) Installing: perl-JSON-2.97001-1.18.noarch [..done]
worker-1  | (14/60) Installing: perl-Lingua-EN-Inflect-1.905-bp155.2.12.noarch [..done]
worker-1  | (15/60) Installing: perl-List-SomeUtils-XS-0.56-1.17.x86_64 [..done]
worker-1  | (16/60) Installing: perl-Module-Build-0.422400-1.24.noarch [..done]
worker-1  | (17/60) Installing: perl-Module-Pluggable-5.2-1.24.noarch [..done]
worker-1  | (18/60) Installing: perl-NetAddr-IP-4.079-1.27.x86_64 [..done]
worker-1  | (19/60) Installing: perl-Package-Stash-XS-0.28-1.26.x86_64 [..done]
worker-1  | (20/60) Installing: perl-Readonly-2.05-1.22.noarch [..done]
worker-1  | (21/60) Installing: perl-SemVer-0.10.0-bp155.1.4.noarch [..done]
worker-1  | (22/60) Installing: perl-String-Format-1.18-bp155.2.12.noarch [..done]
worker-1  | (23/60) Installing: perl-Sub-Identify-0.14-1.26.x86_64 [..done]
worker-1  | (24/60) Installing: perl-Symbol-Util-0.0203-bp155.2.12.noarch [..done]
worker-1  | (25/60) Installing: perl-Task-Weaken-1.06-2.13.noarch [..done]
worker-1  | (26/60) Installing: perl-Test-Fatal-0.014-1.23.noarch [..done]
worker-1  | (27/60) Installing: perl-Test-LongString-0.17-bp155.2.12.noarch [..done]
worker-1  | (28/60) Installing: perl-Tie-IxHash-1.23-1.23.noarch [..done]
worker-1  | (29/60) Installing: perl-Variable-Magic-0.62-1.19.x86_64 [..done]
worker-1  | (30/60) Installing: perl-XML-SAX-Expat-0.51-1.22.noarch [..done]
worker-1  | (31/60) Installing: perl-XML-Writer-0.625-1.24.noarch [..done]
worker-1  | (32/60) Installing: perl-Eval-Closure-0.14-1.22.noarch [..done]
worker-1  | (33/60) Installing: perl-File-ShareDir-1.104-1.22.noarch [..done]
worker-1  | (34/60) Installing: perl-Path-Tiny-0.104-1.22.noarch [..done]
worker-1  | (35/60) Installing: perl-Pod-MinimumVersion-50-bp155.3.10.noarch [..done]
worker-1  | (36/60) Installing: perl-List-SomeUtils-0.56-1.20.noarch [..done]
worker-1  | (37/60) Installing: perl-Code-DRY-0.10-bp155.2.10.x86_64 [..done]
worker-1  | (38/60) Installing: perl-Package-Stash-0.37-1.23.noarch [..done]
worker-1  | (39/60) Installing: perl-constant-boolean-0.02-bp155.2.8.noarch [..done]
worker-1  | (40/60) Installing: perl-PPI-1.276-bp155.1.4.noarch [..done]
worker-1  | (41/60) Installing: perl-B-Hooks-EndOfScope-0.21-1.22.noarch [..done]
worker-1  | (42/60) Installing: perl-XML-Simple-2.24-1.22.noarch [..done]
worker-1  | (43/60) Installing: perl-Specio-0.42-1.17.noarch [..done]
worker-1  | (44/60) Installing: perl-Params-ValidationCompiler-0.26-1.17.noarch [..done]
worker-1  | (45/60) Installing: perl-Pod-Spell-1.25-bp155.1.5.noarch [..done]
worker-1  | (46/60) Installing: perl-Test-Assert-0.0504-bp155.2.8.noarch [..done]
worker-1  | (47/60) Installing: perl-PPIx-Utilities-1.001000-bp155.3.9.noarch [..done]
worker-1  | (48/60) Installing: perl-PPIx-Regexp-0.086-bp155.1.5.noarch [..done]
worker-1  | (49/60) Installing: perl-PPIx-QuoteLike-0.023-bp155.1.5.noarch [..done]
worker-1  | (50/60) Installing: perl-namespace-clean-0.27-1.22.noarch [..done]
worker-1  | (51/60) Installing: perl-Perl-Critic-1.148-bp155.1.4.noarch [..done]
worker-1  | (52/60) Installing: perl-namespace-autoclean-0.28-1.22.noarch [..done]
worker-1  | (53/60) Installing: perl-Perl-Critic-Pulp-99-bp155.2.8.x86_64 [..done]
worker-1  | (54/60) Installing: perl-Perl-Critic-Policy-Variables-ProhibitLoopOnHash-0.008-bp155.2.8.noarch [..done]
worker-1  | (55/60) Installing: perl-DateTime-Locale-1.170000-1.17.noarch [.....done]
worker-1  | (56/60) Installing: perl-Perl-Critic-Community-1.0.3-bp155.1.4.noarch [..done]
worker-1  | (57/60) Installing: perl-Selenium-Remote-Driver-1.49-bp155.2.3.1.x86_64 [..done]
worker-1  | (58/60) Installing: perl-DateTime-TimeZone-2.60-150000.3.3.1.noarch [..done]
worker-1  | (59/60) Installing: perl-DateTime-1.45-1.16.x86_64 [..done]
worker-1  | (60/60) Installing: os-autoinst-distri-opensuse-deps-1.1702033475.f69e77d2-lp155.13340.1.noarch [..done]
worker-1  | /run_openqa_worker.sh: line 34:     9 Terminated              qemu-system-x86_64 -S
worker-1  | [info] [pid:180] worker 1:
worker-1  |  - config file:                      /etc/openqa/workers.ini
worker-1  |  - name used to register:            93239f7bc06e
worker-1  |  - worker address (WORKER_HOSTNAME): worker
worker-1  |  - isotovideo version:               40
worker-1  |  - websocket API version:            1
worker-1  |  - web UI hosts:                     http://172.18.0.9
worker-1  |  - class:                            demohw
worker-1  |  - no cleanup:                       no
worker-1  |  - pool directory:                   /var/lib/openqa/pool/1
worker-1  | [info] [pid:180] Project dir for host http://172.18.0.9 is /var/lib/openqa/share
worker-1  | [info] [pid:180] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:180] Establishing ws connection via ws://172.18.0.9/api/v1/ws/14
worker-1  | [info] [pid:180] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 14
worker-1  | [warn] [pid:180] Websocket connection to http://172.18.0.9/api/v1/ws/14 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:180] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:180] Establishing ws connection via ws://172.18.0.9/api/v1/ws/14
worker-1  | [info] [pid:180] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 14
worker-1  | [warn] [pid:180] Websocket connection to http://172.18.0.9/api/v1/ws/14 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:180] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:180] Establishing ws connection via ws://172.18.0.9/api/v1/ws/14
worker-1  | [info] [pid:180] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 14
worker-1  | [warn] [pid:180] Websocket connection to http://172.18.0.9/api/v1/ws/14 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:180] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:180] Establishing ws connection via ws://172.18.0.9/api/v1/ws/14
worker-1  | [info] [pid:180] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 14
worker-1  | [debug] [pid:180] Accepting job 58 from http://172.18.0.9.
worker-1  | [debug] [pid:180] Setting job 58 from http://172.18.0.9 up
worker-1  | [debug] [pid:180] Preparing Mojo::IOLoop::ReadWriteProcess::Session
worker-1  | [info] [pid:180] +++ setup notes +++
worker-1  | [info] [pid:180] Running on 93239f7bc06e:1 (Linux 6.1.62 #1-NixOS SMP PREEMPT_DYNAMIC Wed Nov  8 13:11:05 UTC 2023 x86_64)
worker-1  | [debug] [pid:180] Job settings:
worker-1  |     ARCH=x86_64
worker-1  |     BACKEND=qemu
worker-1  |     DESKTOP=xfce
worker-1  |     DISTRI=demohw
worker-1  |     FLAVOR=generalhw
worker-1  |     GENERAL_HW_CMD_DIR=/var/lib/openqa/share/tests/openeuler/data/generalhw_scripts
worker-1  |     GENERAL_HW_REBOOT_ARGS=192.168.0.76
worker-1  |     GENERAL_HW_REBOOT_CMD=reboot_SUT_via_SSH.sh
worker-1  |     GENERAL_HW_SOL_ARGS=ttyUSB1    #如果需要连接 vnc 没连串口也要配置
worker-1  |     GENERAL_HW_SOL_CMD=get_sol_dev.sh    #如果需要连接 vnc 没连串口也要配置
worker-1  |     GENERAL_HW_VNC_IP=192.168.0.76    #桌面测试时需要配置
worker-1  |     GENERAL_HW_VNC_PASSWORD=[redacted]
worker-1  |     GENERAL_HW_VNC_PORT=5901    #桌面测试时需要配置
worker-1  |     HDDSIZEGB=100
worker-1  |     INSTALL=1
worker-1  |     JOBTOKEN=FaE9jRmaYObvjmKG
worker-1  |     LOG_LEVEL=debug
worker-1  |     MACHINE=demohw
worker-1  |     NAME=00000058-demohw-v1-generalhw-x86_64-demohw@demohw
worker-1  |     NICTYPE=user
worker-1  |     OPENQA_HOSTNAME=172.18.0.9
worker-1  |     OPENQA_URL=http://172.18.0.9
worker-1  |     PRJDIR=/var/lib/openqa/share
worker-1  |     QEMUPORT=20012
worker-1  |     RETRY_DELAY=5
worker-1  |     RETRY_DELAY_IF_WEBUI_BUSY=60
worker-1  |     SSH_XTERM_WAIT_SUT_ALIVE_TIMEOUT=60
worker-1  |     SUT_IP=192.168.0.76
worker-1  |     TEST=demohw
worker-1  |     TEST_SUITE_NAME=demohw
worker-1  |     VERSION=v1
worker-1  |     VIDEOMODE=vnc
worker-1  |     VNC=91
worker-1  |     VNC_TYPING_LIMIT=7
worker-1  |     WORKER_CLASS=demohw
worker-1  |     WORKER_HOSTNAME=worker
worker-1  |     WORKER_ID=14
worker-1  |     WORKER_INSTANCE=1
worker-1  |     YAML_SCHEDULE=schedule/demohw.yaml
worker-1  | [debug] [pid:180] Symlinked from "/var/lib/openqa/share/tests/demohw" to "/var/lib/openqa/pool/1/demohw"
worker-1  | [info] [pid:180] Preparing cgroup to start isotovideo
worker-1  | [warn] [pid:180] Disabling cgroup usage because cgroup creation failed: mkdir /sys/fs/cgroup/systemd: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/File.pm line 85.
worker-1  | [info] [pid:180] You can define a custom slice with OPENQA_CGROUP_SLICE or indicating the base mount with MOJO_CGROUP_FS.
worker-1  | [info] [pid:180] Starting isotovideo container
worker-1  | [debug] [pid:180] Registered process:183
worker-1  | [info] [pid:180] isotovideo has been started (PID: 183)
worker-1  | [info] [pid:183] 183: WORKING 58
worker-1  | [debug] [pid:183] +++ worker notes +++
worker-1  | [debug] [pid:180] Running job 58 from http://172.18.0.9: 00000058-demohw-v1-generalhw-x86_64-demohw@demohw.
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/status
worker-1  | [debug] [pid:180] Upload concluded (no current module)
worker-1  | [debug] [pid:180] Starting livelog
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/status
worker-1  | [debug] [pid:180] Upload concluded (no current module)
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/status
worker-1  | [debug] [pid:180] Upload concluded (at boot)
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/status
worker-1  | [debug] [pid:180] Upload concluded (at boot)
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/status
worker-1  | [debug] [pid:180] Upload concluded (at boot)
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/status
worker-1  | [debug] [pid:180] Upload concluded (up to boot)
worker-1  | [info] [pid:180] Isotovideo exit status: 0
worker-1  | [debug] [pid:180] Stopping job 58 from http://172.18.0.9: 00000058-demohw-v1-generalhw-x86_64-demohw@demohw - reason: done
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/status
worker-1  | [info] [pid:180] +++ worker notes +++
worker-1  | [info] [pid:180] End time: 2023-12-09 13:50:39
worker-1  | [info] [pid:180] Result: done
worker-1  | [info] [pid:303] Uploading video.ogv
worker-1  | [debug] [pid:303] Uploading artefact video.ogv
worker-1  | [info] [pid:303] Uploading autoinst-log.txt
worker-1  | [debug] [pid:303] Uploading artefact autoinst-log.txt
worker-1  | [info] [pid:303] Uploading worker-log.txt
worker-1  | [debug] [pid:303] Uploading artefact worker-log.txt
worker-1  | [warn] [pid:303] Skipping upload of vars.json because an error occurred when stripping credentials: Wide character in syswrite at /usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/IO/Handle.pm line 483.
worker-1  |     IO::Handle::syswrite(GLOB(0x562a88816bc8), "{\x{a}   \"ARCH\" : \"x86_64\",\x{a}   \"ASSETDIR\" : \"/var/lib/openqa/shar"...) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/File.pm line 145
worker-1  |     Mojo::File::spew(Mojo::File=SCALAR(0x562a88848c50), "{\x{a}   \"ARCH\" : \"x86_64\",\x{a}   \"ASSETDIR\" : \"/var/lib/openqa/shar"...) called at /usr/share/openqa/script/../lib/OpenQA/Log.pm line 217
worker-1  |     OpenQA::Log::redact_settings_in_file("/var/lib/openqa/pool/1/vars.json") called at /usr/share/openqa/script/../lib/OpenQA/Worker/Job.pm line 394
worker-1  |     eval {...} called at /usr/share/openqa/script/../lib/OpenQA/Worker/Job.pm line 394
worker-1  |     OpenQA::Worker::Job::_redact_file("/var/lib/openqa/pool/1/vars.json", "vars.json") called at /usr/share/openqa/script/../lib/OpenQA/Worker/Job.pm line 462
worker-1  |     OpenQA::Worker::Job::__ANON__(Mojo::IOLoop::Subprocess=HASH(0x562a887e0508)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 53
worker-1  |     eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 53
worker-1  |     eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 51
worker-1  |     Mojo::IOLoop::Subprocess::_start(Mojo::IOLoop::Subprocess=HASH(0x562a887e0508), CODE(0x562a887ef648), CODE(0x562a887ef8b8)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/Subprocess.pm line 21
worker-1  |     Mojo::IOLoop::Subprocess::__ANON__(Mojo::IOLoop=HASH(0x562a86b841e0)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop.pm line 72
worker-1  |     Mojo::IOLoop::__ANON__(Mojo::Reactor::Poll=HASH(0x562a853cf518)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Reactor/Poll.pm line 129
worker-1  |     Mojo::Reactor::Poll::_next(Mojo::Reactor::Poll=HASH(0x562a853cf518)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Reactor/Poll.pm line 141
worker-1  |     eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Reactor/Poll.pm line 141
worker-1  |     Mojo::Reactor::Poll::_try(Mojo::Reactor::Poll=HASH(0x562a853cf518), "Timer", CODE(0x562a86b7c690)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Reactor/Poll.pm line 83
worker-1  |     Mojo::Reactor::Poll::one_tick(Mojo::Reactor::Poll=HASH(0x562a853cf518)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Reactor/Poll.pm line 101
worker-1  |     Mojo::Reactor::Poll::start(Mojo::Reactor::Poll=HASH(0x562a853cf518)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop.pm line 134
worker-1  |     Mojo::IOLoop::start(Mojo::IOLoop=HASH(0x562a86b841e0)) called at /usr/share/openqa/script/../lib/OpenQA/Worker.pm line 364
worker-1  |     OpenQA::Worker::exec(OpenQA::Worker=HASH(0x562a886fce70)) called at /usr/share/openqa/script/worker line 125
worker-1  | 
worker-1  | [info] [pid:303] Uploading serial0.txt
worker-1  | [debug] [pid:303] Uploading artefact serial0.txt
worker-1  | [info] [pid:303] Uploading video_time.vtt
worker-1  | [debug] [pid:303] Uploading artefact video_time.vtt
worker-1  | [info] [pid:303] Uploading serial_terminal.txt
worker-1  | [debug] [pid:303] Uploading artefact serial_terminal.txt
worker-1  | [info] [pid:303] Uploading serial_terminal_user.txt
worker-1  | [debug] [pid:303] Uploading artefact serial_terminal_user.txt
worker-1  | [debug] [pid:180] Considering job 58 as regularly done
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/status
worker-1  | [debug] [pid:180] Upload concluded (no current module)
worker-1  | [debug] [pid:180] REST-API call: POST http://172.18.0.9/api/v1/jobs/58/set_done?worker_id=14
worker-1  | [debug] [pid:180] Job 58 from http://172.18.0.9 finished - reason: done
worker-1  | [debug] [pid:180] Cleaning up for next job

Other stuff I've done besides:

  1. delete replicas

    deploy: 
    replicas: $XXX

    parts in both webui and worker docker-compose.yaml,

  2. use webui_default network in worker

    networks:
  default:
    name: webui_default
    driver: bridge
    external: true

  3. set in HOST to http://172.18.0.9 in worker.ini and client.conf

    It is the ip of webui-nginx-1, I get this from docker inspect

  4. set GENERAL_HW_CMD_DIR, ... in worker.ini, I was trying to test another running desktop, they should have no effect, since I've switched the backend to qemu.

btw, is generalhw the backend I need if i just want to test using ssh(for server), vnc(for server and desktop)?

Martchus commented 11 months ago

everything that comes after the worker registration

I use docker-compose up --build start a new worker, rerun the test, here is the log:

I still meant on the web UI side.

Maybe I'll find the time next week to revisit our docker compose setup and possibly reproduce the issue. I would recommend to setup openQA using plain old RPMs in the meantime.

btw, is generalhw the backend I need if i just want to test using ssh(for server), vnc(for server and desktop)?

I'm not sure what you mean by that. If you really don't want to test using a VM but using real hardware instead then generalhw might be the right backend, indeed (see https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc for relevant variables and e.g. https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/data/generalhw_scripts/raspberry_pi_hardware_testing_setup.md for and example using a Raspberry Pi).

okurz commented 11 months ago

@trdthg maybe not the original problem but also suspicious are the entries in the logs with the non-ASCII characters and the according warning, see

worker-1  |     GENERAL_HW_SOL_ARGS=ttyUSB1    #������ vnc 没�串����置
worker-1  |     GENERAL_HW_SOL_CMD=get_sol_dev.sh    #������ vnc 没�串����置
worker-1  |     GENERAL_HW_VNC_IP=192.168.0.76    #��������置
worker-1  |     GENERAL_HW_VNC_PASSWORD=[redacted]
worker-1  |     GENERAL_HW_VNC_PORT=5901    #��������置
...
worker-1  | [warn] [pid:303] Skipping upload of vars.json because an error occurred when stripping credentials: Wide character in syswrite at /usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/IO/Handle.pm line 483.

Could it be that your VNC password has non-ASCII characters included?

trdthg commented 11 months ago

Could it be that your VNC password has non-ASCII characters included?

U_U Maybe not, my vnc password is just 123456

EDIT: and vnc connection is ok

okurz commented 11 months ago

Then I don't know where those characters come from but that's not how it normally looks :)

trdthg commented 11 months ago

just some comment in Chinese, I wrote those myself, (just checked that it's legal to use # or ; as a comment in .ini). I will delete them and try again later.

trdthg commented 11 months ago

I will delete them and try again later.

Didn't see anything new.

webui log

websockets-1  | [info] Listening at "http://0.0.0.0:9527"
scheduler-1   | [info] Listening at "http://0.0.0.0:9529"
nginx-1       | NGINX configuration:
websockets-1  | Web application available at http://0.0.0.0:9527
nginx-1       | events {
nginx-1       | }
websockets-1  | [info] Worker 15 websocket connection closed - 1006
websockets-1  | [info] Worker 15 websocket connection closed - 1006
nginx-1       | 
websockets-1  | [info] Worker 15 websocket connection closed - 1006
nginx-1       | http {
nginx-1       |     upstream webui {
nginx-1       |         server webui:9526;
nginx-1       |     }
nginx-1       | 
nginx-1       |     map $http_upgrade $connection_upgrade {
websockets-1  | [info] Worker 15 websocket connection closed - 1006
webui-1       | Waiting for DB creation
nginx-1       |         default upgrade;
websockets-1  | [info] Worker 15 websocket connection closed - 1006
nginx-1       |         '' close;
webui-1       | [info] Listening at "http://0.0.0.0:9526"
websockets-1  | [info] Worker 15 websocket connection closed - 1006
db-1          | 
websockets-1  | [info] Worker 15 websocket connection closed - 1006
nginx-1       |     }
webui-1       | Web application available at http://0.0.0.0:9526
webui-1       | [info] Manager 19 started
livehandler-1  | Waiting for DB creation
db-1           | PostgreSQL Database directory appears to contain a database; Skipping initialization
webui_db_init-1  | Waiting for DB creation
nginx-1          | 
gru-1            | Waiting for DB creation
livehandler-1    | [info] Listening at "http://0.0.0.0:9528"
nginx-1          |     server {
webui_db_init-1  | psql: error: connection to server at "db" (172.18.0.2), port 5432 failed: Connection refused
webui-1          | [info] Worker 21 started
gru-1            | [info] Resetting all leftover Gru locks after restart
db-1             | 
webui_db_init-1  |  Is the server running on that host and accepting TCP/IP connections?
nginx-1          |         listen       80;
webui-1          | [info] Worker 22 started
webui-1          | [info] Worker 23 started
nginx-1          |         listen       443 ssl;
webui_db_init-1  | [info] Listening at "http://0.0.0.0:9526"
webui_db_init-1  | Web application available at http://0.0.0.0:9526
nginx-1          |         listen       9526;
webui-1          | [info] Worker 24 started
webui-1          | [info] Worker 25 started
nginx-1          |         server_name  localhost;
webui_db_init-1  | [info] Manager 26 started
webui-1          | [info] Worker 26 started
nginx-1          |         client_max_body_size 0;
webui_db_init-1  | [info] Worker 28 started
webui-1          | [info] Worker 27 started
webui_db_init-1  | [info] Worker 29 started
nginx-1          | 
webui-1          | [info] Worker 28 started
nginx-1          |         ssl_certificate     /etc/ssl/certs/openqa.crt;
webui_db_init-1  | [info] Worker 30 started
webui-1          | [info] Worker 29 started
nginx-1          |         ssl_certificate_key /etc/ssl/certs/openqa.key;
webui_db_init-1  | [info] Worker 31 started
webui-1          | [info] Worker 30 started
nginx-1          |         ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
webui_db_init-1  | [info] Worker 32 started
db-1             | 2023-12-11 12:15:40.094 UTC [1] LOG:  starting PostgreSQL 15.1 (Debian 15.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
gru-1            | [info] Worker 21 started
webui-1          | [info] Worker 31 started
nginx-1          |         ssl_ciphers         HIGH:!aNULL:!MD5;
webui_db_init-1  | [info] Worker 33 started
db-1             | 2023-12-11 12:15:40.095 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
nginx-1          | 
webui-1          | [info] Worker 32 started
webui_db_init-1  | [info] Worker 34 started
db-1             | 2023-12-11 12:15:40.095 UTC [1] LOG:  listening on IPv6 address "::", port 5432
nginx-1          |         location /api/v1/ws/ {
webui-1          | [info] Worker 33 started
webui_db_init-1  | [info] Worker 35 started
db-1             | 2023-12-11 12:15:40.100 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
webui-1          | [info] Worker 34 started
webui_db_init-1  | [info] Worker 36 started
db-1             | 2023-12-11 12:15:40.109 UTC [28] LOG:  database system was shut down at 2023-12-11 12:15:36 UTC
webui_db_init-1  | [info] Worker 37 started
webui-1          | [info] Worker 35 started
webui-1          | [info] Worker 36 started
db-1             | 2023-12-11 12:15:40.121 UTC [1] LOG:  database system is ready to accept connections
webui_db_init-1  | [info] Worker 38 started
nginx-1          |             proxy_pass http://websockets:9527;
webui_db_init-1  | [info] Worker 39 started
webui-1          | [info] Worker 37 started
nginx-1          |             proxy_http_version 1.1;
webui_db_init-1  | [info] Worker 40 started
webui-1          | [info] Worker 38 started
nginx-1          |             proxy_set_header Upgrade $http_upgrade;
webui_db_init-1  | [info] Worker 41 started
webui-1          | [info] Worker 39 started
nginx-1          |             proxy_set_header Connection $connection_upgrade;
webui_db_init-1  | [info] Worker 42 started
webui-1          | [info] Worker 40 started
nginx-1          |             proxy_set_header Host $host;
webui_db_init-1  | [info] Worker 43 started
webui-1          | [info] Worker 41 started
nginx-1          |         }
webui_db_init-1  | [info] Worker 44 started
db-1             | 2023-12-11 12:17:54.015 UTC [1] LOG:  received fast shutdown request
nginx-1          | 
webui-1          | [info] Worker 42 started
webui_db_init-1  | [info] Worker 45 started
webui-1          | [info] Worker 43 started
nginx-1          |         location /liveviewhandler/ {
db-1             | 2023-12-11 12:17:54.018 UTC [1] LOG:  aborting any active transactions
webui_db_init-1  | [info] Worker 46 started
webui-1          | [info] Worker 44 started
nginx-1          |             proxy_pass http://livehandler:9528;
db-1             | 2023-12-11 12:17:54.019 UTC [1] LOG:  background worker "logical replication launcher" (PID 31) exited with exit code 1
webui_db_init-1  | [info] Worker 47 started
webui-1          | [info] Worker 45 started
nginx-1          |             proxy_http_version 1.1;
webui_db_init-1  | [info] Worker 48 started
db-1             | 2023-12-11 12:17:54.019 UTC [26] LOG:  shutting down
webui-1          | [info] Worker 46 started
nginx-1          |             proxy_set_header Upgrade $http_upgrade;
webui_db_init-1  | [info] Worker 49 started
webui-1          | [info] Worker 47 started
db-1             | 2023-12-11 12:17:54.022 UTC [26] LOG:  checkpoint starting: shutdown immediate
nginx-1          |             proxy_set_header Connection $connection_upgrade;
webui_db_init-1  | [info] Worker 50 started
webui-1          | [info] Worker 48 started
db-1             | 2023-12-11 12:17:54.064 UTC [26] LOG:  checkpoint complete: wrote 43 buffers (0.3%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.010 s, sync=0.009 s, total=0.045 s; sync files=28, longest=0.006 s, average=0.001 s; distance=158 kB, estimate=158 kB
nginx-1          |             proxy_set_header Host $host;
webui_db_init-1  | [info] Worker 51 started
webui-1          | [info] Worker 49 started
db-1             | 2023-12-11 12:17:54.067 UTC [1] LOG:  database system is shut down
nginx-1          |         }
webui_db_init-1  | [info] Worker 52 started
webui-1          | [info] Creating process id file "/tmp/prefork.pid"
db-1             | 
nginx-1          | 
webui_db_init-1  | [info] Worker 53 started
webui-1          | [info] Worker 50 started
db-1             | PostgreSQL Database directory appears to contain a database; Skipping initialization
nginx-1          |         location /assets/ {
webui_db_init-1  | [info] Worker 54 started
nginx-1          |             root /usr/share/openqa/assets;
db-1             | 
webui_db_init-1  | [info] Worker 55 started
db-1             | 2023-12-11 12:27:10.601 UTC [1] LOG:  starting PostgreSQL 15.1 (Debian 15.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
nginx-1          |         }
webui_db_init-1  | [info] Worker 56 started
nginx-1          | 
db-1             | 2023-12-11 12:27:10.602 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
webui_db_init-1  | [info] Worker 57 started
nginx-1          |         location / {
db-1             | 2023-12-11 12:27:10.602 UTC [1] LOG:  listening on IPv6 address "::", port 5432
webui_db_init-1  | [info] Creating process id file "/tmp/prefork.pid"
nginx-1          |             proxy_set_header X-Forwarded-Host $host:$server_port;
db-1             | 2023-12-11 12:27:10.607 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
nginx-1          |             proxy_set_header X-Forwarded-Server $host;
db-1             | 2023-12-11 12:27:10.615 UTC [28] LOG:  database system was shut down at 2023-12-11 12:17:54 UTC
nginx-1          |             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
db-1             | 2023-12-11 12:27:10.628 UTC [1] LOG:  database system is ready to accept connections
nginx-1          |             proxy_pass http://webui;
nginx-1          |         }
nginx-1          |     }
nginx-1          | }
db-1             | 2023-12-11 12:32:10.664 UTC [26] LOG:  checkpoint starting: time
db-1             | 2023-12-11 12:32:15.030 UTC [26] LOG:  checkpoint complete: wrote 45 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.330 s, sync=0.017 s, total=4.367 s; sync files=17, longest=0.008 s, average=0.001 s; distance=169 kB, estimate=169 kB
db-1             | 2023-12-11 12:37:10.127 UTC [26] LOG:  checkpoint starting: time
db-1             | 2023-12-11 12:37:14.585 UTC [26] LOG:  checkpoint complete: wrote 45 buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.423 s, sync=0.014 s, total=4.458 s; sync files=21, longest=0.008 s, average=0.001 s; distance=139 kB, estimate=166 kB
db-1             | 2023-12-11 12:42:10.663 UTC [26] LOG:  checkpoint starting: time
db-1             | 2023-12-11 12:42:20.849 UTC [26] LOG:  checkpoint complete: wrote 102 buffers (0.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=10.148 s, sync=0.017 s, total=10.187 s; sync files=75, longest=0.009 s, average=0.001 s; distance=283 kB, estimate=283 kB

worker log

worker-1  | Retrieving repository 'devel_openQA' metadata [.............done]
worker-1  | Building repository 'devel_openQA' cache [...done]
worker-1  | Retrieving repository 'Update repository of openSUSE Backports' metadata [...............done]
worker-1  | Building repository 'Update repository of openSUSE Backports' cache [....done]
worker-1  | Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata [...................................................................................................................................................................................................................done]
worker-1  | Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache [....done]
worker-1  | Loading repository data...
worker-1  | Reading installed packages...
worker-1  | Resolving package dependencies...
worker-1  | 
worker-1  | The following 60 NEW packages are going to be installed:
worker-1  |   lsof os-autoinst-distri-opensuse-deps perl-Archive-Zip perl-B-Hooks-EndOfScope perl-B-Keywords perl-Class-Inspector perl-Class-Singleton perl-Class-Tiny perl-Clone perl-Code-DRY perl-DateTime perl-DateTime-Locale perl-DateTime-TimeZone perl-Dist-CheckConflicts perl-Eval-Closure perl-Exception-Base perl-File-Path perl-File-ShareDir perl-IO-String perl-JSON perl-Lingua-EN-Inflect perl-List-SomeUtils perl-List-SomeUtils-XS perl-Module-Build perl-Module-Pluggable perl-NetAddr-IP perl-PPI perl-PPIx-QuoteLike perl-PPIx-Regexp perl-PPIx-Utilities perl-Package-Stash perl-Package-Stash-XS perl-Params-ValidationCompiler perl-Path-Tiny perl-Perl-Critic perl-Perl-Critic-Community perl-Perl-Critic-Policy-Variables-ProhibitLoopOnHash perl-Perl-Critic-Pulp perl-Perl-Tidy perl-Pod-MinimumVersion perl-Pod-Spell perl-Readonly perl-Selenium-Remote-Driver perl-SemVer perl-Specio perl-String-Format perl-Sub-Identify perl-Symbol-Util perl-Task-Weaken perl-Test-Assert perl-Test-Fatal perl-Test-LongString perl-Tie-IxHash perl-Variable-Magic perl-XML-SAX-Expat perl-XML-Simple perl-XML-Writer perl-constant-boolean perl-namespace-autoclean perl-namespace-clean
worker-1  | 
worker-1  | 60 new packages to install.
worker-1  | Overall download size: 8.9 MiB. Already cached: 0 B. After the operation, additional 36.2 MiB will be used.
worker-1  | Continue? [y/n/v/...? shows all options] (y): y
worker-1  | Retrieving: perl-Perl-Tidy-20230912.0.0-lp155.2.1.noarch (devel_openQA_Leap) (1/60), 768.9 KiB    
worker-1  | Retrieving: perl-Perl-Tidy-20230912.0.0-lp155.2.1.noarch.rpm [......................done (473.6 KiB/s)]
worker-1  | Retrieving: lsof-4.91-1.11.x86_64 (Main Repository) (2/60), 330.9 KiB    
worker-1  | Retrieving: lsof-4.91-1.11.x86_64.rpm [.....done (2.8 MiB/s)]
worker-1  | Retrieving: perl-Archive-Zip-1.60-3.3.1.noarch (Main Repository) (3/60), 103.4 KiB    
worker-1  | Retrieving: perl-Archive-Zip-1.60-3.3.1.noarch.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-B-Keywords-1.24-bp155.1.5.noarch (Main Repository) (4/60),  21.2 KiB    
worker-1  | Retrieving: perl-B-Keywords-1.24-bp155.1.5.noarch.rpm [..done (3.9 KiB/s)]
worker-1  | Retrieving: perl-Class-Inspector-1.32-1.22.noarch (Main Repository) (5/60),  32.0 KiB    
worker-1  | Retrieving: perl-Class-Inspector-1.32-1.22.noarch.rpm [..done (2.7 KiB/s)]
worker-1  | Retrieving: perl-Class-Singleton-1.5-1.24.noarch (Main Repository) (6/60),  19.3 KiB    
worker-1  | Retrieving: perl-Class-Singleton-1.5-1.24.noarch.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Class-Tiny-1.006-1.24.noarch (Main Repository) (7/60),  30.8 KiB    
worker-1  | Retrieving: perl-Class-Tiny-1.006-1.24.noarch.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Clone-0.46-bp155.1.5.x86_64 (Main Repository) (8/60),  21.9 KiB    
worker-1  | Retrieving: perl-Clone-0.46-bp155.1.5.x86_64.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Dist-CheckConflicts-0.11-1.23.noarch (Main Repository) (9/60),  24.8 KiB    
worker-1  | Retrieving: perl-Dist-CheckConflicts-0.11-1.23.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Exception-Base-0.2501-bp155.2.8.noarch (Main Repository) (10/60),  52.6 KiB    
worker-1  | Retrieving: perl-Exception-Base-0.2501-bp155.2.8.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-File-Path-2.150000-1.22.noarch (Main Repository) (11/60),  39.5 KiB    
worker-1  | Retrieving: perl-File-Path-2.150000-1.22.noarch.rpm [...done (4.0 KiB/s)]
worker-1  | Retrieving: perl-IO-String-1.08-152.24.x86_64 (Main Repository) (12/60),  17.0 KiB    
worker-1  | Retrieving: perl-IO-String-1.08-152.24.x86_64.rpm [....done (3.8 KiB/s)]
worker-1  | Retrieving: perl-JSON-2.97001-1.18.noarch (Main Repository) (13/60),  92.4 KiB    
worker-1  | Retrieving: perl-JSON-2.97001-1.18.noarch.rpm [...done]
worker-1  | Retrieving: perl-Lingua-EN-Inflect-1.905-bp155.2.12.noarch (Main Repository) (14/60),  60.6 KiB    
worker-1  | Retrieving: perl-Lingua-EN-Inflect-1.905-bp155.2.12.noarch.rpm [...done (4.1 KiB/s)]
worker-1  | Retrieving: perl-List-SomeUtils-XS-0.56-1.17.x86_64 (Main Repository) (15/60),  41.3 KiB    
worker-1  | Retrieving: perl-List-SomeUtils-XS-0.56-1.17.x86_64.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Module-Build-0.422400-1.24.noarch (Main Repository) (16/60), 259.4 KiB    
worker-1  | Retrieving: perl-Module-Build-0.422400-1.24.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Module-Pluggable-5.2-1.24.noarch (Main Repository) (17/60),  33.6 KiB    
worker-1  | Retrieving: perl-Module-Pluggable-5.2-1.24.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-NetAddr-IP-4.079-1.27.x86_64 (Main Repository) (18/60), 120.4 KiB    
worker-1  | Retrieving: perl-NetAddr-IP-4.079-1.27.x86_64.rpm [...done (1.4 MiB/s)]
worker-1  | Retrieving: perl-Package-Stash-XS-0.28-1.26.x86_64 (Main Repository) (19/60),  30.9 KiB    
worker-1  | Retrieving: perl-Package-Stash-XS-0.28-1.26.x86_64.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Readonly-2.05-1.22.noarch (Main Repository) (20/60),  36.5 KiB    
worker-1  | Retrieving: perl-Readonly-2.05-1.22.noarch.rpm [.....done (8.7 KiB/s)]
worker-1  | Retrieving: perl-SemVer-0.10.0-bp155.1.4.noarch (Main Repository) (21/60),  20.4 KiB    
worker-1  | Retrieving: perl-SemVer-0.10.0-bp155.1.4.noarch.rpm [...done]
worker-1  | Retrieving: perl-String-Format-1.18-bp155.2.12.noarch (Main Repository) (22/60),  23.1 KiB    
worker-1  | Retrieving: perl-String-Format-1.18-bp155.2.12.noarch.rpm [...done (4.0 KiB/s)]
worker-1  | Retrieving: perl-Sub-Identify-0.14-1.26.x86_64 (Main Repository) (23/60),  20.2 KiB    
worker-1  | Retrieving: perl-Sub-Identify-0.14-1.26.x86_64.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Symbol-Util-0.0203-bp155.2.12.noarch (Main Repository) (24/60),  30.7 KiB    
worker-1  | Retrieving: perl-Symbol-Util-0.0203-bp155.2.12.noarch.rpm [...done]
worker-1  | Retrieving: perl-Task-Weaken-1.06-2.13.noarch (Main Repository) (25/60),  24.0 KiB    
worker-1  | Retrieving: perl-Task-Weaken-1.06-2.13.noarch.rpm [.....done (2.0 KiB/s)]
worker-1  | Retrieving: perl-Test-Fatal-0.014-1.23.noarch (Main Repository) (26/60),  24.1 KiB    
worker-1  | Retrieving: perl-Test-Fatal-0.014-1.23.noarch.rpm [...done (3.9 KiB/s)]
worker-1  | Retrieving: perl-Test-LongString-0.17-bp155.2.12.noarch (Main Repository) (27/60),  17.8 KiB    
worker-1  | Retrieving: perl-Test-LongString-0.17-bp155.2.12.noarch.rpm [...done (4.0 KiB/s)]
worker-1  | Retrieving: perl-Tie-IxHash-1.23-1.23.noarch (Main Repository) (28/60),  18.4 KiB    
worker-1  | Retrieving: perl-Tie-IxHash-1.23-1.23.noarch.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Variable-Magic-0.62-1.19.x86_64 (Main Repository) (29/60),  62.2 KiB    
worker-1  | Retrieving: perl-Variable-Magic-0.62-1.19.x86_64.rpm [...done]
worker-1  | Retrieving: perl-XML-SAX-Expat-0.51-1.22.noarch (Main Repository) (30/60),  16.1 KiB    
worker-1  | Retrieving: perl-XML-SAX-Expat-0.51-1.22.noarch.rpm [...done (270.4 KiB/s)]
worker-1  | Retrieving: perl-XML-Writer-0.625-1.24.noarch (Main Repository) (31/60),  34.6 KiB    
worker-1  | Retrieving: perl-XML-Writer-0.625-1.24.noarch.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Eval-Closure-0.14-1.22.noarch (Main Repository) (32/60),  25.2 KiB    
worker-1  | Retrieving: perl-Eval-Closure-0.14-1.22.noarch.rpm [....done (4.0 KiB/s)]
worker-1  | Retrieving: perl-File-ShareDir-1.104-1.22.noarch (Main Repository) (33/60),  27.0 KiB    
worker-1  | Retrieving: perl-File-ShareDir-1.104-1.22.noarch.rpm [...done (3.5 KiB/s)]
worker-1  | Retrieving: perl-Path-Tiny-0.104-1.22.noarch (Main Repository) (34/60),  77.7 KiB    
worker-1  | Retrieving: perl-Path-Tiny-0.104-1.22.noarch.rpm [.....done]
worker-1  | Retrieving: perl-Pod-MinimumVersion-50-bp155.3.10.noarch (Main Repository) (35/60),  35.0 KiB    
worker-1  | Retrieving: perl-Pod-MinimumVersion-50-bp155.3.10.noarch.rpm [...done]
worker-1  | Retrieving: perl-List-SomeUtils-0.56-1.20.noarch (Main Repository) (36/60),  46.9 KiB    
worker-1  | Retrieving: perl-List-SomeUtils-0.56-1.20.noarch.rpm [...done]
worker-1  | Retrieving: perl-Code-DRY-0.10-bp155.2.10.x86_64 (Main Repository) (37/60),  44.0 KiB    
worker-1  | Retrieving: perl-Code-DRY-0.10-bp155.2.10.x86_64.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: perl-Package-Stash-0.37-1.23.noarch (Main Repository) (38/60),  35.7 KiB    
worker-1  | Retrieving: perl-Package-Stash-0.37-1.23.noarch.rpm [..done]
worker-1  | Retrieving: perl-constant-boolean-0.02-bp155.2.8.noarch (Main Repository) (39/60),  23.6 KiB    
worker-1  | Retrieving: perl-constant-boolean-0.02-bp155.2.8.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-PPI-1.276-bp155.1.4.noarch (Main Repository) (40/60), 349.9 KiB    
worker-1  | Retrieving: perl-PPI-1.276-bp155.1.4.noarch.rpm [...done (3.3 MiB/s)]
worker-1  | Retrieving: perl-B-Hooks-EndOfScope-0.21-1.22.noarch (Main Repository) (41/60),  30.3 KiB    
worker-1  | Retrieving: perl-B-Hooks-EndOfScope-0.21-1.22.noarch.rpm [.....done]
worker-1  | Retrieving: perl-XML-Simple-2.24-1.22.noarch (Main Repository) (42/60),  81.4 KiB    
worker-1  | Retrieving: perl-XML-Simple-2.24-1.22.noarch.rpm [......done (799.8 KiB/s)]
worker-1  | Retrieving: perl-Specio-0.42-1.17.noarch (Main Repository) (43/60), 160.8 KiB    
worker-1  | Retrieving: perl-Specio-0.42-1.17.noarch.rpm [....done (3.8 KiB/s)]
worker-1  | Retrieving: perl-Params-ValidationCompiler-0.26-1.17.noarch (Main Repository) (44/60),  34.0 KiB    
worker-1  | Retrieving: perl-Params-ValidationCompiler-0.26-1.17.noarch.rpm [...done]
worker-1  | Retrieving: perl-Pod-Spell-1.25-bp155.1.5.noarch (Main Repository) (45/60),  44.2 KiB    
worker-1  | Retrieving: perl-Pod-Spell-1.25-bp155.1.5.noarch.rpm [....done (3.9 KiB/s)]
worker-1  | Retrieving: perl-Test-Assert-0.0504-bp155.2.8.noarch (Main Repository) (46/60),  34.6 KiB    
worker-1  | Retrieving: perl-Test-Assert-0.0504-bp155.2.8.noarch.rpm [.....done]
worker-1  | Retrieving: perl-PPIx-Utilities-1.001000-bp155.3.9.noarch (Main Repository) (47/60),  36.2 KiB    
worker-1  | Retrieving: perl-PPIx-Utilities-1.001000-bp155.3.9.noarch.rpm [...done]
worker-1  | Retrieving: perl-PPIx-Regexp-0.086-bp155.1.5.noarch (Main Repository) (48/60), 329.3 KiB    
worker-1  | Retrieving: perl-PPIx-Regexp-0.086-bp155.1.5.noarch.rpm [....done (3.1 MiB/s)]
worker-1  | Retrieving: perl-PPIx-QuoteLike-0.023-bp155.1.5.noarch (Main Repository) (49/60),  86.5 KiB    
worker-1  | Retrieving: perl-PPIx-QuoteLike-0.023-bp155.1.5.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-namespace-clean-0.27-1.22.noarch (Main Repository) (50/60),  24.0 KiB    
worker-1  | Retrieving: perl-namespace-clean-0.27-1.22.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Perl-Critic-1.148-bp155.1.4.noarch (Main Repository) (51/60), 894.5 KiB    
worker-1  | Retrieving: perl-Perl-Critic-1.148-bp155.1.4.noarch.rpm [.....done (2.0 MiB/s)]
worker-1  | Retrieving: perl-namespace-autoclean-0.28-1.22.noarch (Main Repository) (52/60),  27.7 KiB    
worker-1  | Retrieving: perl-namespace-autoclean-0.28-1.22.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Perl-Critic-Pulp-99-bp155.2.8.x86_64 (Main Repository) (53/60), 238.3 KiB    
worker-1  | Retrieving: perl-Perl-Critic-Pulp-99-bp155.2.8.x86_64.rpm [....done]
worker-1  | Retrieving: perl-Perl-Critic-Policy-Variables-ProhibitLoopOnHash-0.008-bp155.2.8.noarch (Main Repository) (54/60),  21.4 KiB    
worker-1  | Retrieving: perl-Perl-Critic-Policy-Variables-ProhibitLoopOnHash-0.008-bp155.2.8.noarch.rpm [...done]
worker-1  | Retrieving: perl-DateTime-Locale-1.170000-1.17.noarch (Main Repository) (55/60),   2.5 MiB    
worker-1  | Retrieving: perl-DateTime-Locale-1.170000-1.17.noarch.rpm [.........done (4.2 MiB/s)]
worker-1  | Retrieving: perl-Perl-Critic-Community-1.0.3-bp155.1.4.noarch (Main Repository) (56/60), 160.0 KiB    
worker-1  | Retrieving: perl-Perl-Critic-Community-1.0.3-bp155.1.4.noarch.rpm [..done (2.1 KiB/s)]
worker-1  | Retrieving: perl-Selenium-Remote-Driver-1.49-bp155.2.3.1.x86_64 (Update repository of openSUSE Backports) (57/60), 816.3 KiB    
worker-1  | Retrieving: perl-Selenium-Remote-Driver-1.49-bp155.2.3.1.x86_64.rpm [........done (3.4 MiB/s)]
worker-1  | Retrieving: perl-DateTime-TimeZone-2.60-150000.3.3.1.noarch (Update repository with updates from SUSE Linux Enterprise 15) (58/60), 354.3 KiB    
worker-1  | Retrieving: perl-DateTime-TimeZone-2.60-150000.3.3.1.noarch.rpm [....done (184.2 KiB/s)]
worker-1  | Retrieving: perl-DateTime-1.45-1.16.x86_64 (Main Repository) (59/60), 143.0 KiB    
worker-1  | Retrieving: perl-DateTime-1.45-1.16.x86_64.rpm [..done (2.2 KiB/s)]
worker-1  | Retrieving: os-autoinst-distri-opensuse-deps-1.1702280286.cc5583fe-lp155.13341.1.noarch (devel_openQA) (60/60),   8.2 KiB    
worker-1  | Retrieving: os-autoinst-distri-opensuse-deps-1.1702280286.cc5583fe-lp155.13341.1.noarch.rpm [......done (5.1 KiB/s)]
worker-1  | 
worker-1  | Checking for file conflicts: [..done]
worker-1  | ( 1/60) Installing: perl-Perl-Tidy-20230912.0.0-lp155.2.1.noarch [..done]
worker-1  | ( 2/60) Installing: lsof-4.91-1.11.x86_64 [..done]
worker-1  | ( 3/60) Installing: perl-Archive-Zip-1.60-3.3.1.noarch [..done]
worker-1  | ( 4/60) Installing: perl-B-Keywords-1.24-bp155.1.5.noarch [..done]
worker-1  | ( 5/60) Installing: perl-Class-Inspector-1.32-1.22.noarch [..done]
worker-1  | ( 6/60) Installing: perl-Class-Singleton-1.5-1.24.noarch [..done]
worker-1  | ( 7/60) Installing: perl-Class-Tiny-1.006-1.24.noarch [..done]
worker-1  | ( 8/60) Installing: perl-Clone-0.46-bp155.1.5.x86_64 [..done]
worker-1  | ( 9/60) Installing: perl-Dist-CheckConflicts-0.11-1.23.noarch [..done]
worker-1  | (10/60) Installing: perl-Exception-Base-0.2501-bp155.2.8.noarch [..done]
worker-1  | (11/60) Installing: perl-File-Path-2.150000-1.22.noarch [..done]
worker-1  | (12/60) Installing: perl-IO-String-1.08-152.24.x86_64 [..done]
worker-1  | (13/60) Installing: perl-JSON-2.97001-1.18.noarch [..done]
worker-1  | (14/60) Installing: perl-Lingua-EN-Inflect-1.905-bp155.2.12.noarch [..done]
worker-1  | (15/60) Installing: perl-List-SomeUtils-XS-0.56-1.17.x86_64 [..done]
worker-1  | (16/60) Installing: perl-Module-Build-0.422400-1.24.noarch [..done]
worker-1  | (17/60) Installing: perl-Module-Pluggable-5.2-1.24.noarch [..done]
worker-1  | (18/60) Installing: perl-NetAddr-IP-4.079-1.27.x86_64 [..done]
worker-1  | (19/60) Installing: perl-Package-Stash-XS-0.28-1.26.x86_64 [..done]
worker-1  | (20/60) Installing: perl-Readonly-2.05-1.22.noarch [..done]
worker-1  | (21/60) Installing: perl-SemVer-0.10.0-bp155.1.4.noarch [..done]
worker-1  | (22/60) Installing: perl-String-Format-1.18-bp155.2.12.noarch [..done]
worker-1  | (23/60) Installing: perl-Sub-Identify-0.14-1.26.x86_64 [..done]
worker-1  | (24/60) Installing: perl-Symbol-Util-0.0203-bp155.2.12.noarch [..done]
worker-1  | (25/60) Installing: perl-Task-Weaken-1.06-2.13.noarch [..done]
worker-1  | (26/60) Installing: perl-Test-Fatal-0.014-1.23.noarch [..done]
worker-1  | (27/60) Installing: perl-Test-LongString-0.17-bp155.2.12.noarch [..done]
worker-1  | (28/60) Installing: perl-Tie-IxHash-1.23-1.23.noarch [..done]
worker-1  | (29/60) Installing: perl-Variable-Magic-0.62-1.19.x86_64 [..done]
worker-1  | (30/60) Installing: perl-XML-SAX-Expat-0.51-1.22.noarch [..done]
worker-1  | (31/60) Installing: perl-XML-Writer-0.625-1.24.noarch [..done]
worker-1  | (32/60) Installing: perl-Eval-Closure-0.14-1.22.noarch [..done]
worker-1  | (33/60) Installing: perl-File-ShareDir-1.104-1.22.noarch [..done]
worker-1  | (34/60) Installing: perl-Path-Tiny-0.104-1.22.noarch [..done]
worker-1  | (35/60) Installing: perl-Pod-MinimumVersion-50-bp155.3.10.noarch [..done]
worker-1  | (36/60) Installing: perl-List-SomeUtils-0.56-1.20.noarch [..done]
worker-1  | (37/60) Installing: perl-Code-DRY-0.10-bp155.2.10.x86_64 [..done]
worker-1  | (38/60) Installing: perl-Package-Stash-0.37-1.23.noarch [..done]
worker-1  | (39/60) Installing: perl-constant-boolean-0.02-bp155.2.8.noarch [..done]
worker-1  | (40/60) Installing: perl-PPI-1.276-bp155.1.4.noarch [..done]
worker-1  | (41/60) Installing: perl-B-Hooks-EndOfScope-0.21-1.22.noarch [..done]
worker-1  | (42/60) Installing: perl-XML-Simple-2.24-1.22.noarch [..done]
worker-1  | (43/60) Installing: perl-Specio-0.42-1.17.noarch [..done]
worker-1  | (44/60) Installing: perl-Params-ValidationCompiler-0.26-1.17.noarch [..done]
worker-1  | (45/60) Installing: perl-Pod-Spell-1.25-bp155.1.5.noarch [..done]
worker-1  | (46/60) Installing: perl-Test-Assert-0.0504-bp155.2.8.noarch [..done]
worker-1  | (47/60) Installing: perl-PPIx-Utilities-1.001000-bp155.3.9.noarch [..done]
worker-1  | (48/60) Installing: perl-PPIx-Regexp-0.086-bp155.1.5.noarch [..done]
worker-1  | (49/60) Installing: perl-PPIx-QuoteLike-0.023-bp155.1.5.noarch [..done]
worker-1  | (50/60) Installing: perl-namespace-clean-0.27-1.22.noarch [..done]
worker-1  | (51/60) Installing: perl-Perl-Critic-1.148-bp155.1.4.noarch [..done]
worker-1  | (52/60) Installing: perl-namespace-autoclean-0.28-1.22.noarch [..done]
worker-1  | (53/60) Installing: perl-Perl-Critic-Pulp-99-bp155.2.8.x86_64 [..done]
worker-1  | (54/60) Installing: perl-Perl-Critic-Policy-Variables-ProhibitLoopOnHash-0.008-bp155.2.8.noarch [..done]
worker-1  | (55/60) Installing: perl-DateTime-Locale-1.170000-1.17.noarch [.....done]
worker-1  | (56/60) Installing: perl-Perl-Critic-Community-1.0.3-bp155.1.4.noarch [..done]
worker-1  | (57/60) Installing: perl-Selenium-Remote-Driver-1.49-bp155.2.3.1.x86_64 [..done]
worker-1  | (58/60) Installing: perl-DateTime-TimeZone-2.60-150000.3.3.1.noarch [..done]
worker-1  | (59/60) Installing: perl-DateTime-1.45-1.16.x86_64 [..done]
worker-1  | (60/60) Installing: os-autoinst-distri-opensuse-deps-1.1702280286.cc5583fe-lp155.13341.1.noarch [..done]
worker-1  | /run_openqa_worker.sh: line 34:     9 Terminated              qemu-system-x86_64 -S
worker-1  | [info] [pid:287] worker 1:
worker-1  |  - config file:                      /etc/openqa/workers.ini
worker-1  |  - name used to register:            de1427a126e7
worker-1  |  - worker address (WORKER_HOSTNAME): worker
worker-1  |  - isotovideo version:               40
worker-1  |  - websocket API version:            1
worker-1  |  - web UI hosts:                     http://172.18.0.9
worker-1  |  - class:                            demohw
worker-1  |  - no cleanup:                       no
worker-1  |  - pool directory:                   /var/lib/openqa/pool/1
worker-1  | [info] [pid:287] Project dir for host http://172.18.0.9 is /var/lib/openqa/share
worker-1  | [info] [pid:287] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:287] Establishing ws connection via ws://172.18.0.9/api/v1/ws/15
worker-1  | [info] [pid:287] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 15
worker-1  | [warn] [pid:287] Websocket connection to http://172.18.0.9/api/v1/ws/15 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:287] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:287] Establishing ws connection via ws://172.18.0.9/api/v1/ws/15
worker-1  | [info] [pid:287] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 15
worker-1  | [warn] [pid:287] Websocket connection to http://172.18.0.9/api/v1/ws/15 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:287] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:287] Establishing ws connection via ws://172.18.0.9/api/v1/ws/15
worker-1  | [info] [pid:287] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 15
worker-1  | [warn] [pid:287] Websocket connection to http://172.18.0.9/api/v1/ws/15 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:287] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:287] Establishing ws connection via ws://172.18.0.9/api/v1/ws/15
worker-1  | [info] [pid:287] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 15
worker-1  | [debug] [pid:287] Accepting job 59 from http://172.18.0.9.
worker-1  | [debug] [pid:287] Setting job 59 from http://172.18.0.9 up
worker-1  | [debug] [pid:287] Preparing Mojo::IOLoop::ReadWriteProcess::Session
worker-1  | [info] [pid:287] +++ setup notes +++
worker-1  | [info] [pid:287] Running on de1427a126e7:1 (Linux 6.1.65 #1-NixOS SMP PREEMPT_DYNAMIC Sun Dec  3 06:32:13 UTC 2023 x86_64)
worker-1  | [debug] [pid:287] Job settings:
worker-1  |     ARCH=x86_64
worker-1  |     BACKEND=qemu
worker-1  |     DESKTOP=xfce
worker-1  |     DISTRI=demohw
worker-1  |     FLAVOR=generalhw
worker-1  |     HDDSIZEGB=100
worker-1  |     INSTALL=1
worker-1  |     JOBTOKEN=qbgKDWmM_1PE9jOP
worker-1  |     LOG_LEVEL=debug
worker-1  |     MACHINE=demohw
worker-1  |     NAME=00000059-demohw-v1-generalhw-x86_64-demohw@demohw
worker-1  |     NICTYPE=user
worker-1  |     OPENQA_HOSTNAME=172.18.0.9
worker-1  |     OPENQA_URL=http://172.18.0.9
worker-1  |     PRJDIR=/var/lib/openqa/share
worker-1  |     QEMUPORT=20012
worker-1  |     RETRY_DELAY=5
worker-1  |     RETRY_DELAY_IF_WEBUI_BUSY=60
worker-1  |     SSH_XTERM_WAIT_SUT_ALIVE_TIMEOUT=60
worker-1  |     TEST=demohw
worker-1  |     TEST_SUITE_NAME=demohw
worker-1  |     VERSION=v1
worker-1  |     VIDEOMODE=vnc
worker-1  |     VNC=91
worker-1  |     VNC_TYPING_LIMIT=7
worker-1  |     WORKER_CLASS=demohw
worker-1  |     WORKER_HOSTNAME=worker
worker-1  |     WORKER_ID=15
worker-1  |     WORKER_INSTANCE=1
worker-1  |     YAML_SCHEDULE=schedule/demohw.yaml
worker-1  | [debug] [pid:287] Symlinked from "/var/lib/openqa/share/tests/demohw" to "/var/lib/openqa/pool/1/demohw"
worker-1  | [info] [pid:287] Preparing cgroup to start isotovideo
worker-1  | [warn] [pid:287] Disabling cgroup usage because cgroup creation failed: mkdir /sys/fs/cgroup/systemd: Permission denied at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/File.pm line 85.
worker-1  | [info] [pid:287] You can define a custom slice with OPENQA_CGROUP_SLICE or indicating the base mount with MOJO_CGROUP_FS.
worker-1  | [info] [pid:287] Starting isotovideo container
worker-1  | [debug] [pid:287] Registered process:290
worker-1  | [info] [pid:287] isotovideo has been started (PID: 290)
worker-1  | [info] [pid:290] 290: WORKING 59
worker-1  | [debug] [pid:290] +++ worker notes +++
worker-1  | [debug] [pid:287] Running job 59 from http://172.18.0.9: 00000059-demohw-v1-generalhw-x86_64-demohw@demohw.
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (no current module)
worker-1  | [debug] [pid:287] Starting livelog
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (no current module)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (no current module)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (no current module)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (at boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (up to boot)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (no current module)
worker-1  | [info] [pid:287] Isotovideo exit status: 0
worker-1  | [debug] [pid:287] Stopping job 59 from http://172.18.0.9: 00000059-demohw-v1-generalhw-x86_64-demohw@demohw - reason: done
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [info] [pid:287] +++ worker notes +++
worker-1  | [info] [pid:287] End time: 2023-12-11 12:40:20
worker-1  | [info] [pid:287] Result: done
worker-1  | [info] [pid:413] Uploading video.ogv
worker-1  | [debug] [pid:413] Uploading artefact video.ogv
worker-1  | [info] [pid:413] Uploading autoinst-log.txt
worker-1  | [debug] [pid:413] Uploading artefact autoinst-log.txt
worker-1  | [info] [pid:413] Uploading worker-log.txt
worker-1  | [debug] [pid:413] Uploading artefact worker-log.txt
worker-1  | [info] [pid:413] Uploading vars.json
worker-1  | [debug] [pid:413] Uploading artefact vars.json
worker-1  | [info] [pid:413] Uploading serial0.txt
worker-1  | [debug] [pid:413] Uploading artefact serial0.txt
worker-1  | [info] [pid:413] Uploading video_time.vtt
worker-1  | [debug] [pid:413] Uploading artefact video_time.vtt
worker-1  | [info] [pid:413] Uploading serial_terminal.txt
worker-1  | [debug] [pid:413] Uploading artefact serial_terminal.txt
worker-1  | [info] [pid:413] Uploading serial_terminal_user.txt
worker-1  | [debug] [pid:413] Uploading artefact serial_terminal_user.txt
worker-1  | [debug] [pid:287] Considering job 59 as regularly done
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/status
worker-1  | [debug] [pid:287] Upload concluded (no current module)
worker-1  | [debug] [pid:287] REST-API call: POST http://172.18.0.9/api/v1/jobs/59/set_done?worker_id=15
worker-1  | [debug] [pid:287] Job 59 from http://172.18.0.9 finished - reason: done
worker-1  | [debug] [pid:287] Cleaning up for next job
worker-1  | [warn] [pid:287] Websocket connection to http://172.18.0.9/api/v1/ws/15 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:287] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:287] Establishing ws connection via ws://172.18.0.9/api/v1/ws/15
worker-1  | [info] [pid:287] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 15
worker-1  | [warn] [pid:287] Websocket connection to http://172.18.0.9/api/v1/ws/15 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:287] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:287] Establishing ws connection via ws://172.18.0.9/api/v1/ws/15
worker-1  | [info] [pid:287] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 15
worker-1  | [warn] [pid:287] Websocket connection to http://172.18.0.9/api/v1/ws/15 finished by remote side with code 1006, no reason - trying again in 10 seconds
worker-1  | [info] [pid:287] Registering with openQA http://172.18.0.9
worker-1  | [info] [pid:287] Establishing ws connection via ws://172.18.0.9/api/v1/ws/15
worker-1  | [info] [pid:287] Registered and connected via websockets with openQA host http://172.18.0.9 and worker ID 15

i change:

-    assert_screen 'bootloader';
+    assert_screen 'bootloaderr', timeout => 100;

to try to reproduce 504, here is the screenshot.

image