search

OSInside / kiwi-functional-tests

KIWI openQA functional tests based on the integration tests in OBS
GNU General Public License v2.0
3 stars 4 forks source link

Test failures on openqa.opensuse.org #9

Open dcermak opened 3 years ago

dcermak commented 3 years ago

All items on that list (except the busted LUKS image) are vmdk images, which leads me to believe that something is wrong with vmdk on o3, whereas it works on my box. @okurz @foursixnine @asmorodskyi any ideas?

okurz commented 2 years ago

Following https://openqa.opensuse.org/tests/2044724#settings I can see the setting for HDD_1 being kiwi-test-image-disk-simple.x86_64-1.42.1-Build11.19.vmdk but I can not find that asset over https://openqa.opensuse.org/admin/assets so it was either never there or (likely) was pruned already. https://openqa.opensuse.org/admin/job_templates/85 so far is configured for only the default 5GB and 4.8GB are used so likely any other image does not fit meaning other images will need to be deleted. Also the history https://openqa.opensuse.org/tests/2044724#next_previous shows that jobs did work. So I have now assigned 30GB to the group so I suggest you try again.

dcermak commented 2 years ago

I have retriggered all the tests yesterday but sadly all the tests involving the vmdk images are still failing. Here is the full list of tests:

It really looks like openQA is not downloading these images properly. When I look at https://openqa.opensuse.org/admin/assets then that page only lists the vmdk images for the Leap and none for Tumbleweed (the vmdk images are also missing under the "Logs & Assets" tab there).

@okurz Do you know if the openQA logs could be in some way helpful here?

asmorodskyi commented 2 years ago

if you open os-autoinst.log you will see such lines in very beginning : 

[2021-11-25T01:11:38.102292+01:00] [info] +++ setup notes +++
[2021-11-25T01:11:38.102627+01:00] [info] Running on openqaworker4:7 (Linux 5.3.18-59.34-default #1 SMP Thu Nov 11 12:18:45 UTC 2021 (a2a53aa) x86_64)
[2021-11-25T01:11:38.108770+01:00] [debug] Found HDD_1, caching kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk
[2021-11-25T01:11:38.112266+01:00] [info] Downloading kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk, request #480464 sent to Cache Service
[2021-11-25T01:12:03.561175+01:00] [info] Download of kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk processed:
[info] [#480464]
Cache size of "/var/lib/openqa/cache" is 48 GiB, with limit 50 GiB
[info] [#480464]
Downloading "kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" from "http://openqa1-opensuse/tests/2055611/asset/hdd/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk"
[info] [#480464]
Size of "/var/lib/openqa/cache/openqa1-opensuse/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" is 954 MiB, with ETag ""3b9d0000-5d18c5fe40064""
[info] [#480464]
Download of "/var/lib/openqa/cache/openqa1-opensuse/kiwi-test-image-lvm.x86_64-1.42.1-Build12.28.vmdk" successful, new cache size is 49 GiB

so asset is downloaded BUT it is vmdk not qcow and here I pass mike to @okurz because I have no idea if openQA support VMWare images ( spoiler : most probably not )

dcermak commented 2 years ago

I have quickly taken a peek at /var/log/factory/hdd from ariel and the assets are not there:

dancermak@ariel:~> ll /var/lib/openqa/factory/hdd/|grep vmdk
-rw-r--r-- 1 geekotest nogroup   816513024 Nov 24 17:43 kiwi-test-image-disk-simple.x86_64-1.15.3-Build11.10.vmdk
-rw-r--r-- 1 geekotest nogroup   844431360 Nov 24 17:43 kiwi-test-image-lvm.x86_64-1.15.3-Build11.10.vmdk
-rw-r--r-- 1 geekotest nogroup   249823232 Nov 24 17:43 kiwi-test-image-overlayroot.x86_64-1.15.3-Build11.10.vmdk

But some assets are there and the corresponding tests still failed… (e.g. https://openqa.opensuse.org/tests/2055673 uses kiwi-test-image-overlayroot.x86_64-1.15.3-Build11.10.vmdk)

okurz commented 2 years ago

Right now the cases look quite curious. https://openqa.opensuse.org/tests/2055674#step/boot/3 looks like the asset is correctly downloaded and found but looks like some qemu devices can not be correctly read. https://openqa.opensuse.org/tests/2055607/#step/boot/1 says that the disk image can not be read at all. Also https://openqa.opensuse.org/tests/2055607/#downloads does not list it although it was downloaded.

Regarding the "fatal: Invalid revision range" in https://openqa.opensuse.org/tests/2055607/#investigation I reported https://progress.opensuse.org/issues/102942 but that should not concern us for now.

For the other cases, if you can clearly reproduce a case that asssets are downloaded but then tests fail and the asset is gone then please report a ticket for openQA so that we can further look into this.

asmorodskyi commented 2 years ago

Right now the cases look quite curious. https://openqa.opensuse.org/tests/2055674#step/boot/3 looks like the asset is correctly downloaded and found but looks like some qemu devices can not be correctly read. https://openqa.opensuse.org/tests/2055607/#step/boot/1 says that the disk image can not be read at all. Also https://openqa.opensuse.org/tests/2055607/#downloads does not list it although it was downloaded.

Regarding the "fatal: Invalid revision range" in https://openqa.opensuse.org/tests/2055607/#investigation I reported https://progress.opensuse.org/issues/102942 but that should not concern us for now.

For the other cases, if you can clearly reproduce a case that asssets are downloaded but then tests fail and the asset is gone then please report a ticket for openQA so that we can further look into this.

@okurz something to take into consideration . @dcermak stating that same thing is working on his local instance . I did some comparison of log files from his local instance and one thing which I noticed that he has os-autoinst version=23 which saying about potential regression

dcermak commented 2 years ago

Reported as https://progress.opensuse.org/issues/103203

okurz commented 2 years ago

@asmorodskyi good hint. @dcermak thanks for the ticket. Asked for more information (if possible) in the ticket.

dcermak commented 2 years ago

The vmdk disk failures have been fixed via https://github.com/os-autoinst/os-autoinst/pull/1907.