[BUG] SCS-compatible IaaS tests fail due to entropy test

mbuechse commented 9 months ago

Current info

I think we are seeing several problems.

wavestack: bug (resource leak) leading to subsequent failure

resource got leaked when Zuul runner was terminated
this run is marked RETRY after 7 minutes without any logs: https://zuul.scs.community/t/SCS/build/7b8978bb4b724c17beed6fd44365e67c/console
if the runner was terminated during the entropy test, this test didn't get to remove its resources
the subsequent run then fails because the resource (a named keypair) already exists: https://zuul.scs.community/t/SCS/build/bb4ac774e07d41478bae31efb59e17d9/console

The first run didn't produce any logs. Now the relevant part of the log of the second run:

Testing standard Entropy ...
Reference: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0101-v1-entropy.md ...
INFO: Flavor 'SCS-16V-64-500s_GNa-14h' missing recommended attributes: hw_rng:allowed=True; additionally, missing optional attributes: hw_rng:rate_bytes, hw_rng:rate_period
CRITICAL: ConflictException('ConflictException: 409')
... returned 0 errors, 1 aborts

Locally, with debug output activated, I see this

DEBUG: Exception info
Traceback (most recent call last):
  File "/home/matthiasbuechse/opensrc/scs/standards/Tests/iaas/entropy/./entropy-check.py", line 404, in main
    with TestEnvironment(conn) as env:
  File "/home/matthiasbuechse/opensrc/scs/standards/Tests/iaas/entropy/./entropy-check.py", line 275, in __enter__
    self.prepare()
  File "/home/matthiasbuechse/opensrc/scs/standards/Tests/iaas/entropy/./entropy-check.py", line 167, in prepare
    self.keypair = self.conn.compute.create_keypair(name=KEYPAIR_NAME)
  File "/home/matthiasbuechse/.local/lib/python3.10/site-packages/openstack/compute/v2/_proxy.py", line 628, in create_keypair
    return self._create(_keypair.Keypair, **attrs)
  File "/home/matthiasbuechse/.local/lib/python3.10/site-packages/openstack/proxy.py", line 594, in _create
    return res.create(self, base_path=base_path)
  File "/home/matthiasbuechse/.local/lib/python3.10/site-packages/openstack/resource.py", line 1522, in create
    self._translate_response(response, **response_kwargs)
  File "/home/matthiasbuechse/.local/lib/python3.10/site-packages/openstack/resource.py", line 1271, in _translate_response
    exceptions.raise_from_response(response, error_message=error_message)
  File "/home/matthiasbuechse/.local/lib/python3.10/site-packages/openstack/exceptions.py", line 263, in raise_from_response
    raise cls(
openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://api.wavestack.de:8774/v2.1/os-keypairs, Key pair 'scs-0101-keypair' already exists.

gx-scs: probably incorrect application credentials in Zuul secrets

This error can be seen in every run on gx-scs:

Testing standard Entropy ...
Reference: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0101-v1-entropy.md ...
INFO: Image 'Ubuntu 18.04' missing recommended attributes: hw_rng_model=virtio
CRITICAL: HttpException('HttpException: 403')
... returned 0 errors, 1 aborts

I can't reproduce this locally when I use the correct application credentials, so I presume that the Zuul secrets contain some application credential that I once created with the wrong permissions. I will update the Zuul secrets.

Initial comment:

see, e.g.,

berendt commented 9 months ago

@mbuechse Can you please add the interesting parts of the logs here. Sometimes we loose old logs.

mbuechse commented 9 months ago

@berendt Yes, I will do that. So far I'm still trying to figure out the interesting parts though ;)

SovereignCloudStack / standards