Fix failing eden tests - Githubissues

milan-zededa commented 2 months ago

Multiple patches in this PR:

Increased the default EVE version to the latest 12.4
Removed workflow dependency on Smoke test suite. With reusable workflows this does not work properly in the EVE repository. The other test suites are triggered regardless of the outcome of Smoke tests. The only effect this dependency has, is that other workflows are delayed from starting until Smoke tests finish, increasing the total execution time of all tests.
Moved tests that depend on virtualization (apps must be deployed inside VMs) to a separate test suite. The primary reason is that the HW-assisted nested virtualization is performing poorly on the buildjet runners and we are getting many failures (not related to EVE code). For every other test it is therefore better to run EVE with acceleration disabled. A secondary reason is that most of these tests are from the networking test suite, which is already quite long and takes well over an hour to execute. It therefore makes sense to move some tests to a different test suite. Please note that when the Virtualization test suite is run locally, all tests are passing. However, on buildjet runners they are expected to fail until we resolve the issue with the nested virtualization.
Fixed 'eden pod modify' (wrong timeout + app state to wait for had to be modified to reflect recent EVE changes)
Fixed ctrl_cert_change test (improper use of -check-new)
App 'nodered' needs more time to get deployed
Test publish_location is now skipped and likely will be removed. This is because after the recent changes in the EVE/wwan microservice, injecting a fake GPS location data is no longer possible.
Modified EVE-upgrade tests to use newer base EVE version (> 12.0). This is because the policy_version of fscrypt was bumped from 1 to 2 and this change is not backward compatible. Meaning we cannot downgrade from 12.X to 11.Y or older.

Please note that we are still getting some occasional kernel crashes on buildjet runners (even with accel=false), so even stable tests may fail once in a while.

uncleDecart commented 2 months ago

Forgot the sticker :D :shipit:

And also I see that LPC LOC tests are failing, right?

Edit: and some of the Smoke tests are still failing and they fail on rebooting. I wonder if it has to do with Buildet runners as well

milan-zededa commented 2 months ago

Forgot the sticker :D :shipit:

And also I see that LPC LOC tests are failing, right?

Edit: and some of the Smoke tests are still failing and they fail on rebooting. I wonder if it has to do with Buildet runners as well

Yes, all of those were due to kernel crashes that we are occasionally getting.

milan-zededa commented 2 months ago

Runners are without Internet connectivity, will try again tomorrow...

milan-zededa commented 2 months ago

Kernel crashes are quite often. I wonder if the "noisy neighbor" (quoting the guy from Buildjet) is actually us and these problems are more frequent when we run many eden workflows at the same time (likely scheduled on the same host(s)).

uncleDecart commented 2 months ago

Okay, anyways I'm merging this PR and telemetry one, let's try to get as much info as possible

uncleDecart commented 2 months ago

Let's try to run it on local runner, get statistics from it and compare them

lf-edge / eden

Fix failing eden tests #986