search

containers / qm

QM is a containerized environment for running Functional Safety qm (Quality Management) software
https://github.com/containers/qm
GNU General Public License v2.0
20 stars 20 forks source link

add tier-0 test case #486

Closed pengshanyu closed 3 days ago

pengshanyu commented 2 weeks ago

We have several test cases for FFI, but we are missing some basic/simple sanity tests for qm, if it is installed properly and/or if environment started successfully. This is a draft version. Since the naming of the test files and test methods are simple, I hope to use this version as a starting point for improving the sanity test. We would like to name this test plan tier-0, but need to rename the current tier-0 first, so I temporarily named it tier-2

resolve https://github.com/containers/qm/issues/342

pengshanyu commented 2 weeks ago

Please add the tests to .packit.yaml to be visible on github tests

Should I do it now in this PR or should I wait until this PR is merged before doing it?

Yarboa commented 2 weeks ago

Please add the tests to .packit.yaml to be visible on github tests

Should I do it now in this PR or should I wait until this PR is merged before doing it?

MHO, if this new packit test introduced, why not let it run, this way we confirm that test merged are working

Now, i see you calked the tests tier-2 I think that current tier-0 tests should be renamed to a different name, like multi-bluchi-agents, that is the real purpose of it, maybe we will remove it from tests later on.

pengshanyu commented 1 week ago

MHO, if this new packit test introduced, why not let it run, this way we confirm that test merged are working

Now, i see you calked the tests tier-2 I think that current tier-0 tests should be renamed to a different name, like multi-bluchi-agents, that is the real purpose of it, maybe we will remove it from tests later on.

Hi @Yarboa, I renamed current tier-0 tests to multi-bluechi-agents, but there are also tier-0 tests in qm-connectivity and qm-unit-files, how should they be renamed or do you want them to be tier-0 along with the sanity tests?

Yarboa commented 1 week ago

MHO, if this new packit test introduced, why not let it run, this way we confirm that test merged are working Now, i see you calked the tests tier-2 I think that current tier-0 tests should be renamed to a different name, like multi-bluchi-agents, that is the real purpose of it, maybe we will remove it from tests later on.

Hi @Yarboa, I renamed current tier-0 tests to multi-bluechi-agents, but there are also tier-0 tests in qm-connectivity and qm-unit-files, how should they be renamed or do you want them to be tier-0 along with the sanity tests?

qm-connectivity exist in new tier-0 but it could run in the test, since it checks for multi connectivity

related to qm-unit-files, i think it is already running in ffi

Please update the test of tier-0 similar to ffi tests Please use this one https://github.com/containers/qm/blob/main/.packit.yaml#L75

Please as you mentioned rename tier-0 tests as mentioned and please enable old tier-0 for manual trigger only as per this https://packit.dev/posts/manual-triggering#manual-only-triggering-of-jobs

pengshanyu commented 1 week ago

@Yarboa thank you for the reply. qm-connectivity run with run-test-e2e the new tier-0 test run with set-ffi-env-e2e they are different test plan, I'm not sure how to run them both as tier-0 at one tmt run

Yarboa commented 1 week ago

/packit test --identifier qm-tier-0

Yarboa commented 6 days ago

LGTM, @pengshanyu well done Please squash all comments, to have clean commits history before merge

pengshanyu commented 5 days ago

@Yarboa, thank you so much for the review. I added the Polarion-id.

Yarboa commented 4 days ago

/packit test --identifier qm-tier-0

pengshanyu commented 4 days ago

Hi @Yarboa , after this PR merged, we should also exclude this sanity test cases in the p-l-p test plan, right?

Yarboa commented 3 days ago

/packit test --identifier e2e-multi-bluechi-agents

Yarboa commented 3 days ago

/packit test --identifier e2e-multi-bluechi-agents

seems that tests is failing need to check maybe we have an issue with default values https://artifacts.dev.testing-farm.io/ea25bbf3-3563-4e9f-a7f4-5058fb393c4e/work-multi-bluechi-agentsg24c48ar/log.txt

pengshanyu commented 3 days ago

/packit test --identifier e2e-multi-bluechi-agents

pengshanyu commented 3 days ago

/packit test --identifier e2e-multi-bluechi-agents