search

shadow-maint / shadow

Upstream shadow tree
Other
306 stars 237 forks source link

System test framework #1131

Open ikerexxe opened 6 days ago

ikerexxe commented 6 days ago

As discussed at length, this is the implementation of the new system tests framework for shadow. This is a proof of concept that contains the key elements to be able to run basic user (i.e. useradd, usermod) and group (i.e. usermod) tests. If you like the framework the rest of the functionality will be added in the future.

Some useful facts:

In addition, the PR contains a set of tests that make use of the framework to test the functionality. These tests are located under tests/system/tests. They offer a good example of the high-level API and how to model the test cases.

Finally, the system test are run in the CI. This way we make sure that the code can run successfully and avoid any regressions.

Closes: https://github.com/shadow-maint/shadow/issues/835

ikerexxe commented 6 days ago

@hallyn I'd appreciate your feedback on this PR, particularly the tests in the tests/system/tests folder. This PR is quite extensive, so I'd like to focus our initial review on these tests. They showcase the new testing framework and how it interacts with the shadow utilities and other tools. Your input on this core part will be valuable as we move forward with the review.

hallyn commented 6 days ago

Thanks for the ping, I'm putting this on my next_actions list.

ikerexxe commented 6 days ago

For the moment, let's just ignore the CI errors. I'll fix them when I figure out how to do it

alejandro-colomar commented 6 days ago

How should I run the tests in my computer?

Also, are those tests (potentially) destructive? That is, do I need to run them inside a container?

Or does the test suite spin up containers as necessary?

hallyn commented 5 days ago

I'm only partly through this (at patch 5), but it looks good so far, thanks.

ikerexxe commented 3 days ago

How should I run the tests in my computer?

Also, are those tests (potentially) destructive? That is, do I need to run them inside a container?

Or does the test suite spin up containers as necessary?

The instructions from doc/contributions/build_install.md still apply. Running them will create the container, build and install shadow and run the tests inside the container.

You can of course run the test on you own system, but I'd recommend against it since you may break your system with a shadow change. I plan to extend the documentation to explain how the framework works, the API, how to integrate it, etc. But that will come in another PR, once we all agree that the PoC is in a good shape and it can be merged.

I'm only partly through this (at patch 5), but it looks good so far, thanks.

So you've already gone through the first test cases. What do you think about them? Is the API clear to manage shadow and other utilities clear enough?

alejandro-colomar commented 3 days ago

How should I run the tests in my computer? Also, are those tests (potentially) destructive? That is, do I need to run them inside a container? Or does the test suite spin up containers as necessary?

The instructions from doc/contributions/build_install.md still apply. Running them will create the container, build and install shadow and run the tests inside the container.

You can of course run the test on you own system, but I'd recommend against it since you may break your system with a shadow change. I plan to extend the documentation to explain how the framework works, the API, how to integrate it, etc. But that will come in another PR, once we all agree that the PoC is in a good shape and it can be merged.

Just to check if I've understood that:

So, if I run share/container-build.sh, this new test suite will be run?

ikerexxe commented 3 days ago

So, if I run share/container-build.sh, this new test suite will be run?

Yes, the test suite will run within share/container-build.sh

ikerexxe commented 3 days ago

Ok, now the CI is in the state where I wanted it to be last week. Alpine build is failing due to https://gitlab.alpinelinux.org/alpine/aports/-/issues/16633. This is already fixed but it takes some time for those changes to arrive to users.