Skipping selected tests

[adhawkins]

I'm trying to upgrade the Alpine package to the latest version, and add in the ability to run the tests during the package build.

I'm finding that a number of the tests are failing due to the way the Alpine CI process works (doing the build inside a Docker container, hence meaning the borg mount tests will fail as FUSE isn't available).

You may be able to see the output of a build here: https://gitlab.alpinelinux.org/adhawkins/aports/-/jobs/473511

Is there any way to skip selected tests with the test system you're using?

[KenKundert]

How is it that you can install Borg in the container but not FUSE?

I don't know of a way to skip tests without modifying the test code itself. It might be possible add a list of dependencies to each test, and skip the tests for which the a dependency is not satisfied.

[adhawkins]

To get FUSE working in the container, you have to pass the host's fuse device into the container. I'll check with the people at Alpine to see if there's any chance this could be done in their CI system, but I suspect the answer will be 'No'.

Will look at patching the tests to remove those that rely on FUSE.

Thanks for the response.

[adhawkins]

Ok, I've patched test-cases.nt to remove all the 'mount' and 'unmount' tests. However, it seems like they're still being run.

I can see that the changes from the patch have been applied to test-cases.nt. Is there something else I need to do to remove these tests?

[adhawkins]

Ah, apologies, think I've sorted it. Didn't realise that the 'compare' tests would also indirectly invoke mount.

With those tests removed all completes now.

[KenKundert]

I want to thank you for supporting Emborg on Alpine. Please let me know if you would like me to add the concept of dependencies into the tests so you do not have to manually hack up the test cases each time there is a release.

[adhawkins]

If there were a way of skipping any tests (or perhaps categories of tests) then that would be useful.

pytest does have the concept of skipping tests:

pytest -k 'not test_manual'

If some mechanism like this could be used that would certainly help.

[KenKundert]

I have updated the github repository with a version that skips the tests that require Fuse if you define the following environment variable in your shell:

export MISSING_DEPENDENCIES=fuse

Alternatively, you can put this in the tox.ini file as described in tests/README. Let me know if this satisfies your needs.

[adhawkins]

That's great, we're very nearly there.

I forgot to mention that there was one other test that fails:

initialize_configs = None, args = '--config test7 list'
expected = {'exit_status': 2, 'output': '\n        bu error: borg list:\n              Repository [^ ]*/emborg/tests/repositories does not exist\\.'}
expected_type = 'error regex', cmp_dirs = '', remove = '', dependencies = ''

    @parametrize(
        path = f'{tests_dir}/test-cases.nt',
        key = 'emborg with configs',
        schema = emborg_schema
    )
    def test_emborg_with_configs(
        initialize_configs,
        args, expected, expected_type, cmp_dirs, remove, dependencies
    ):
        if skip_test(dependencies):
            return
        with cd(tests_dir):
            tester = EmborgTester(args, expected, expected_type, cmp_dirs, remov  e)
            passes = tester.run()
            if not passes:
                result = tester.get_result()
                expected = tester.get_expected()
>               assert result == expected
E               AssertionError: assert {'exit_status...s not exist.'} == {'exit_  status...not exist\\.'}
E                 Omitting 1 identical items, use -vv to show
E                 Differing items:
E                 {'output': '\n        bu error: borg list:\n            Repository /home/builder/aports/community/emborg/src/emborg-master/tests/repositories   does not exist.'} != {'output': '\n        bu error: borg list:\n            Repository [^ ]*/emborg/tests/repositories does not exist\\.'}
E                 Use -v to get the full diff

tests/test_emborg.py:223: AssertionError
----------------------------- Captured stdout call -----------------------------
bu error: borg list:
    Repository /home/builder/aports/community/emborg/src/emborg-master/tests/rep  ositories does not exist.

=========================== short test summary info ============================
FAILED tests/test_emborg.py::test_emborg_with_configs[76] - AssertionError: a...

I had to patch the regex as follows:

@@ -766,7 +687,7 @@
         args: --config test7 list
         expected:
             > bu error: borg list:
-            >     Repository [^ ]*/emborg/tests/repositories does not exist\.
+            >     Repository [^ ]*/emborg.*/tests/repositories does not exist\.
         expected_type: error regex

     -

In this case, 'emborg-master' is because I was doing a test from master. Usually this would be something like 'emborg-v1.25'. I download a tarball from github using (for example) the following URL:

https://github.com/KenKundert/emborg/archive/v1.25/emborg-v1.25.tar.gz

which would contain a directory 'emborg-1.25'. This seems to trip up the regex that's checking the output of the test where the repo doesn't exist.

[KenKundert]

Oh, that was and independent problem. Thanks. Try it again. Should work this time.

[adhawkins]

Perfect. All tests now passing.

Still waiting for 1.25 to be merged into Alpine. Doesn't seem much point in rushing a release out just for this test code changes. I'll try to make sure that I remember to remove the patch and run the tests with the exclusion when you make the next release.

Thanks for the swift response.

[KenKundert]

I have uploaded the new version to pypi. Thank you!