:sparkles: Add support for multiple clients in unit tests

[gabrieldemarmiesse]

To select all podman tests: pytest -m podman, same for docker. To change the podman executable pytest --podman-exe=podman4.2. Same for docker. If podman is not present on the system, all podman tests are SKIPPED. Same with docker.

There are no global variables here except the params but I think it's reasonnable as they're only constant strings.

@LewisGaul I know we won't agree with the fact that there is no default parameters in the unit tests, but can you still take a look? Most of the code in this PR comes from your previous PRs on the subject.

The result looks like this:

$ python -m pytest -vv ./tests/python_on_whales/components/test_image.py y
======================================================================= test session starts =======================================================================
platform linux -- Python 3.10.13, pytest-7.4.3, pluggy-1.0.0 -- /opt/conda/bin/python
cachedir: .pytest_cache
rootdir: /projects/open_source/python-on-whales
configfile: pyproject.toml
plugins: mock-3.12.0
collected 42 items

tests/python_on_whales/components/test_image.py::test_load_json[json_file0] PASSED                                                                          [  2%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file1] PASSED                                                                          [  4%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file2] PASSED                                                                          [  7%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file3] PASSED                                                                          [  9%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file4] PASSED                                                                          [ 11%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file5] PASSED                                                                          [ 14%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file6] PASSED                                                                          [ 16%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file7] PASSED                                                                          [ 19%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file8] PASSED                                                                          [ 21%]
tests/python_on_whales/components/test_image.py::test_load_json[json_file9] PASSED                                                                          [ 23%]
tests/python_on_whales/components/test_image.py::test_image_repr PASSED                                                                                     [ 26%]
tests/python_on_whales/components/test_image.py::test_image_remove[docker client] PASSED                                                                    [ 28%]
tests/python_on_whales/components/test_image.py::test_image_remove[podman client] SKIPPED (Unable to get version with command 'podman version'. Reason:
executable not found)                                                                                                                                       [ 30%]
tests/python_on_whales/components/test_image.py::test_image_save_load[docker client] PASSED                                                                 [ 33%]
tests/python_on_whales/components/test_image.py::test_save_iterator_bytes PASSED                                                                            [ 35%]
tests/python_on_whales/components/test_image.py::test_filter_when_listing PASSED                                                                            [ 38%]
tests/python_on_whales/components/test_image.py::test_filter_when_listing_old_signature PASSED                                                              [ 40%]
tests/python_on_whales/components/test_image.py::test_use_first_argument_to_filter PASSED                                                                   [ 42%]
tests/python_on_whales/components/test_image.py::test_save_iterator_bytes_and_load PASSED                                                                   [ 45%]
tests/python_on_whales/components/test_image.py::test_save_iterator_bytes_and_load_from_iterator PASSED                                                     [ 47%]
tests/python_on_whales/components/test_image.py::test_save_iterator_bytes_and_load_from_iterator_list_of_images PASSED                                      [ 50%]
tests/python_on_whales/components/test_image.py::test_image_list PASSED                                                                                     [ 52%]
tests/python_on_whales/components/test_image.py::test_image_bulk_reload PASSED                                                                              [ 54%]
tests/python_on_whales/components/test_image.py::test_image_list_tags PASSED                                                                                [ 57%]
tests/python_on_whales/components/test_image.py::test_pull_not_quiet PASSED                                                                                 [ 59%]
tests/python_on_whales/components/test_image.py::test_pull_not_quiet_multiple_images PASSED                                                                 [ 61%]
tests/python_on_whales/components/test_image.py::test_pull_not_quiet_multiple_images_break PASSED                                                           [ 64%]
tests/python_on_whales/components/test_image.py::test_copy_from_and_to PASSED                                                                               [ 66%]
tests/python_on_whales/components/test_image.py::test_copy_from_and_to_directory PASSED                                                                     [ 69%]
tests/python_on_whales/components/test_image.py::test_prune PASSED                                                                                          [ 71%]
tests/python_on_whales/components/test_image.py::test_remove_nothing PASSED                                                                                 [ 73%]
tests/python_on_whales/components/test_image.py::test_no_such_image_with_multiple_functions[inspect] PASSED                                                 [ 76%]
tests/python_on_whales/components/test_image.py::test_no_such_image_with_multiple_functions[remove] PASSED                                                  [ 78%]
tests/python_on_whales/components/test_image.py::test_no_such_image_with_multiple_functions[push] PASSED                                                    [ 80%]
tests/python_on_whales/components/test_image.py::test_no_such_image_save PASSED                                                                             [ 83%]
tests/python_on_whales/components/test_image.py::test_no_such_image_save_generator PASSED                                                                   [ 85%]
tests/python_on_whales/components/test_image.py::test_no_such_image_tag PASSED                                                                              [ 88%]
tests/python_on_whales/components/test_image.py::test_exists PASSED                                                                                         [ 90%]
tests/python_on_whales/components/test_image.py::test_copy_from_default_pull PASSED                                                                         [ 92%]
tests/python_on_whales/components/test_image.py::test_copy_from_pull PASSED                                                                                 [ 95%]
tests/python_on_whales/components/test_image.py::test_copy_to_default_pull PASSED                                                                           [ 97%]
tests/python_on_whales/components/test_image.py::test_copy_to_pull PASSED                                                                                   [100%]

================================================================= 41 passed, 1 skipped in 52.09s ==================================================================

[LewisGaul]

Before I take a proper look, did you take a look at https://github.com/gabrieldemarmiesse/python-on-whales/pull/522? I think this would be a nice precursor PR, with the focus on moving to using the ctr_client fixture without any distraction of parameterisation.

[gabrieldemarmiesse]

Before I take a proper look, did you take a look at https://github.com/gabrieldemarmiesse/python-on-whales/pull/522? I think this would be a nice precursor PR, with the focus on moving to using the ctr_client fixture without any distraction of parameterisation.

Indeed I did take a look at it, that's where I took most of this code from :)

[LewisGaul]

Overall this approach seems fine, I'm not particularly convinced it's better than what I worked on (and the case of tests only supported by one runtime needs thinking about), but at a certain point we should just settle on something - better to have something available that works!

[gabrieldemarmiesse]

I think we should try to have pytestmark = @pytest.mark.parametrize("ctr_client", [docker_client, podman_client], indirect=True) in as many files as possible. Not having this means there are some weird test present that don't work with all clients. I'm even open to divide them into multiple files so that it's easy to track. What do you think?

[LewisGaul]

I think we should try to have pytestmark = @pytest.mark.parametrize("ctr_client", [docker_client, podman_client], indirect=True) in as many files as possible. Not having this means there are some weird test present that don't work with all clients. I'm even open to divide them into multiple files so that it's easy to track. What do you think?

I'm in favour of marking parameterisation at a per-file level, it feels like one step towards my preference for the default of parameterising and removing some of the verbosity/explicitness. In fact this seems like a nice compromise given it will often be whole subcommands that are only supported by one of docker/podman (buildx, swarm, compose, pod, ...).

As I was stating before, one of my concerns is that someone might add a new test file and miss that they should be parameterising the new tests, however I think if they were to try using ctr_client it would complain about request.param not being available if there was no parameterisation?

Not having this means there are some weird test present that don't work with all clients.

Could you elaborate on this?

I know a bunch of tests currently don't work with podman for reasons such as a difference in convention in image naming, as well as some subcommands not being supported by podman. At some point I could look into getting more tests working with podman, e.g. the image tests could probably be made more generic (mark as xfail for now).

[gabrieldemarmiesse]

I think if they were to try using ctr_client it would complain about request.param not being available if there was no parameterisation?

Indeed I don't see how it would work.

Could you elaborate on this?

If one single test can't handle all clients, then it means that we can't use pytestmark, thus making it easier to spot missing features.

[gabrieldemarmiesse]

@LewisGaul I added the xfail as requested. I believe we can merge this PRs unless you have other recommendations. When the whole test_image.py has been parametrize, I expect that is should be possible to use pytestmark. And later, it should be possible to add if self.client_config.client_type == "podman": ... in the right places in the codebase to fix the xfails of podman. Better support for podman coming right up :)

[gabrieldemarmiesse]

My pleasure!