search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.59k stars 2.4k forks source link

apiv2: test using docker-py integration tests #5386

Closed gartnera closed 3 years ago

gartnera commented 4 years ago

/kind feature

Description

Consider using the docker-py integration tests to measure the progress on the apiv2 development effort. This is what the output of the tests look like with dockerd:

root@podman:~/docker-py/tests/integration# pytest
============================================================================== test session starts ===============================================================================
platform linux -- Python 3.6.9, pytest-4.3.1, py-1.8.1, pluggy-0.13.1
rootdir: /root/docker-py, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
collected 387 items

api_build_test.py .......s....s..........X                                                                                                                                 [  6%]
api_client_test.py .....                                                                                                                                                   [  7%]
api_config_test.py .....                                                                                                                                                   [  8%]
api_container_test.py ....xF..........x.........................................xx.....................                                                                    [ 29%]
api_exec_test.py .....................                                                                                                                                     [ 35%]
api_healthcheck_test.py ....                                                                                                                                               [ 36%]
api_image_test.py ...s........s.....                                                                                                                                       [ 40%]
api_network_test.py ...........................                                                                                                                            [ 47%]
api_plugin_test.py ...........                                                                                                                                             [ 50%]
api_secret_test.py .....                                                                                                                                                   [ 51%]
api_service_test.py ................s..........................................                                                                                            [ 67%]
api_swarm_test.py .ssss.Xs...x.......                                                                                                                                      [ 72%] 
api_volume_test.py .........                                                                                                                                               [ 74%]
client_test.py .....                                                                                                                                                       [ 75%]
context_api_test.py ..                                                                                                                                                     [ 76%] 
errors_test.py .                                                                                                                                                           [ 76%]
models_containers_test.py ...................F.................                                                                                                            [ 86%]
models_images_test.py ...............                                                                                                                                      [ 89%]
models_networks_test.py ....                                                                                                                                               [ 90%]
models_nodes_test.py .                                                                                                                                                     [ 91%]
models_resources_test.py .                                                                                                                                                 [ 91%]
models_services_test.py ..............X.                                                                                                                                   [ 95%]
models_swarm_test.py ..                                                                                                                                                    [ 96%]
models_volumes_test.py ..                                                                                                                                                  [ 96%]
regression_test.py ......                                                                                                                                                  [ 98%]
credentials/store_test.py EEEEEE                                                                                                                                           [ 99%]
credentials/utils_test.py .                                                                                                                                                [100%]

This is what the output of the tests look like with podman system service -t 0:

root@podman:~/docker-py/tests/integration# pytest
============================================================================== test session starts ===============================================================================
platform linux -- Python 3.6.9, pytest-4.3.1, py-1.8.1, pluggy-0.13.1
rootdir: /root/docker-py, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
collected 387 items

api_build_test.py F..FFFFFFFFFFFFFFFFFF..x                                                                                                                                 [  6%]
api_client_test.py .....                                                                                                                                                   [  7%]
api_config_test.py EEEEE                                                                                                                                                   [  8%]
api_container_test.py FFFFxxFEFFFFFFFFFFxxFFEFEFEFFFFFFFFFFEFEFEFEFFFFFFEFFFFFFFFEFFFFFFFEFFFxxxxFFFFFFEFEFEFEFEFEFFFFEFEFEFEFEFF                                          [ 29%]
api_exec_test.py FFFFFFFFEFFFFFFFFFFFFF                                                                                                                                    [ 35%]
api_healthcheck_test.py FEFEFEFE                                                                                                                                           [ 36%]
api_image_test.py FFFFFFFEFF...s..FEFF                                                                                                                                     [ 40%]
api_network_test.py FFFFFFFFFFFFF.FF.FFFFFEFFFFF                                                                                                                           [ 47%]
api_plugin_test.py FFFFFFFFFFF                                                                                                                                             [ 50%]
api_secret_test.py EEEEE                                                                                                                                                   [ 51%]
api_service_test.py EEEEEEEEEEEEEEEEsEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE                                                                                            [ 67%]
api_swarm_test.py FssssFxsFFFxFFFFFFF                                                                                                                                      [ 72%]
api_volume_test.py F.F.FFF.F                                                                                                                                               [ 74%]
client_test.py ..F..                                                                                                                                                       [ 75%]
context_api_test.py ..                                                                                                                                                     [ 76%]
errors_test.py F                                                                                                                                                           [ 76%]
models_containers_test.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF                                                                                                            [ 86%]
models_images_test.py FFFFFFFFFFF.F.F                                                                                                                                      [ 89%]
models_networks_test.py FFFF                                                                                                                                               [ 90%]
models_nodes_test.py F                                                                                                                                                     [ 91%]
models_resources_test.py F                                                                                                                                                 [ 91%]
models_services_test.py EEEEEEEEEEEEEExE                                                                                                                                   [ 95%]
models_swarm_test.py FF                                                                                                                                                    [ 96%]
models_volumes_test.py FF                                                                                                                                                  [ 96%]
regression_test.py .F.FFFE                                                                                                                                                 [ 98%]
credentials/store_test.py EEEEEE                                                                                                                                           [ 99%]
credentials/utils_test.py .                                                                                                                                                [100%]

There's clearly a long way to go on development. This would also help potential users understand what is supported by podman apiv2 and perhaps even help scope future development efforts.

baude commented 4 years ago

take a closer look at the tests ... they use swarm and a bunch of stuff we will never do.

baude commented 4 years ago

they also dont support multiple registries and consider docker.io to be the one and only registries making their short name use difficult to deal with.

gartnera commented 4 years ago

take a closer look at the tests ... they use swarm and a bunch of stuff we will never do.

Yep those aren't features I care about either. But it's a great way to demonstrate to potential adopters what works with apiv2 (in a very granular fashion). I'm not even necessarily saying run it with your normal CI pipelines, maybe just run it for every release.

they also dont support multiple registries and consider docker.io to be the one and only registries making their short name use difficult to deal with.

This doesn't seem to be particularly relevant to this issue (unless I'm misunderstanding).

If you want apiv2 to be compatible with the docker api there's unfortunately a lot of annoying ego (and industry capturing) decisions to deal with and overcome.

baude commented 4 years ago

it matters because several of the tests fail because of it. again, so many of the tests use functions that we won't support I wonder how to make any of it applicable.

github-actions[bot] commented 4 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 4 years ago

@gartnera Did you ever look any further into this? Even if we could get a subset of the tests to work, it would be a big step forward.

gartnera commented 4 years ago

No I haven't done anything more. I reran the tests this morning against master, here's the results:

============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-4.3.1, py-1.8.1, pluggy-0.13.1
rootdir: /root/docker-py, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
timeout: 60.0s
timeout method: signal
timeout func_only: False
collected 387 items

api_build_test.py F...FFFF.FF.FF.FF.FFF..x                               [  6%]
api_client_test.py .....                                                 [  7%]
api_config_test.py EEEEE                                                 [  8%]
api_container_test.py ....xFF.FFF...F.xF...FFFFFFFFFFFFFF....F.FFFFF.FFF [ 21%]
..FF...Fxx.....FFEFFFFF.F.FFFFFF                                         [ 29%]
api_exec_test.py FFFFFFFFFFFFFFFFFFFFF                                   [ 35%]
api_healthcheck_test.py FFFF                                             [ 36%]
api_image_test.py ..FFFFFFF...s..FFF                                     [ 40%]
api_network_test.py FFFFFFFFFFFFF.FF.FFFFFFFFFF                          [ 47%]
api_plugin_test.py FFFFFFFFFFF                                           [ 50%]
api_secret_test.py EEEEE                                                 [ 51%]
api_service_test.py EEEEEEEEEEEEEEEEsEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE [ 65%]
EEEEEEE                                                                  [ 67%]
api_swarm_test.py FssssFxsFFFxFFFFFFF                                    [ 72%]
api_volume_test.py F.F.FFF.F                                             [ 74%]
client_test.py ..F..                                                     [ 75%]
context_api_test.py ..                                                   [ 76%]
errors_test.py F                                                         [ 76%]
models_containers_test.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          [ 86%]
models_images_test.py FFFF..FFFFF.F.F                                    [ 89%]
models_networks_test.py FFFF                                             [ 90%]
models_nodes_test.py F                                                   [ 91%]
models_resources_test.py .                                               [ 91%]
models_services_test.py EEEEEEEEEEEEEExE                                 [ 95%]
models_swarm_test.py FF                                                  [ 96%]
models_volumes_test.py FF                                                [ 96%]
regression_test.py F...FF                                                [ 98%]
credentials/store_test.py EEEEEE                                         [ 99%]
credentials/utils_test.py .                                              [100%]

Looks like there was decent improvement in api_build_test.py and api_container_test.py.

rhatdan commented 4 years ago

FYI @jwhonce @baude

rhatdan commented 4 years ago

@gartnera Have you continued to work on this?

rhatdan commented 4 years ago

@gartnera Would be nice to see how we have progressed.

gartnera commented 4 years ago

Looks like most of the docker-py tests fail now with errors like:

API versions below 1.21 are no longer supported by this library

Worked around that by editing MINIMUM_DOCKER_API_VERSION in docker-py/docker/constants.py

Now I get these results:

============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-4.3.1, py-1.8.1, pluggy-0.13.1
rootdir: /root/docker-py, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
timeout: 60.0s
timeout method: signal
timeout func_only: False
collected 387 items

api_build_test.py F..FFFFFFFFFFFFFFFFFF..x                               [  6%]
api_client_test.py .....                                                 [  7%]
api_config_test.py EEEEE                                                 [  8%]
api_container_test.py ....xFF.FFF...F.xF...FFFFFFFFFFFFFF....F.FFFFF.F.. [ 21%]
..FF...Fxx.....FF........FFFFFF                                          [ 29%]
api_exec_test.py .FFFFFFFFFFFFFFFFFFFF                                   [ 35%]
api_healthcheck_test.py FFFF                                             [ 36%]
api_image_test.py ..FFFFF.F...s..FFF                                     [ 40%]
api_network_test.py FFFFFFFFFFFFF.FF.FFFFFFFFFF                          [ 47%]
api_plugin_test.py FFFFFFFFFFF                                           [ 50%]
api_secret_test.py EEEEE                                                 [ 51%]
api_service_test.py EEEEEEEEEEEEEEEEsEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE [ 65%]
EEEEEEE                                                                  [ 67%]
api_swarm_test.py FssssFxsFFFxFFFFFFF                                    [ 72%]
api_volume_test.py F.F.FFF.F                                             [ 74%]
client_test.py ..F..                                                     [ 75%]
context_api_test.py ..                                                   [ 76%]
errors_test.py F                                                         [ 76%]
models_containers_test.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          [ 86%]
models_images_test.py FFFF..FFFFF.F.F                                    [ 89%]
models_networks_test.py FFFF                                             [ 90%]
models_nodes_test.py F                                                   [ 91%]
models_resources_test.py .                                               [ 91%]
models_services_test.py EEEEEEEEEEEEEExE                                 [ 95%]
models_swarm_test.py FF                                                  [ 96%]
models_volumes_test.py FF                                                [ 96%]
regression_test.py ....FF                                                [ 98%]
credentials/store_test.py EEEEEE                                         [ 99%]
credentials/utils_test.py .                                              [100%]

There are six more passing tests (71 -> 77) since the last time I tested.

The most prominent errors looks like this:

During handling of the above exception, another exception occurred:
models_containers_test.py:324: in test_restart
    container = client.containers.run("alpine", "sleep 100", detach=True)
../../docker/models/containers.py:803: in run
    detach=detach, **kwargs)
../../docker/models/containers.py:861: in create
    resp = self.client.api.create_container(**create_kwargs)
../../docker/api/container.py:430: in create_container
    return self.create_container_from_config(config, name)
../../docker/api/container.py:441: in create_container_from_config
    return self._result(res, True)
../../docker/api/client.py:267: in _result
    self._raise_for_status(response)
../../docker/api/client.py:263: in _raise_for_status
    raise create_api_error_from_http_exception(e)
../../docker/errors.py:31: in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
E   docker.errors.APIError: 500 Server Error: Internal Server Error ("NewFromLocal(): unable to find a name and tag match for alpine in repotags: no such image")
___________________________ ContainerTest.test_start ___________________________
../../docker/api/client.py:261: in _raise_for_status
    response.raise_for_status()
/usr/lib/python3/dist-packages/requests/models.py:935: in raise_for_status
    raise HTTPError(http_error_msg, response=self)
E   requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/containers/create

This appears to be because podman doesn't pull images if they don't exist. To reproduce: 

root@podman:~# docker run -it --rm alpine /bin/sh
docker: Error response from daemon: NewFromLocal(): unable to find a name and tag match for alpine in repotags: no such image.
See 'docker run --help'.
root@podman:~#

I also can't run containers if I don't explicitly set the command which I suspect is the cause of other test failures:

root@podman:~# docker run -it --rm alpine
unable to upgrade to tcp, received 409

Version info:

root@podman:~# docker version
Client:
 Version:           18.09.7
 API version:       1.39
 Go version:        go1.10.1
 Git commit:        2d0083d
 Built:             Fri Aug 16 14:20:06 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: linux/amd64/ubuntu-18.04
 Podman Engine:
  Version:          2.0.0-dev
  APIVersion:       1.0.0
  Arch:             amd64
  BuildTime:        2020-06-09T20:25:53Z
  Experimental:     true
  GitCommit:        fbe09d78e91c9ac5cadc8b00a67c7d7f89d64868-dirty
  GoVersion:        go1.13.4
  KernelVersion:    5.3.0-26-generic
  MinAPIVersion:    1.0.0
  Os:               linux
 Engine:
  Version:          2.0.0-dev
  API version:      1.0.0 (minimum version 1.0.0)
  Go version:       go1.13.4
  Git commit:       fbe09d78e91c9ac5cadc8b00a67c7d7f89d64868-dirty
  Built:            Tue Jun  9 20:25:53 2020
  OS/Arch:          linux/amd64
  Experimental:     true
rhatdan commented 4 years ago

@jwhonce Is our Docker version too old?

jwhonce commented 4 years ago

@rhatdan I'll research this how they are determining version. Via ping we report compatibility API v1.40.0 the latest.

@gartnera What value did you use for MINIMUM_DOCKER_API_VERSION ? TIA

gartnera commented 4 years ago

MINIMUM_DOCKER_API_VERSION = '1.0'

github-actions[bot] commented 4 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 4 years ago

@jwhonce @gartnera Is this still an issue?

gartnera commented 4 years ago

You could close if if you'd like. I'll try and file some more specific issues. I reran the tests today, the output is below. Doesn't look like there's been much movement over the past month or two.

============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-4.3.1, py-1.8.1, pluggy-0.13.1
rootdir: /root/docker-py, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
timeout: 60.0s
timeout method: signal
timeout func_only: False
collected 387 items

api_build_test.py F..FFFFFFFFFFFFFFFFFF..x                               [  6%]
api_client_test.py .....                                                 [  7%]
api_config_test.py EEEEE                                                 [  8%]
api_container_test.py ....xFF.FFF...F.xF...FFFFFFFFFFFFFF....F.FFFFF.F.. [ 21%]
..FF....xx.....FF........FFFFFF                                          [ 29%]
api_exec_test.py .FFFFFFFFFFFFFFFFFFFF                                   [ 35%]
api_healthcheck_test.py FFFF                                             [ 36%]
api_image_test.py ..FFFFF.F...s..FFF                                     [ 40%]
api_network_test.py FFFFFFFFFFFFF.FF.FFFFFFFFFF                          [ 47%]
api_plugin_test.py FFFFFFFFFFF                                           [ 50%]
api_secret_test.py EEEEE                                                 [ 51%]
api_service_test.py EEEEEEEEEEEEEEEEsEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE [ 65%]
EEEEEEE                                                                  [ 67%]
api_swarm_test.py FssssFxsFFFxFFFFFFF                                    [ 72%]
api_volume_test.py .F..FF...                                             [ 74%]
client_test.py ..F..                                                     [ 75%]
context_api_test.py ..                                                   [ 76%]
errors_test.py F                                                         [ 76%]
models_containers_test.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          [ 86%]
models_images_test.py FFFF..FFFFF.F..                                    [ 89%]
models_networks_test.py FFFF                                             [ 90%]
models_nodes_test.py F                                                   [ 91%]
models_resources_test.py .                                               [ 91%]
models_services_test.py EEEEEEEEEEEEEExE                                 [ 95%]
models_swarm_test.py FF                                                  [ 96%]
models_volumes_test.py .F                                                [ 96%]
regression_test.py ....F.                                                [ 98%]
credentials/store_test.py EEEEEE                                         [ 99%]
credentials/utils_test.py .                                              [100%]
rhatdan commented 4 years ago

My understanding of the issues here is that a lot of docker-py tests rely on swarm and some network features, and rely on docker.io as the only registry. If docker-py wants to make their tests work with Podman or a subset, then we can do that. For now, we have been adding docker-py tests to podman but not the entire test suite.

If a community member wants to take this on it would be appreciated, but the core team has decided this is not a priority.

rhatdan commented 4 years ago

Still not a priority.

riyad commented 4 years ago

I have the same issue as in https://github.com/containers/podman/issues/5386#issuecomment-641564961

Using Ansible's docker_container tasks (using the docker python package in the background) fail with the same error: fatal: [test]: FAILED! => {"changed": false, "msg": "Docker API version is 1.0.0. Minimum version required is 1.20."}

Intrigued by https://github.com/containers/podman/issues/5386#issuecomment-642271561 I tried to find out.

Running curl --unix-socket /run/podman/podman.sock http://localhost/version | jq on Fedora 32 with Podman 2.0.6:

{
  "Platform": {
    "Name": "linux/amd64/fedora-32"
  },
  "Components": [
    {
      "Name": "Podman Engine",
      "Version": "2.0.6",
      "Details": {
        "APIVersion": "1.0.0",
        "Arch": "amd64",
        "BuildTime": "2020-09-01T19:26:51Z",
        "Experimental": "true",
        "GitCommit": "",
        "GoVersion": "go1.14.6",
        "KernelVersion": "5.4.0-47-generic",
        "MinAPIVersion": "1.0.0",
        "Os": "linux"
      }
    }
  ],
  "Version": "2.0.6",
  "ApiVersion": "1.0.0",
  "MinAPIVersion": "1.0.0",
  "GitCommit": "",
  "GoVersion": "go1.14.6",
  "Os": "linux",
  "Arch": "amd64",
  "KernelVersion": "5.4.0-47-generic",
  "Experimental": true,
  "BuildTime": "2020-09-01T19:26:51Z"
}

It seems Podman actually reports it "only" supports ApiVersion: "1.0.0". Maybe the reported version should be updated to match the actually supported API version.

riyad commented 4 years ago

I've separately reported this in #7682.

rhatdan commented 4 years ago

Well a fix for this has been merged, does this make this any better?

rhatdan commented 3 years ago

@gartnera Could you give us another run?

riyad commented 3 years ago

If it's of any help ... I just ran the integration tests myself. :slightly_smiling_face:

Distro: Ubuntu 20.04 Podman: 2.1.1 docker-py: master

============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-4.3.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/riyad/src/docker-py, inifile: pytest.ini
plugins: cov-2.6.1, timeout-1.3.3
collected 389 items

tests/integration/api_build_test.py F..FFFFF.FF.FF.FFFFFF..x             [  6%]
tests/integration/api_client_test.py .....                               [  7%]
tests/integration/api_config_test.py EEEEE                               [  8%]
tests/integration/api_container_test.py ....xFF..FF...F.xF...FFFF..FFFFF [ 16%]
FFF....F.FFFFF.F..........xx.....FF........FFFFFF                        [ 29%]
tests/integration/api_exec_test.py F........FF......FFFF                 [ 34%]
tests/integration/api_healthcheck_test.py FFFF                           [ 35%]
tests/integration/api_image_test.py ...F.FF.F...s..FFF                   [ 40%]
tests/integration/api_network_test.py FFFFFFFFFFFFF.FF.FFFFFFFFFFF       [ 47%]
tests/integration/api_plugin_test.py FFFFFFFFFFF                         [ 50%]
tests/integration/api_secret_test.py EEEEE                               [ 51%]
tests/integration/api_service_test.py EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE [ 60%]
EEEEEEEEEEEEEEEEEEEEEEEEE                                                [ 67%]
tests/integration/api_swarm_test.py FFFFFFxsFFFxFFFFFFF                  [ 71%]
tests/integration/api_volume_test.py .F..FF...                           [ 74%]
tests/integration/client_test.py ..F..                                   [ 75%]
tests/integration/context_api_test.py FFF                                [ 76%]
tests/integration/errors_test.py F                                       [ 76%]
tests/integration/models_containers_test.py .FFF.F..FFFF.FFFFFFFF.FF.... [ 83%]
.F.F.FFF.                                                                [ 86%]
tests/integration/models_images_test.py FFFF..FFFFF.F..                  [ 89%]
tests/integration/models_networks_test.py FFFF                           [ 91%]
tests/integration/models_nodes_test.py F                                 [ 91%]
tests/integration/models_resources_test.py .                             [ 91%]
tests/integration/models_services_test.py EEEEEEEEEEEEEExE               [ 95%]
tests/integration/models_swarm_test.py FF                                [ 96%]
tests/integration/models_volumes_test.py .F                              [ 96%]
tests/integration/regression_test.py F...F.                              [ 98%]
tests/integration/credentials/store_test.py EEEEEE                       [ 99%]
tests/integration/credentials/utils_test.py .                            [100%]

[...]

= 171 failed, 118 passed, 2 skipped, 8 xfailed, 1 warnings, 90 error in 1407.22 seconds =

full log

rhatdan commented 3 years ago

The interesting point would be to go into some of these to discover what is causing the failures. Is this just features that we don't implement like docker swarm. We know that the networking stack is someone different between Docker and Podman and we might be failing on some of these.

riyad commented 3 years ago

hmm ... running the tests against Podman master there seem to be a few regressions (118 passing for Podman 2.1.1, 108 passing for Podman 2.2.0-dev)

============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-4.3.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/riyad/src/docker-py, inifile: pytest.ini
plugins: cov-2.6.1, timeout-1.3.3
collected 389 items

tests/integration/api_build_test.py F...FFFF.FF.FF..F..FF..x             [  6%]
tests/integration/api_client_test.py .....                               [  7%]
tests/integration/api_config_test.py EEEEE                               [  8%]
tests/integration/api_container_test.py ..F.XFF..FFFFFF.xF.FFFFFFFFFFFFF [ 16%]
FFF....F.......F.........Fxx.....F......F..FFFFFF                        [ 29%]
tests/integration/api_exec_test.py F........FF......FFFF                 [ 34%]
tests/integration/api_healthcheck_test.py FFFF                           [ 35%]
tests/integration/api_image_test.py ...F.FF.F...s..FFF                   [ 40%]
tests/integration/api_network_test.py FFFFFFFFFFFFF.FF...FFFFFFFF.       [ 47%]
tests/integration/api_plugin_test.py FFFFFFFFFFF                         [ 50%]
tests/integration/api_secret_test.py EEEEE                               [ 51%]
tests/integration/api_service_test.py EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE [ 60%]
EEEEEEEEEEEEEEEEEEEEEEEEE                                                [ 67%]
tests/integration/api_swarm_test.py FFFFFFxsFFFxFFFFFFF                  [ 71%]
tests/integration/api_volume_test.py .F..FF...                           [ 74%]
tests/integration/client_test.py ..F..                                   [ 75%]
tests/integration/context_api_test.py FFF                                [ 76%]
tests/integration/errors_test.py F                                       [ 76%]
tests/integration/models_containers_test.py FFFFFFFFFFF.FFFFFFFFFFFFFFFF [ 83%]
FFFFFFFFF                                                                [ 86%]
tests/integration/models_images_test.py FFFF..FFFFF.F..                  [ 89%]
tests/integration/models_networks_test.py F.FF                           [ 91%]
tests/integration/models_nodes_test.py F                                 [ 91%]
tests/integration/models_resources_test.py F                             [ 91%]
tests/integration/models_services_test.py EEEEEEEEEEEEEExE               [ 95%]
tests/integration/models_swarm_test.py FF                                [ 96%]
tests/integration/models_volumes_test.py .F                              [ 96%]
tests/integration/regression_test.py F....F                              [ 98%]
tests/integration/credentials/store_test.py EEEEEE                       [ 99%]
tests/integration/credentials/utils_test.py .                            [100%]

[...]

= 181 failed, 108 passed, 2 skipped, 7 xfailed, 1 xpassed, 1 warnings, 90 error in 994.67 seconds =

full log

Diffing the tests with Podman 2.1.1(-) against those with Podman master/2.2.0-dev(+) we get roughly this:

--- tmp/pytest-integration-2.1.1.log  2020-11-25 22:35:40.882826773 +0100
+++ tmp/pytest-integration-master.log 2020-11-25 22:28:27.416694886 +0100
@@ -4,15 +4,15 @@
 plugins: cov-2.6.1, timeout-1.3.3
 collected 389 items

-tests/integration/api_build_test.py F..FFFFF.FF.FF.FFFFFF..x             [  6%]
+tests/integration/api_build_test.py F...FFFF.FF.FF..F..FF..x             [  6%]
 tests/integration/api_client_test.py .....                               [  7%]
 tests/integration/api_config_test.py EEEEE                               [  8%]
-tests/integration/api_container_test.py ....xFF..FF...F.xF...FFFF..FFFFF [ 16%]
-FFF....F.FFFFF.F..........xx.....FF........FFFFFF                        [ 29%]
+tests/integration/api_container_test.py ..F.XFF..FFFFFF.xF.FFFFFFFFFFFFF [ 16%]
+FFF....F.......F.........Fxx.....F......F..FFFFFF                        [ 29%]
 tests/integration/api_exec_test.py F........FF......FFFF                 [ 34%]
 tests/integration/api_healthcheck_test.py FFFF                           [ 35%]
 tests/integration/api_image_test.py ...F.FF.F...s..FFF                   [ 40%]
-tests/integration/api_network_test.py FFFFFFFFFFFFF.FF.FFFFFFFFFFF       [ 47%]
+tests/integration/api_network_test.py FFFFFFFFFFFFF.FF...FFFFFFFF.       [ 47%]
 tests/integration/api_plugin_test.py FFFFFFFFFFF                         [ 50%]
 tests/integration/api_secret_test.py EEEEE                               [ 51%]
 tests/integration/api_service_test.py EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE [ 60%]
@@ -22,16 +22,16 @@
 tests/integration/client_test.py ..F..                                   [ 75%]
 tests/integration/context_api_test.py FFF                                [ 76%]
 tests/integration/errors_test.py F                                       [ 76%]
-tests/integration/models_containers_test.py .FFF.F..FFFF.FFFFFFFF.FF.... [ 83%]
-.F.F.FFF.                                                                [ 86%]
+tests/integration/models_containers_test.py FFFFFFFFFFF.FFFFFFFFFFFFFFFF [ 83%]
+FFFFFFFFF                                                                [ 86%]
 tests/integration/models_images_test.py FFFF..FFFFF.F..                  [ 89%]
-tests/integration/models_networks_test.py FFFF                           [ 91%]
+tests/integration/models_networks_test.py F.FF                           [ 91%]
 tests/integration/models_nodes_test.py F                                 [ 91%]
-tests/integration/models_resources_test.py .                             [ 91%]
+tests/integration/models_resources_test.py F                             [ 91%]
 tests/integration/models_services_test.py EEEEEEEEEEEEEExE               [ 95%]
 tests/integration/models_swarm_test.py FF                                [ 96%]
 tests/integration/models_volumes_test.py .F                              [ 96%]
-tests/integration/regression_test.py F...F.                              [ 98%]
+tests/integration/regression_test.py F....F                              [ 98%]
 tests/integration/credentials/store_test.py EEEEEE                       [ 99%]
 tests/integration/credentials/utils_test.py .                            [100%]

[...]

-= 171 failed, 118 passed, 2 skipped, 8 xfailed, 1 warnings, 90 error in 1407.22 seconds =
+= 181 failed, 108 passed, 2 skipped, 7 xfailed, 1 xpassed, 1 warnings, 90 error in 994.67 seconds =
riyad commented 3 years ago

@rhatdan here's one example of what you've been asking for. :wink: It took me a while to dig through, because I'm not familiar with any of the code bases (yet). :sweat_smile:

I tried to investigate one of the regressions in tests/integration/api_container_test.py and came across the CreateContainerTest.test_group_id_ints test.

A container is incidentally created with "NetworkMode": "default". Trying to start it later fails with "no net configurations found in /etc/cni/net.d". "NetworkMode" is set to "default" as a default/fallback value by docker-py and has been for at least 4 years. Although neither documented in the CLI nor in the API docs (under HostConfig->NetworkMode) "default" seems to get special treatment within the Docker engine (but I'm not sure how far it goes). In the case of Podman it seemed to work for version 2.1.1 ... someone here might have a better idea why it started failing. FWIW changing the fallback value in docker-py to "private" makes the test pass.

Grepping through the logs the "no net configurations found in /etc/cni/net.d" error message is the (superficial?) cause of failure for 50 of the API tests ATM. :neutral_face:

Details

First request

URL: http+docker://localhost/v1.40/containers/create
Request: {'data': '{"Tty": false, "OpenStdin": false, "StdinOnce": false, "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["id", "-G"], "Image": "alpine:3.10", "NetworkDisabled": false, "HostConfig": {"NetworkMode": "default", "GroupAdd": ["1000", "1001"]}}', 'params': {'name': None}, 'headers': {'Content-Type': 'application/json'}}
Response: 201 {'Id': 'bed3d9155440b349dd869563d1884d40fb66ce738918d0dc11b4e1b69a1c226f', 'Warnings': []}

NOTE: everything besides "Cmd": ["id", "-G"], "Image": "alpine:3.10" and "GroupAdd": ["1000", "1001"] are library defaults.

Second request:

URL: http+docker://localhost/v1.40/containers/bed3d9155440b349dd869563d1884d40fb66ce738918d0dc11b4e1b69a1c226f/start
Request: {}
Response: 500 {'cause': 'command rootless-cni-infra [alloc bed3d9155440b349dd869563d1884d40fb66ce738918d0dc11b4e1b69a1c226f default thirsty_lumiere] in container fa01b23a8d01dccc6ecc680b9ba84cbf1a7a653a2dba78887f98148bc86fb545 failed with status 1, stdout="", stderr="no net configurations found in /etc/cni/net.d\\n"', 'message': 'command rootless-cni-infra [alloc bed3d9155440b349dd869563d1884d40fb66ce738918d0dc11b4e1b69a1c226f default thirsty_lumiere] in container fa01b23a8d01dccc6ecc680b9ba84cbf1a7a653a2dba78887f98148bc86fb545 failed with status 1, stdout="", stderr="no net configurations found in /etc/cni/net.d\\n"', 'response': 500}
riyad commented 3 years ago

Another regression is the ListContainersTest::test_list_containers test.

When listing containers (i.e. GET /v1.40/containers/json?all=1) the "Image" field has values like `"docker.io/library/sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c" which throws off any code that expects it to be the same as when the container was created (here it was "alpine:3.10"). This is again a regression from Podman 2.1.1.

I tracked it down to https://github.com/containers/podman/blob/ad2439264d401af0443be564ccc68169a8517db4/libpod/boltdb_state_internal.go#L383 where it unmarshalled data for config.RootfsImageName seems to already have the wrong value.

Update

I tried to figure out if the wrong value comes from creating the container ... after a restart of my computer I can't reproduce it any more (neither through CLI nor API). :sweat_smile:

Update2

After running a full suite of tests the error is back. :confused:

Update3

I found the condition which "breaks" this test. There's another test called ImageCollectionTest.test_save_and_load which triggers it. The test saves an image (here "alpine:3.10") into a file and reads it into the local repository again.

NOTE: when loading the image Podman adds the weird "docker.io/library/sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c" tag where the SHA corresponds to the actual image id). I.e. the image has now two tags. It seems Docker doesn't add the extra tag.

When we then run the original test (i.e. ListContainersTest.test_list_containers) it creates a container from the alpine:3.10 image. Here's where the weird thing happens:

I.e. whatever changed since Podman 2.1.1 it seems to be in the code creating the container.

The following (trans-)script shows what happens. It can be run in the Python interpreter (assuming docker-py is installed).

Preparation


import docker
import tempfile

TEST_IMG = 'alpine:3.10'

client = docker.from_env()

# POST http+docker://localhost/v1.40/images/create
image = client.images.pull(TEST_IMG)

#
# adapted from ImageCollectionTest.test_save_and_load (can be run against Podman 2.1.1 or 2.2-dev)
#

# GET http+docker://localhost/v1.40/images/alpine:3.10/json
image = client.images.get(TEST_IMG)
image.id
# => 'sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c'
image.tags
# => ['docker.io/library/alpine:3.10']

with tempfile.TemporaryFile() as f:
    # GET http+docker://localhost/v1.40/images/sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c/get
    stream = image.save()
    for chunk in stream:
        f.write(chunk)
    f.seek(0)
    # POST http+docker://localhost/v1.40/images/load
    # GET http+docker://localhost/v1.40/images/docker.io/library/sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c/json
    result = client.images.load(f.read())

result[0].id
# => 'sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c'
result[0].tags
# => ['docker.io/library/sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c', 'docker.io/library/alpine:3.10']

What happens with Podman 2.1.1

#
# adapted from ListContainersTest.test_list_containers (with Podman 2.1.1)
#

cntr = client.api.create_container(TEST_IMG, 'true')
# => {'Id': 'c76755a7abf76f8839a6d9c5fc6588c58fbb9b92831d33da9b6dffd5cce195c2', 'Warnings': []}
cntr['Id']
# => 'c76755a7abf76f8839a6d9c5fc6588c58fbb9b92831d33da9b6dffd5cce195c2'

all_containers = client.api.containers(all=True)
# => [{'Id': 'c76755a7abf76f8839a6d9c5fc6588c58fbb9b92831d33da9b6dffd5cce195c2', 'Names': ['/heuristic_chatelet'], 'Image': 'alpine:3.10', 'ImageID': 'be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c', 'Command': 'true', 'Created': 1606512387, 'Ports': None, 'Labels': {}, 'State': 'stopped', 'Status': '', 'NetworkSettings': None, 'Mounts': None, 'Name': '', 'Config': None, 'NetworkingConfig': None, 'Platform': None, 'AdjustCPUShares': False}]

filtered = [c for c in all_containers if c['Id'].startswith(cntr['Id'])]
# => [{'Id': 'c76755a7abf76f8839a6d9c5fc6588c58fbb9b92831d33da9b6dffd5cce195c2', 'Names': ['/heuristic_chatelet'], 'Image': 'alpine:3.10', 'ImageID': 'be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c', 'Command': 'true', 'Created': 1606512387, 'Ports': None, 'Labels': {}, 'State': 'stopped', 'Status': '', 'NetworkSettings': None, 'Mounts': None, 'Name': '', 'Config': None, 'NetworkingConfig': None, 'Platform': None, 'AdjustCPUShares': False}]
filtered[0]['Image']
# => 'alpine:3.10'
assert filtered[0]['Image'].endswith(TEST_IMG)  # PASSES

What happens with Podman 2.2.0-dev

#
# adapted from ListContainersTest.test_list_containers (with Podman 2.2.0-dev)
#

cntr = client.api.create_container(TEST_IMG, 'true')
# => {'Id': 'fbdfdfdbf4ddf69785275fa8bcd30a9a0e4ab8827d00552c2251de5549b6c4d9', 'Warnings': []}
cntr['Id']
# => 'fbdfdfdbf4ddf69785275fa8bcd30a9a0e4ab8827d00552c2251de5549b6c4d9'

all_containers = client.api.containers(all=True)
# => [{'Id': 'fbdfdfdbf4ddf69785275fa8bcd30a9a0e4ab8827d00552c2251de5549b6c4d9', 'Names': ['/clever_bohr'], 'Image': 'docker.io/library/sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c', 'ImageID': 'be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c', 'Command': 'true', 'Created': 1606511556, 'Ports': None, 'Labels': {}, 'State': 'exited', 'Status': '', 'NetworkSettings': None, 'Mounts': None, 'Name': '', 'Config': None, 'NetworkingConfig': None, 'Platform': None, 'AdjustCPUShares': False}]

filtered = [c for c in all_containers if c['Id'].startswith(cntr['Id'])]
# => [{'Id': 'fbdfdfdbf4ddf69785275fa8bcd30a9a0e4ab8827d00552c2251de5549b6c4d9', 'Names': ['/clever_bohr'], 'Image': 'docker.io/library/sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c', 'ImageID': 'be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c', 'Command': 'true', 'Created': 1606511556, 'Ports': None, 'Labels': {}, 'State': 'exited', 'Status': '', 'NetworkSettings': None, 'Mounts': None, 'Name': '', 'Config': None, 'NetworkingConfig': None, 'Platform': None, 'AdjustCPUShares': False}]
filtered[0]['Image']
# => 'docker.io/library/sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c'
assert filtered[0]['Image'].endswith(TEST_IMG)  # FAILS
riyad commented 3 years ago

@rhatdan I've added quite some details ... maybe someone with more insight could have a look at it? Should I open separate issues for them?

baude commented 3 years ago

i would say yes. i can look at them. repros help alot!

riyad commented 3 years ago

FYI: I've created separate issues for the two regressions I've found.

riyad commented 3 years ago

I've run the tests again and we're making progress :smile: We're now passing 21 tests more than Podman 2.1.1.

riyad commented 3 years ago

The next regression I'm looking at is PortTest.test_port. It's weird, because it's one of three tests that still fail with the "no net configurations found in /etc/cni/net.d" error messages from #8544.

It tries to configure a container with 3 exposed ports and port mappings:

curl -sS --unix-socket /var/run/user/1000/podman/podman.sock -XPOST 'http://localhost/v1.40/containers/create' -H 'Content-Type: application/json' -d '{"ExposedPorts": {"1111/tcp": {}, "2222/tcp": {}, "3333/udp": {}}, "Tty": false, "OpenStdin": false, "StdinOnce": false, "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["sleep", "60"], "Image": "alpine:3.10", "NetworkDisabled": false, "HostConfig": {"NetworkMode": "bridge", "PortBindings": {"1111/tcp": [{"HostIp": "127.0.0.1", "HostPort": "4567"}], "2222/tcp": [{"HostIp": "127.0.0.1", "HostPort": "4568"}], "3333/udp": [{"HostIp": "127.0.0.1", "HostPort": "4569"}]}}}'
# => {"Id":"8a175a1ceb8e08a58203a6b47c574b688bf635a2bb2373e3f605099c5c1da753","Warnings":[]}

But when starting the container it fails with:

curl -sS --unix-socket /var/run/user/1000/podman/podman.sock -XPOST 'http://localhost/v1.40/containers/8a175a1ceb8e08a58203a6b47c574b688bf635a2bb2373e3f605099c5c1da753/start'
# => {"cause":"command rootless-cni-infra [alloc 8a175a1ceb8e08a58203a6b47c574b688bf635a2bb2373e3f605099c5c1da753 bridge cranky_napier] in container ff322e52aca832ecab78f4595fdc1968c19b3d6aaef4b58af685c16b735e26bf failed with status 1, stdout=\"\", stderr=\"no net configurations found in /etc/cni/net.d\\n\"","message":"command rootless-cni-infra [alloc 8a175a1ceb8e08a58203a6b47c574b688bf635a2bb2373e3f605099c5c1da753 bridge cranky_napier] in container ff322e52aca832ecab78f4595fdc1968c19b3d6aaef4b58af685c16b735e26bf failed with status 1, stdout=\"\", stderr=\"no net configurations found in /etc/cni/net.d\\n\"","response":500}

I've not had time yet to whittle down the repro to see what exactly triggers the failure, but it seems suspicions that its yet another "NetworkMode". :thinking:

kwiesmueller commented 3 years ago

Hey there, I'm currently trying to make a project work with podman and am also getting the unable to upgrade to tcp, received 409 using the podman socket with docker-cli.

It happens for basic commands like docker run hello-world /bin/bash.

Client: Docker Engine - Community
 Version:           20.10.0-rc2
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        dca98e3
 Built:             Wed Dec  2 22:19:20 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: linux/amd64/fedora-33
 Podman Engine:
  Version:          2.1.1
  APIVersion:       2.0.0
  Arch:             amd64
  BuildTime:        2020-10-07T18:21:20+02:00
  Experimental:     true
  GitCommit:        
  GoVersion:        go1.15.2
  KernelVersion:    5.9.11-200.fc33.x86_64
  MinAPIVersion:    2.0.0
  Os:               linux
 Engine:
  Version:          2.1.1
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.15.2
  Git commit:       
  Built:            Wed Oct  7 18:21:20 2020
  OS/Arch:          linux/amd64
  Experimental:     true
riyad commented 3 years ago

The bug in #8547 gets more and more confusing by the minute :confounded: ... I tried to compare Podman 2.1.1 with 3.0-dev (e2f91207fc3e515c0a9f828433aaf80727f4b7c5) and CLI with API behavior:

yajo commented 3 years ago

take a closer look at the tests ... they use swarm and a bunch of stuff we will never do.

I don't know if you're aware of the test filtering features pytest has. You can test specific files of the suite, or even filter them with expressions using -k or fixtures (depending on how the test suite is organized). Probably it would be a good idea to filter out tests that are known to fail and still run the ones that should.

You can maintain all those settings in a pytest.ini file, or through CLI flags in the CI command.

rhatdan commented 3 years ago

Interested in opening a PR to do this?

riyad commented 3 years ago

This is how my pytest.ini currently looks like:

[pytest]
addopts =
    --tb=short
    -rxs
    --no-header
    -v
    -k 'not SwarmTest and not NodesTest and not ServiceTest and not SecretAPITest and not TestStore and not ConfigAPITest and not test_create_network_attachable and not test_create_network_ingress and not test_create_inspect_network_with_scope'

junit_suite_name = docker-py
junit_family = xunit2

testpaths =
    tests/integration/

I set testpaths to only run the integration tests (i.e. exercising the API and not testing docker-py internals). The -k option currently filters all tests that rely on Docker Swarm either directly or indirectly (~116 tests at the moment). The rest of the options is basically for the output to be in a useful format (for me).

@rhatdan Is there a rough list of (API) features/aspects that Podman is not going to replicate? e.g.

rhatdan commented 3 years ago

That is a good starting list. @baude @mheon WDYT?

jwhonce commented 3 years ago

@riyad There are some outstanding endpoints like /network/prune that are TBD on being implemented.

riyad commented 3 years ago

Because @matejvasek asked over in another PR:

I've put together some scripts and documentation on how to run the docker-py integration tests against Podman. Please have a look at https://github.com/riyad/podman-integration-test-utils

I hope someone finds it useful. :smiley:

matejvasek commented 3 years ago

@riyad nice, I'll try it out.

riyad commented 3 years ago

I hope all the issues I'm opening aren't getting on your nerves :sweat_smile:

TomSweeneyRedHat commented 3 years ago

@riyad never! But we'd love it if you sent in a PR or two too! :smiley:

github-actions[bot] commented 3 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 3 years ago

@riyad Is this still an issue?

riyad commented 3 years ago

IMHO this issue can be closed. :+1:

If I'm allowed to reinterpret the original issue: it was steering Podman's API efforts towards something that could be described as "ecosystem compatibility". With all the effort put in fixing issues uncovered by the docker-py test suite as well as the docker-compose compatibility I'd attest that this matter is taken seriously. :smile: Although there're still gaps in compatibility it's enough to address them individually (as has been the case lately anyway).

With regard to the docker-py test suite I've put together https://github.com/riyad/podman-integration-test-utils as a way to document and automate running docker-py's integration tests against Podman's API service. It also has some rudimentary tools (and data) to display and track the progress in API compatibility.

To summarize Podman's progress ... is quite impressive to be honest: Passing tests per Podman verison:

These are the "conservative" numbers: you'll generally get +2 if using crun and there're 1-3 tests which are flaky.