API: gitlab-runner docker test suite fails when used with podman

[skorhone]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description This is informational issue that should be closed when test suite execution passes.

gitlab-runner contains a test suite for verifying Docker API compatibility. Considering that Podman's API should be docker compatible, podman should pass gitlab-runners test suite without modifications.

Instructions on where to locate gitlab-runner tests can be located at: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4185

Describe the results you received: Tests fail. Connection counters after tests seem to be way off:

DEBU[2741] IdleTracker 0xc000468110:new 82/2515 connection(s) 
DEBU[2741] IdleTracker 0xc000468110:active 83/2516 connection(s) 

Describe the results you expected: Tests pass

[skorhone]

I hope you don't mind opening this as an issue. I just wanted a common place where to share information about gitlab test suites. My thought was that you can use existing test sets such as gitlab runners test set to verify API compatibility.

[mheon]

I think we'd prefer individual issues for bugs found by said test suites, but trackers for overall test suites might not be the worst thing. @rhatdan @baude Thoughts?

[baude]

does it perform swarm based tests? or things like renaminig containers?

[skorhone]

I don't think it does either. I can check tomorrow.

Suite does seem to include bunch of interesting API calls that currently fail with podman. The way that GitLab runner uses containers isn't the most typical way of using containers, so I'd expect these tests to reveal issues that would be otherwise hard to find.

[baude]

@skorhone Ok great. Would love to see you on IRC so we can collab on this.

[skorhone]

I have created a naive cirrus-ci project which demonstrates how it could be run. Unfortunately I just can't get podman to run properly in a container created by cirrus-ci. I get permissions errors when creating network and tmpfs :-(

Project is here: https://github.com/skorhone/libpod-gitlab-it/tree/cirrus Example run can be located here: https://cirrus-ci.com/task/5682090681303040

[c-goes]

@skorhone Maybe configure podman to use slirp4ns? This worked well when I tried to run podman in LXD containers. But I'm not sure if it helps here.

[skorhone]

I guess I could give rootless a try. It shouldn't be all too hard to install required components and run podman as non-root - I bet that cirrus ci is running build container without privileges anyways.

[skorhone]

Updated test sets to run as rootless and it still fails. Now error is a bit different though and failure happens when pulling image: https://cirrus-ci.com/task/4677718585376768

I'll try slrip4netss as root next. Otoh I think that it might be just easiest to get a vm

Edit: There is no command line option to force network for root user to slirp4netns for podman system service

I tried to keep cirrus ci code as simple as possible so others may use it. So please try and report you results here. I'll get back on this after my vacation

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@skorhone Is this still an issue?

[skorhone]

@rhatdan Haven't checked recently. I was away on 4week vacation, so I have few things that I need to sort out at work before I can get back on this.

I'd assume that it will still fail, if API used for pulling images has not changed. Pull implementation should hijack connection and post status on progress. Last time I checked, podman's API returned immediately and didn't report progress

Once that is sorted out I can do further diagnosis.

[c-goes]

@skorhone I tried something similar with Travis-CI. I made it rootfull because it seems to be possible with Travis-CI.

https://github.com/c-goes/podman-gitlab-testing/blob/master/.travis.yml

https://travis-ci.org/github/c-goes/podman-gitlab-testing/builds/721686928

I don't know yet how to run the Podman API in background correctly (I add & after the command). The API seems to stop working somewhere in the build. Is this what you expect as output?

[skorhone]

I would expect to see debug messages from podman, if integration tests were calling the podman service.

Instead of having multiple shell steps, you could try creating a small shell script that starts service and runs tests. CI might just kill container process between steps

Also, since your build is running on a container and not on a vm, podman service might not have enough privileges to run

[c-goes]

Thanks, I made some progess, now with output from both Gitlab Test and Podman API

https://travis-ci.org/github/c-goes/podman-gitlab-testing/builds/721980418

As you can see, some tests are able to run now, but after the API keeps writing these lines until travis stops because of too many lines.

[PODMAN_API] WARNING: The same type, major and minor should not be used for multiple devices.
[PODMAN_API] WARNING: The same type, major and minor should not be used for multiple devices.
[PODMAN_API] WARNING: The same type, major and minor should not be used for multiple devices.

Here are the last lines before this happens

[GITLAB_TEST] Using docker image sha256:27105793dc2bbf270ec87d6a9ba041bba52cfef66384251980a4d55d03ed736c for docker:18-dind with digest docker.io/library/docker@sha256:86df3c3573065f2c6f24cd925fd5bc3a0aff899bdf664ff4d2e3ebab26d96bed ...
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="IdleTracker 0xc000010d80:idle 26/6659 connection(s)"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="IdleTracker 0xc000010d80:active 25/6659 connection(s)"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="APIHandler -- Method: GET URL: /v1.25/networks"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="IdleTracker 0xc000010d80:idle 26/6660 connection(s)"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="IdleTracker 0xc000010d80:active 25/6660 connection(s)"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="APIHandler -- Method: DELETE URL: /v1.25/containers/runner--project-0-concurrent-0-ec65409ca81f5b49-docker-0?force=1&v=1"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=info msg="Request Failed(Not Found): no container with name or ID runner--project-0-concurrent-0-ec65409ca81f5b49-docker-0 found: no such container"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="IdleTracker 0xc000010d80:idle 26/6661 connection(s)"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="IdleTracker 0xc000010d80:active 25/6661 connection(s)"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="APIHandler -- Method: POST URL: /v1.25/containers/create?name=runner--project-0-concurrent-0-ec65409ca81f5b49-docker-0"
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/var/run/containers/storage]@27105793dc2bbf270ec87d6a9ba041bba52cfef66384251980a4d55d03ed736c\""
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/var/run/containers/storage]@27105793dc2bbf270ec87d6a9ba041bba52cfef66384251980a4d55d03ed736c\""
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="exporting opaque data as blob \"sha256:27105793dc2bbf270ec87d6a9ba041bba52cfef66384251980a4d55d03ed736c\""
[PODMAN_API] time="2020-08-28T10:57:25Z" level=debug msg="User mount runner--project-0-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8:/builds options []"
[PODMAN_API] WARNING: The same type, major and minor should not be used for multiple devices.
[PODMAN_API] WARNING: The same type, major and minor should not be used for multiple devices.
[PODMAN_API] WARNING: The same type, major and minor should not be used for multiple devices.
[PODMAN_API] WARNING: The same type, major and minor should not be used for multiple devices.

Apparently, the test starts Docker in a container. Maybe this doesn't make much sense for this test. I'm not sure how to proceed.

[skorhone]

I have a feeling that test fails because podman doesn't currently handle pull (create image) request same way that docker does and because container creation code path is more or less broken

Docker hijacks the connection and streams progress updates until image is available. Podman just sends single line and handles request asynchronously on the background.

Container creation in podman API unfortunately does not currently use same code path with podman cli. API handles many of the parameters and sets some of default values incorrectly. Ihmo correct way to resolve these issues is to share code path with cli and API. I know what would be required to implement it correctly, just don't have the time ☹️ I think rewriting and testing API for that part would take few days

[mheon]

Container creation is definitely on our radar, that's being tracked already - I don't know when we'll be able to get to it, unfortunately.

The image pull thing is new, I think we need a fresh issue for that.

[psgreco]

Don't know if this is appropriate here, but this is how I've been testing this. CentOS8, with 2.0.5 + https://github.com/containers/podman/commit/cb61a2d858d7874c66bafcf6d17c17bb9d7849e8 backported (so it can create the volumes). Gitlab runner seems to think that everything worked, but the commands sent though the API are not really executed

[c-goes]

I am not yet sure how to reproduce the image pull bug. Will look into it a few days.

[c-goes]

Test suite output is different now.

https://travis-ci.org/github/c-goes/podman-gitlab-testing/jobs/737844838

This WARNING: The same type, major and minor should not be used for multiple devices. isn't there any more.

[rhatdan]

@c-goes @skorhone @baude is this still and issue in the main branch?

[baude]

it shouldnt be ... we have people using runner