crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.26k stars 242 forks source link

[QE] fix e2e test failure of podman #4424

Closed lilyLuLiu closed 2 days ago

lilyLuLiu commented 3 weeks ago

Test case Pull image locally, push to registry, deploy sometimes failed:

stderr: Cannot connect to Podman. Please verify your connection to the Linux system using podman system connection list, or try podman machine init and podman machine start to manage a new Linux VM Error: unable to connect to Podman socket: failed to connect: dial tcp 192.168.130.11:22: i/o timeout

But this error never appear in nightly run. And I checked the machine, podman work well.

this is b/c somehow it is trying to use podman remote instead of podman.

praveenkumar commented 3 weeks ago

this is b/c somehow it is trying to use podman remote instead of podman.

Can you point out the feature file for that? Also I think we want to use podman remote instead podman on system since it is not default available for mac/windows.

adrianriobo commented 3 weeks ago

Tests should be using podman-remote for win/mac and podman for linux

lilyLuLiu commented 3 weeks ago

@praveenkumar , This failure all happened on tests of Linux.

praveenkumar commented 2 weeks ago

Tests should be using podman-remote for win/mac and podman for linux

@adrianriobo Does that mean for win/mac it is using podman socket which is running inside the crc VM or using podman-machine vm?

adrianriobo commented 2 weeks ago

Yeah, for win and mac any scenario making use of podman is relying on "our" podman (crc-VM)

praveenkumar commented 2 weeks ago

Yeah, for win and mac any scenario making use of podman is relying on "our" podman (crc-VM)

@adrianriobo so in this case we should also do same for linux right?

adrianriobo commented 2 weeks ago

Well, we are not making it for testing our podman setup, but b/c we rely on some container engine. In the end we somehow would like to test the interaction between podman / crc.

I would suggest keep the same strategy, as in future we remove the podman socket or make more tricky to make use of it ..we may end on a situation where we need to use podman machine on win and mac and keep using podman on linux.

dunno...WDYT?

praveenkumar commented 2 weeks ago

What we want to have our test case follow same steps on all platform to have reliable metrics in case of this https://github.com/crc-org/crc/blob/main/test/e2e/features/story_openshift.feature#L55-L66 (which is causing issue).

Well, we are not making it for testing our podman setup, but b/c we rely on some container engine. In the end we somehow would like to test the interaction between podman / crc.

In this case I would suggest to have a different test case only for linux which use the podman installed on the system instead using podman-socket from the VM.

adrianriobo commented 2 weeks ago

Well the intention should be scenario spec should be the same the tech underneath implementing it is controlled by code (in this case use "our" podman socket for win and mac and podman for linux).

And this is fine...what I do not know is why on Linux is not working as expected and it is complaining about the socket. And this should be target of this issue, know why and fix it

albfan commented 2 weeks ago

From source:

https://github.com/containers/podman/blob/7567cc1fcd0764b60386ecbeacaaf575a067e926/cmd/podman/root.go#L120-L124

https://github.com/containers/podman/blob/9b8b2bf684aeb76199467d351d02a014befdcb0b/cmd/podman/registry/remote.go#L21-L23

If we just call on purpose podman remote, should we avoid that on linux?

https://github.com/crc-org/crc/blob/6652e57dbb6d3d317db49dd9ba71edf4de86c043/test/e2e/features/story_openshift.feature#L55-L66

praveenkumar commented 2 weeks ago

It is because we have https://github.com/crc-org/crc/blob/main/test/extended/crc/cmd/cmd.go#L214C2-L217C44 logic so what I suspect that on linux since we are running podman-remote command we have to make sure we execute crc podman-env command.

adrianriobo commented 2 weeks ago

Ohh my bad from top of my head I thought that should be using podman not podman-remote...about the crc podman-env this is kind of executed with line 57 on that scenario: Given podman command is available

That line goes to https://github.com/crc-org/crc/blob/d9f83bcc27dc8717469091553c90d9b97c2b5105/test/e2e/testsuite/testsuite.go#L942

lilyLuLiu commented 2 weeks ago

After investigation, the podman-remote command failed because CONTAINER_HOST is set as default network system-mode, but actual network is user-mode. The network mode is changed to user-mode in @story_microshift case, and didn't change back to default.