search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.83k stars 2.42k forks source link

Podman 2.2.1 client not connecting to Podman 2.0.5 (nighty rhcos 4.7) due to API version mismatch? #9098

Closed gbraad closed 3 years ago

gbraad commented 3 years ago

When I execute the following command

PS> .\podman.exe system connection add --identity C:\Users\gbraad\.crc\machines\crc\id_ecdsa --port 2222 crc core@127.0.0.1

an error is returned:

 Error: failed to parse 'podman info' results: json: cannot unmarshal number into Go struct field Version.version.APIVersion of type string

The command

PS> ssh -i C:\Users\gbraad\.crc\machines\crc\id_ecdsa -p 2222 core@127.0.0.1 podman info

provides the following output:

host:
  arch: amd64
  buildahVersion: 1.15.1
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-2.rhaos4.6.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: e5c6f4123734b4e16cbd22b8f9475a6d02a52e8e'
  cpus: 4
  distribution:
    distribution: '"rhcos"'
    version: "4.7"
  eventLogger: file
  hostname: crc-ctkkw-master-0
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-240.10.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 213295104
  memTotal: 9403904000
  ociRuntime:
    name: runc
    package: runc-1.0.0-81.rhaos4.6.git5b757d4.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.0
      libseccomp: 2.4.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.4-2.module+el8.3.0+8221+97165c3f.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 0
  swapTotal: 0
  uptime: 28m 54.87s
registries:
  search:
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-3.module+el8.3.0+8221+97165c3f.x86_64
      Version: |-
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 1600877882
  BuiltTime: Wed Sep 23 16:18:02 2020
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.0.5

The client used was: https://github.com/containers/podman/releases/tag/v2.2.1 podman-remote-release-windows.zip

gbraad commented 3 years ago

Using https://github.com/containers/podman/releases/tag/v2.0.6 seems to work on the Command Prompt: 

C:\Users\gbraad\Downloads\podman>.\podman.exe system connection add --identity C:\Users\gbraad\.crc\machines\crc\id_ecdsa --port 2222 crc core@127.0.0.1

C:\Users\gbraad\Downloads\podman>podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

C:\Users\gbraad\Downloads\podman>podman version
Client:
Version:      2.0.6
API Version:  1
Go Version:   go1.14.6
Git Commit:   27362ba1ad8879ea71610fa68a651a1651e0180f
Built:        Wed Sep  2 01:08:00 2020
OS/Arch:      windows/amd64

Server:
Version:      2.0.5
API Version:  0
Go Version:   go1.14.7
Built:        Thu Sep 24 00:18:02 2020
OS/Arch:      linux/amd64

C:\Users\gbraad\Downloads\podman>

Although it works from PowerShell, it will always complain about the console mode (Filed as issue: #9100)

PS> ./podman ps
time="2021-01-26T14:28:58+08:00" level=error msg="Failed to set console mode for cli" error="The parameter is incorrect."
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
PS> ./podman version
time="2021-01-26T14:29:15+08:00" level=error msg="Failed to set console mode for cli" error="The parameter is incorrect."
Client:
Version:      2.0.6
API Version:  1
Go Version:   go1.14.6
Git Commit:   27362ba1ad8879ea71610fa68a651a1651e0180f
Built:        Wed Sep  2 01:08:00 2020
OS/Arch:      windows/amd64

Server:
Version:      2.0.5
API Version:  0
Go Version:   go1.14.7
Built:        Thu Sep 24 00:18:02 2020
OS/Arch:      linux/amd64

In other words; there is no backwards compatibility for the client when using a higher version with an older server component.

baude commented 3 years ago

@gbraad in your first comment, after you added the connection ... did it get set to the default? podman system connection ls should indicate the default connection with a *. If it isn't set, try setting the default and seeing if your command works.

gbraad commented 3 years ago

It was the only connection (and default)

crc*    C:\Users\gbraad\.crc\machines\crc\id_ecdsa  ssh://core@127.0.0.1:2222/run/user/1000/podman/podman.sock

you noticed it failed with:

Error: failed to parse 'podman info' results: json: cannot unmarshal number into Go struct field Version.version.APIVersion of type string

In the info this says for RHCOS (podman 2.0.5):

version:
  APIVersion: 1
  Built: 1600877882
  BuiltTime: Wed Sep 23 16:18:02 2020
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.0.5

While on Fedora 33 (podman 2.1.1) this is:

version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 22:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

notet the APIVersion.

afbjorklund commented 3 years ago

It changed in c4b49afad37b4e64a1aa2b90f4d3b04626fabd44 - I don't think there is compatibility between 2.0 and 2.1, so you need to use a 2.0 client ?

Same thing with 1.9, due to using a different protocol and socket. So far, 2.2 server seems to work with 3.0 client...

But the API bumped from 2.0.0 to 3.0.0 so there are no guarantees.

Version API Commit
1.9 varlink
2.0 1 5c8e7ed0dee117c9d423849f09f82d15957893a8
2.1 2.0.0 c4b49afad37b4e64a1aa2b90f4d3b04626fabd44
2.2 2.1.0 02843f881f9271440e7eaad8db231ddf20e33e51
3.0 3.0.0 12f835d125bda9c6971258d54e627b8b2acf859a
gbraad commented 3 years ago

For now I am stuck on 2.0.6 as client which seems to work. So, yeah. Can confirm this behaviour, but not as expected.

afbjorklund commented 3 years ago

From what I can tell, the podman3 server is still using podman2 protocol. Or I guess that should read "Podman 2.1+"

Client:
Version:      3.0.0-rc1
API Version:  3.0.0
Go Version:   go1.15.5
Git Commit:   1cbe815ee74e835f72ee7c63857126b5aedbee39
Built:        Fri Jan 22 17:28:01 2021
OS/Arch:      linux/amd64

Server:
Version:      3.0.0-rc1
API Version:  2.0.0
Go Version:   go1.15.5
Git Commit:   1cbe815ee74e835f72ee7c63857126b5aedbee39
Built:        Fri Jan 22 17:27:52 2021
OS/Arch:      linux/amd64

I'm just going to write "Podman 2.1 or later" for my requirements, and leave support for 1.6 and 2.0 as RHEL exclusive...

mheon commented 3 years ago

Eh?

Alright, that's a bug. Supposed to be 3.0 on both ends.

afbjorklund commented 3 years ago

Alright, that's a bug. Supposed to be 3.0 on both ends.

Should I open an issue, seems to be the same with RC2 ?

Client:
Version:      3.0.0-rc2
API Version:  3.0.0
Go Version:   go1.15.5
Git Commit:   510a983a752a136a5df0bdbff9d14db0490956b2
Built:        Sun Jan 31 18:56:25 2021
OS/Arch:      linux/amd64

Server:
Version:      3.0.0-rc2
API Version:  2.0.0
Go Version:   go1.15.5
Git Commit:   510a983a752a136a5df0bdbff9d14db0490956b2
Built:        Sun Jan 31 18:56:08 2021
OS/Arch:      linux/amd64

Should this really be "Libpod" twice ?

        // LibpodTree supports Libpod endpoints
        LibpodTree = VersionTree(iota)
        // CompatTree supports Libpod endpoints
        CompatTree

Not sure difference between "CurrentAPIVersion" and "APIVersion" ?

    // APIVersion provides the current and minimal API versions for compat and libpod endpoint trees
    // Note: GET|HEAD /_ping is never versioned and provides the API-Version and Libpod-API-Version headers to allow
    //       clients to shop for the Version they wish to support
    APIVersion = map[VersionTree]map[VersionLevel]semver.Version{
        LibpodTree: {
            CurrentAPIVersion: semver.MustParse("2.0.0"),
            MinimalAPIVersion: semver.MustParse("2.0.0"),
        },
        CompatTree: {
            CurrentAPIVersion: semver.MustParse("1.40.0"),
            MinimalAPIVersion: semver.MustParse("1.24.0"),
        },
    }
// APIVersion is the version for the remote
// client API.  It is used to determine compatibility
// between a remote podman client and its backend
var APIVersion = semver.MustParse("3.0.0")
afbjorklund commented 3 years ago

Found a third place, in the bindings (podman-remote)

        // APIVersion - podman will fail to run if this value is wrong
        APIVersion = semver.MustParse("2.0.0")

If only that is changed, it's compatible-like-python:

Error: Cannot connect to the Podman socket, make sure there is a Podman REST API service running.: server API version is too old. Client "3.0.0" server "2.0.0"

Fail to see where the version negotiation is taking place ?

It seems to assume that server version is always newer.

mheon commented 3 years ago

Yep, please open a bug. I thought I hit everywhere when I bumped us to 3.0.0-dev, but clearly I missed one.

rhatdan commented 3 years ago

Since this is now fixed in podman 3.0, I am going to close this issue. Reopen if I am mistaken.

afbjorklund commented 3 years ago

Since this is now fixed in podman 3.0, I am going to close this issue. Reopen if I am mistaken.

It probably won't fix it for podman 2.0, so there one has to keep using the podman 2.0 client...

Might want to document those shortcomings, when running older servers ? (before version 2.1)

But seems to be working for podman 2.2, so as long as client is 2.1+ (not 1.9, not 2.0) it's OK.

Client:
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.5
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Server:
Version:      3.0.0-dev
API Version:  3.0.0
Go Version:   go1.15.5
Git Commit:   c421127dd7f700829a8e5265d8ddad102061bebc
Built:        Fri Feb  5 13:19:30 2021
OS/Arch:      linux/amd64
Client:
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Server:
Version:      3.0.0-dev
API Version:  3.0.0
Go Version:   go1.15.5
Git Commit:   c421127dd7f700829a8e5265d8ddad102061bebc
Built:        Fri Feb  5 13:19:30 2021
OS/Arch:      linux/amd64

If I understand the original bug report (above), this might be something to document on the RHCOS 4.7 side ?