User socket events endpoint returns error 500

[bertmelis]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description I'm using Debian 10 with podman from the Kubik project as outlined in the podman documentation: https://podman.io/getting-started/installation.

The user events socket endpoint returns error 500:

$ curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/events
{"cause":"failed to get cursor: cannot assign requested address","message":"failed to get journal cursor: failed to get cursor: cannot assign requested address","response":500}

Steps to reproduce the issue:

1. Install podman following the docs: https://podman.io/getting-started/installation

Describe the results you received:

{"cause":"failed to get cursor: cannot assign requested address","message":"failed to get journal cursor: failed to get cursor: cannot assign requested address","response":500}

Describe the results you expected: The command to wait for events.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.14
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 4
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: server
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.19.0-16-amd64
  linkmode: dynamic
  memFree: 9554034688
  memTotal: 16453894144
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.4
  swapFree: 1024454656
  swapTotal: 1024454656
  uptime: 14h 14m 10.39s (Approximately 0.58 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/bert/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 4
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/bert/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 34
  runRoot: /run/user/1000/containers
  volumePath: /home/bert/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown,now 100:3.0.1-2 amd64 [installed]
podman/unknown 100:3.0.1-2 arm64
podman/unknown 100:3.0.1-2 armhf
podman/unknown 100:3.0.1-2 ppc64el
podman/unknown 100:3.0.1-2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/ No

Additional environment details (AWS, VirtualBox, physical, etc.): physical

[Luap99]

Can you do a podman pull alpine for example and try again. I think the problem is that you don't have any podman events in the journal.

[bertmelis]

Doesn't change.

Mind that it returns an error, it doesn't stay empty. As I understand it should "hang" waiting for something to show but it immediately returns with the error.

[Luap99]

Do see anything in journalctl --user SYSLOG_IDENTIFIER=podman?

[bertmelis]

Should this be done with 'sudo' or without? Because without I don't have sufficient permissions. And with sudo the journal is empty.

I do have a few (rootless) containers running and they are running fine.

[Luap99]

This should work as rootless. podman events uses the journald backend by default. Your user has apparently no enough permissions to use it. I recommend changing the events_logger to file in /etc/containers/containers.conf

[bertmelis]

Does this file exist by default?

[Luap99]

Depends on the packaging. If it does not exists you can copy it from /usr/share/containers/containers.conf to this location.

[bertmelis]

It works by setting events_logger to file. I'll close the issue for now. Is there anything that can be done to make it work out-of-the-box?

[mheon]

Any chance you can try again with 3.1.0? @ashley-cui did a lot of fixes to the journald backend for that release.

[Luap99]

3.1 cannot fix this when journalctl --user SYSLOG_IDENTIFIER=podman does not work rootless. This is a distro specific configuration problem.

[bertmelis]

Any chance you can try again with 3.1.0? @ashley-cui did a lot of fixes to the journald backend for that release.

I'm inclined to say no, as 3.1.0 is not in the Kubik project's repo.