search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.26k stars 2.37k forks source link

Speed of listing images - caching layer possible? #6984

Closed srcshelton closed 4 years ago

srcshelton commented 4 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman is incredibly slow at listing images:

# uptime ; time podman image ls | tee podman.out ; echo -n 'Images: ' ; wc -l < podman.out ; uptime
 10:11:07 up 40 days, 5 min,  0 users,  load average: 0.74, 0.49, 0.36
REPOSITORY                                  TAG              IMAGE ID      CREATED         SIZE
localhost/latest                            latest           2137118fa991  14 minutes ago  145 MB
localhost/8640fbb                           latest           2137118fa991  14 minutes ago  145 MB
<none>                                      <none>           1b95d0e0f038  20 minutes ago  145 MB
localhost/service.dev-lang.php              7.4.6            83958d7cf800  10 hours ago    265 MB
<none>                                      <none>           1a6537708309  10 hours ago    1.07 GB
<none>                                      <none>           827478021b95  17 hours ago    265 MB
<none>                                      <none>           e54a2110da2a  17 hours ago    1.07 GB
<none>                                      <none>           bba5c68560d2  18 hours ago    265 MB
<none>                                      <none>           a746f88e2c5a  18 hours ago    1.07 GB
localhost/service.mail-filter.spamassassin  3.4.4-r4         9c77b26ad853  24 hours ago    262 MB
<none>                                      <none>           4a0f1d6ca4f9  24 hours ago    888 MB
localhost/service.net-mail.imapproxy        1.2.8_p14843-r1  5509a4159054  37 hours ago    84.6 MB
<none>                                      <none>           d98e8726b038  37 hours ago    715 MB
localhost/service.app-forensics.lynis       3.0.0            5c1ff064d6c9  39 hours ago    64.8 MB
<none>                                      <none>           24b70fe09f3f  39 hours ago    703 MB
localhost/service.net-misc.zxtm             20.1             1f1c998aa4dd  40 hours ago    786 MB
<none>                                      <none>           651a12f05477  40 hours ago    976 MB
localhost/service.sys-apps.watchdog         5.15             2bba1b5ed4d2  40 hours ago    49 MB
<none>                                      <none>           e8dc701c7347  40 hours ago    690 MB
localhost/service.app-editors.vim           8.2.0360         b6fc4c9aaae7  40 hours ago    118 MB
<none>                                      <none>           4f702fb492a2  40 hours ago    755 MB
localhost/service.app-admin.syslog-ng       3.26.1-r1        42a2ab10b9a8  40 hours ago    93.6 MB
<none>                                      <none>           531fa9f62e32  40 hours ago    725 MB
localhost/service.dev-vcs.subversion        1.13.0-r1        04014827e5bc  40 hours ago    104 MB
<none>                                      <none>           a2697e172704  40 hours ago    737 MB
localhost/service.mail-filter.spampd        2.53             50ad516ca164  40 hours ago    226 MB
<none>                                      <none>           7071cfc413d9  40 hours ago    846 MB
<none>                                      <none>           3e751a6fccda  40 hours ago    226 MB
<none>                                      <none>           993d86969ac4  40 hours ago    845 MB
localhost/service.mail-filter.postgrey      1.36-r1          5cae88413500  40 hours ago    154 MB
<none>                                      <none>           0c056650a424  40 hours ago    782 MB
localhost/service.dev-db.redis              5.0.8            0e825459ec94  40 hours ago    65.5 MB
<none>                                      <none>           227178f692ed  40 hours ago    889 MB
localhost/service.mail-mta.postfix          3.5.1            55c9f9397a5d  40 hours ago    219 MB
<none>                                      <none>           73f026fc7422  40 hours ago    1.02 GB
localhost/service.app-text.pinfo            0.6.13           290cb761fa71  41 hours ago    130 MB
<none>                                      <none>           6b47d6b38e7c  41 hours ago    771 MB
<none>                                      <none>           a4994f266c78  41 hours ago    265 MB
<none>                                      <none>           5b43b853651d  41 hours ago    1.07 GB
localhost/service.net-misc.openntpd         6.2_p3-r1        7e603a3479be  41 hours ago    60.2 MB
<none>                                      <none>           c5297f2a00ce  41 hours ago    1.05 GB
localhost/service.mail-filter.opendkim      2.10.3-r17       07b87a7b9821  41 hours ago    111 MB
<none>                                      <none>           2c572c3a0bce  41 hours ago    740 MB
localhost/service.net-dns.bind              9.14.12          7e02c7c7d1b1  41 hours ago    115 MB
<none>                                      <none>           5671c8a1b829  41 hours ago    925 MB
localhost/service.net-misc.memcached        1.6.6            ab472d0133ae  41 hours ago    140 MB
<none>                                      <none>           d3de4ad95250  41 hours ago    768 MB
localhost/service.dev-db.mariadb            10.4.13          8b934738cf9b  41 hours ago    398 MB
<none>                                      <none>           ead902013745  41 hours ago    1.2 GB
localhost/service.sys-apps.man-db           2.8.7            8ce43f1bc54c  41 hours ago    76.4 MB
<none>                                      <none>           457b1b2dbbda  41 hours ago    725 MB
localhost/service.www-servers.lighttpd      1.4.55           ed60d095bafe  41 hours ago    174 MB
<none>                                      <none>           091afc6bb66e  41 hours ago    820 MB
localhost/service.sys-apps.less             551              4d61b37f5a32  41 hours ago    124 MB
<none>                                      <none>           2e448d4a74ca  41 hours ago    760 MB
localhost/service.sys-apps.irqbalance       1.6.0            e89520ada1db  41 hours ago    70.2 MB
<none>                                      <none>           392d37b0a91d  41 hours ago    710 MB
<none>                                      <none>           eecc5bfb5292  42 hours ago    87.8 MB
<none>                                      <none>           7cef898b4bee  42 hours ago    760 MB
localhost/service.net-mail.fetchmail        6.4.1            8b6194ad2407  42 hours ago    202 MB
<none>                                      <none>           273cbde80c1b  42 hours ago    1.01 GB
localhost/service.net-mail.dovecot          2.3.10.1         a0356994eba3  42 hours ago    153 MB
<none>                                      <none>           bd261e1b8555  42 hours ago    963 MB
localhost/service.net-misc.dhcp             4.4.2-r2         bacc53cb3709  42 hours ago    58.6 MB
<none>                                      <none>           cfd440c9732f  42 hours ago    700 MB
localhost/service.net-im.bitlbee            3.6-r1           fcd9c6127f51  42 hours ago    89.5 MB
<none>                                      <none>           a2532be19182  42 hours ago    720 MB
localhost/gentoo-build                      latest           633c831fa6e6  43 hours ago    690 MB
<none>                                      <none>           ec71a931d071  43 hours ago    1.72 GB
localhost/gentoo-base                       latest           d23182392cf4  43 hours ago    1.72 GB
localhost/gentoo-init                       latest           30121a0f5fe4  44 hours ago    1.03 GB
localhost/gentoo-stage3                     latest           62fa243ea174  44 hours ago    1.03 GB
localhost/gentoo-env                        latest           51b1a2937896  44 hours ago    10.7 kB
<none>                                      <none>           875171b2fb2e  44 hours ago    84.6 MB
<none>                                      <none>           036e3165830a  44 hours ago    715 MB
docker.io/gentoo/stage3-amd64               latest           c70e33aa5447  2 days ago      1.03 GB
<none>                                      <none>           99806bd548bc  3 days ago      76.4 MB
<none>                                      <none>           6a3eec76c0dc  3 days ago      725 MB
<none>                                      <none>           8942c2b4040c  3 days ago      130 MB
<none>                                      <none>           471797f24984  3 days ago      130 MB
<none>                                      <none>           a494c4c4e9a3  3 days ago      130 MB
<none>                                      <none>           9a126560fa30  3 days ago      64.7 MB
<none>                                      <none>           64b12d1da299  3 days ago      713 MB
<none>                                      <none>           b6f6c85855a7  3 days ago      130 MB
<none>                                      <none>           133c13bce431  3 days ago      765 MB
<none>                                      <none>           d2ae06642b6d  4 days ago      1.72 GB
<none>                                      <none>           93a4a1551100  4 days ago      64.7 MB
<none>                                      <none>           a25d79808bbe  4 days ago      713 MB
<none>                                      <none>           a01db39c2fba  4 days ago      130 MB
<none>                                      <none>           335459b6aa0b  4 days ago      765 MB
<none>                                      <none>           b735ce1e4c46  4 days ago      1.03 GB
<none>                                      <none>           9e706d873344  4 days ago      130 MB
<none>                                      <none>           acdcdbf10855  4 days ago      765 MB
<none>                                      <none>           aafa412132d4  4 days ago      64.6 MB
<none>                                      <none>           d5863505f64a  4 days ago      713 MB
<none>                                      <none>           938664454317  4 days ago      130 MB
<none>                                      <none>           033f8471c92e  4 days ago      765 MB
<none>                                      <none>           897850baa2d5  4 days ago      130 MB
<none>                                      <none>           96a42719dd95  4 days ago      771 MB
<none>                                      <none>           3a416608c34b  4 days ago      130 MB
<none>                                      <none>           d779aac271a9  4 days ago      765 MB
<none>                                      <none>           b8741e14c100  4 days ago      130 MB
<none>                                      <none>           d5f6d4e654ef  4 days ago      765 MB
<none>                                      <none>           1d3e0003d596  4 days ago      130 MB
<none>                                      <none>           078b176dd455  4 days ago      765 MB
<none>                                      <none>           09af27287834  4 days ago      130 MB
<none>                                      <none>           dd7be3e83bc2  4 days ago      765 MB
<none>                                      <none>           167df1234901  4 days ago      130 MB
<none>                                      <none>           75571706ad27  4 days ago      765 MB
localhost/web.app-admin.webapp-config       1.55-r1          afdc2a772a89  4 days ago      238 MB
localhost/dell-dsu                          20.07.00         8041c5888010  5 days ago      422 MB
<none>                                      <none>           8c398976a067  5 days ago      174 MB
<none>                                      <none>           64dd3d03ae72  5 days ago      786 MB
<none>                                      <none>           8f9dec94908c  5 days ago      49 MB
<none>                                      <none>           d088335cf686  5 days ago      118 MB
<none>                                      <none>           5049eabc8f28  5 days ago      93.6 MB
<none>                                      <none>           2bee355de09f  5 days ago      104 MB
<none>                                      <none>           d463a33e09a5  5 days ago      226 MB
<none>                                      <none>           7740732bd361  5 days ago      226 MB
<none>                                      <none>           5d0be37da8cc  5 days ago      154 MB
<none>                                      <none>           4bb00fcea7d0  5 days ago      65.5 MB
<none>                                      <none>           1047f8a9eb82  5 days ago      219 MB
<none>                                      <none>           471f7af65e62  5 days ago      130 MB
<none>                                      <none>           bbb32803166b  5 days ago      265 MB
<none>                                      <none>           f31bcbc949f5  5 days ago      60.2 MB
<none>                                      <none>           e51b189b71be  5 days ago      111 MB
<none>                                      <none>           cf4846d12b2e  5 days ago      115 MB
<none>                                      <none>           cd71cb19b201  5 days ago      140 MB
<none>                                      <none>           e98a8ffeee6b  5 days ago      398 MB
<none>                                      <none>           c279b0640619  5 days ago      64.6 MB
localhost/service.app-forensics.lynis       2.7.5            0791c23fb86d  5 days ago      64.6 MB
<none>                                      <none>           f8e81f5d48d2  5 days ago      124 MB
<none>                                      <none>           41bac80c13ad  5 days ago      70.2 MB
<none>                                      <none>           84e335060b5c  5 days ago      84.6 MB
<none>                                      <none>           2de3375a3cae  5 days ago      202 MB
<none>                                      <none>           b12bb4dc698f  5 days ago      154 MB
<none>                                      <none>           d69354bd6f24  5 days ago      58.6 MB
<none>                                      <none>           d1ce3b8ae1b1  5 days ago      89.6 MB
<none>                                      <none>           c1505d73c140  5 days ago      1.73 GB
<none>                                      <none>           359ec59b50c8  6 days ago      1.03 GB
localhost/buildweb-app-admin.webapp-config  1.55-r1          bd8e9847625c  8 days ago      1.1 GB
localhost/dell-dsu                          20.06.01         bcb40bee57bf  11 days ago     422 MB
localhost/dell-ism                          351              9f6900e9b5b5  11 days ago     185 MB
docker.io/github/super-linter               v3.2.0           a41ee5cec3f6  13 days ago     2.25 GB
localhost/buddyworks-linter                 latest           49c755953f29  2 weeks ago     145 MB
docker.io/hadolint/hadolint                 latest           02830ef1af8b  3 weeks ago     9.03 MB
docker.io/library/ubuntu                    18.04            8e4ce0a6ce69  4 weeks ago     66.6 MB
docker.io/library/centos                    8                831691599b88  4 weeks ago     223 MB
docker.io/library/busybox                   latest           1c35c4412082  6 weeks ago     1.44 MB
docker.io/library/alpine                    3                a24bb4013296  6 weeks ago     5.85 MB
docker.io/projectatomic/dockerfile-lint     latest           e59cf123e1f0  2 months ago    304 MB
docker.io/koalaman/shellcheck               stable           3d5a3cb1aa47  3 months ago    6.69 MB
docker.io/pulsesecure/vtm                   20.1             1232e940dcc2  4 months ago    808 MB
docker.io/replicated/dockerfilelint         latest           4be3dc374e89  6 months ago    268 MB
docker.io/library/node                      10.16.3-alpine   b95baba1cfdb  11 months ago   80.7 MB
docker.io/redcoolbeans/dockerlint           latest           0f351f204f51  17 months ago   45.4 MB

real    1m20.049s
user    1m8.151s
sys     0m31.684s
Images: 157
 10:12:27 up 40 days, 6 min,  0 users,  load average: 1.58, 0.84, 0.50

... which is about half a second per image.

The host system has 4 enterprise spinning disks in a RAID-6 array, with the filesystems podman is using residing on LVM volumes... so this shouldn't be too much of a bottleneck.

Can the amount of data that podman needs to interrogate per image be reduced (perhaps the image size could be hidden by default if this has to be computed on the fly)? On the other hand, doing time podman image ls -q to show only the digests also took 1m18.990s, so it looks as if data is being computed regardless of the output filter - or the the delay lies elsewhere.

Alternatively, could podman maintain a cache or database of image attributes which is kept updated internally, allowing image listing to be near-instantaneous?

(Listing a specific image, e.g. podman image ls docker is much faster... but sometimes a view of all images is needed).

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.2
Git Commit:   201c9505b88f451ca877d29a73ed0f1836bb96c7
Built:        Wed Jul  8 17:39:51 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.17, commit: 41877362fc4685d55e0473d2e4a1cbe5e1debee0'
  cpus: 8
  distribution:
    distribution: gentoo
    version: unknown
  eventLogger: file
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.38-gentoo
  linkmode: dynamic
  memFree: 733523968
  memTotal: 8132182016
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 25209147392
  swapTotal: 25769787392
  uptime: 960h 14m 44.42s (Approximately 40.00 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 9
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "false"
  graphRoot: /space/podman/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2465
  runRoot: /space/podman/run
  volumePath: /space/podman/volumes
version:
  APIVersion: 1
  Built: 1594229991
  BuiltTime: Wed Jul  8 17:39:51 2020
  GitCommit: 201c9505b88f451ca877d29a73ed0f1836bb96c7
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.2
vrothberg commented 4 years ago

Thanks for opening the issue! We're tracking the topic already over in https://github.com/containers/podman/issues/6288, so I am going to close this one here. Feel free to join the conversation over in https://github.com/containers/podman/issues/6288.

github-actions[bot] commented 4 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 4 years ago

This issue was supposed to be closed.