kata-containers / tests

Kata Containers tests, CI, and metrics
https://katacontainers.io/
Apache License 2.0
140 stars 196 forks source link

docker: dnf -y update failing using 9pfs #2358

Closed GabyCT closed 3 years ago

GabyCT commented 4 years ago

While trying to perform a dnf -y update on a Fedora 31 container we have the following error

10:10:04 
10:10:04 Total download size: 1.2 M
10:10:04 Downloading Packages:
10:10:04 shadow-utils-4.6-17.fc31.x86_64.rpm             6.2 MB/s | 1.2 MB     00:00    
10:10:04 --------------------------------------------------------------------------------
10:10:04 Total                                           2.9 MB/s | 1.2 MB     00:00     
10:10:04 Running transaction check
10:10:04 Transaction check succeeded.
10:10:04 Running transaction test
10:10:04 Transaction test succeeded.
10:10:04 Running transaction
10:10:04   Preparing        :                                                        1/1 
10:10:04   Upgrading        : shadow-utils-2:4.6-17.fc31.x86_64                      1/2 
10:10:04   Verifying        : shadow-utils-2:4.6-17.fc31.x86_64                      1/2 
10:10:04   Verifying        : shadow-utils-2:4.6-16.fc31.x86_64                      2/2 
10:10:04 
10:10:04 Failed:
10:10:04   shadow-utils-2:4.6-16.fc31.x86_64      shadow-utils-2:4.6-17.fc31.x86_64     
10:10:04 
10:10:04 
10:10:04 Stderr: Error unpacking rpm package shadow-utils-2:4.6-17.fc31.x86_64
10:10:04 Error: Transaction failed

Fedora 31 was released some time ago, it is not clear what make the regression.

GabyCT commented 4 years ago

We are running the following code

$ docker run -ti --runtime kata-runtime Fedora sh
$ dnf -y update
GabyCT commented 4 years ago

This is also present on ubuntu

GabyCT commented 4 years ago

any idea of why this is happening @fidencio? thanks

fidencio commented 4 years ago

Any chance to get the logs from /var/log/dnf.log (on the container)? It'd give us a better idea of what may be the causing the issue.

GabyCT commented 4 years ago
2020-03-10T15:42:47Z INFO --- logging initialized ---
2020-03-10T15:42:47Z DDEBUG timer: config: 133 ms
2020-03-10T15:42:47Z DEBUG DNF version: 4.2.18
2020-03-10T15:42:47Z DDEBUG Command: dnf -y update
2020-03-10T15:42:47Z DDEBUG Installroot: /
2020-03-10T15:42:47Z DDEBUG Releasever: 31
2020-03-10T15:42:47Z DEBUG cachedir: /var/cache/dnf
2020-03-10T15:42:47Z DDEBUG Base command: update
2020-03-10T15:42:47Z DDEBUG Extra commands: ['-y', 'update']
2020-03-10T15:42:47Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2020-03-10T15:42:47Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2020-03-10T15:42:47Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2020-03-10T15:42:47Z DEBUG User-Agent: constructed: 'libdnf (Fedora 31; container; Linux.x86_64)'
2020-03-10T15:42:47Z DEBUG repo: downloading from remote: fedora-modular
2020-03-10T15:42:52Z DEBUG fedora-modular: using metadata from Wed Oct 23 22:53:13 2019.
2020-03-10T15:42:52Z DEBUG repo: downloading from remote: updates-modular
2020-03-10T15:43:01Z DEBUG updates-modular: using metadata from Fri Mar  6 22:30:57 2020.
2020-03-10T15:43:01Z DEBUG repo: downloading from remote: updates
2020-03-10T15:43:15Z DEBUG updates: using metadata from Mon Mar  9 21:47:41 2020.
2020-03-10T15:43:15Z DEBUG repo: downloading from remote: fedora
2020-03-10T15:44:54Z DEBUG fedora: using metadata from Wed Oct 23 22:52:47 2019.
2020-03-10T15:44:54Z INFO Last metadata expiration check: 0:00:02 ago on Tue Mar 10 15:43:55 2020.
2020-03-10T15:45:00Z DDEBUG timer: sack setup: 133143 ms
2020-03-10T15:45:00Z DEBUG --> Starting dependency resolution
2020-03-10T15:45:02Z DEBUG ---> Package mkpasswd.x86_64 5.5.6-1.fc31 will be installed
2020-03-10T15:45:02Z DEBUG ---> Package shared-mime-info.x86_64 1.15-1.fc31 will be installed
2020-03-10T15:45:02Z DEBUG ---> Package whois-nls.noarch 5.5.6-1.fc31 will be installed
2020-03-10T15:45:02Z DEBUG ---> Package glib2.x86_64 2.62.4-2.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package glib2.x86_64 2.62.5-1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libarchive.x86_64 3.4.0-1.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libarchive.x86_64 3.4.2-1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libsss_idmap.x86_64 2.2.2-3.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libsss_idmap.x86_64 2.2.3-13.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libsss_nss_idmap.x86_64 2.2.2-3.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libsss_nss_idmap.x86_64 2.2.3-13.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libtirpc.x86_64 1.2.5-0.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libtirpc.x86_64 1.2.5-1.rc2.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libxcrypt.x86_64 4.4.12-1.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libxcrypt.x86_64 4.4.15-1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package pcre.x86_64 8.43-2.fc31.1 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package pcre.x86_64 8.43-3.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package pcre2.x86_64 10.34-5.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package pcre2.x86_64 10.34-7.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package shadow-utils.x86_64 2:4.6-16.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package shadow-utils.x86_64 2:4.6-17.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package sssd-client.x86_64 2.2.2-3.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package sssd-client.x86_64 2.2.3-13.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package sudo.x86_64 1.8.29-1.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package sudo.x86_64 1.9.0-0.1.b1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package vim-minimal.x86_64 2:8.2.236-1.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package vim-minimal.x86_64 2:8.2.348-1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG --> Finished dependency resolution
2020-03-10T15:45:02Z DDEBUG timer: depsolve: 2097 ms
2020-03-10T15:45:02Z INFO Dependencies resolved.
2020-03-10T15:45:04Z INFO =============================================================================================================================================================================================
 Package                                           Architecture                            Version                                            Repository                                Size
=============================================================================================================================================================================================
Upgrading:
 glib2                                             x86_64                                  2.62.5-1.fc31                                      updates                                  2.6 M
 libarchive                                        x86_64                                  3.4.2-1.fc31                                       updates                                  384 k
 libsss_idmap                                      x86_64                                  2.2.3-13.fc31                                      updates                                   46 k
 libsss_nss_idmap                                  x86_64                                  2.2.3-13.fc31                                      updates                                   53 k
 libtirpc                                          x86_64                                  1.2.5-1.rc2.fc31                                   updates                                   98 k
 libxcrypt                                         x86_64                                  4.4.15-1.fc31                                      updates                                  126 k
 pcre                                              x86_64                                  8.43-3.fc31                                        updates                                  192 k
 pcre2                                             x86_64                                  10.34-7.fc31                                       updates                                  261 k
 shadow-utils                                      x86_64                                  2:4.6-17.fc31                                      updates                                  1.2 M
 sssd-client                                       x86_64                                  2.2.3-13.fc31                                      updates                                  110 k
 sudo                                              x86_64                                  1.9.0-0.1.b1.fc31                                  updates                                  1.0 M
 vim-minimal                                       x86_64                                  2:8.2.348-1.fc31                                   updates                                  645 k
Installing dependencies:
 whois-nls                                         noarch                                  5.5.6-1.fc31                                       updates                                   34 k
Installing weak dependencies:
 mkpasswd                                          x86_64                                  5.5.6-1.fc31                                       updates                                   41 k
 shared-mime-info                                  x86_64                                  1.15-1.fc31                                        updates                                  303 k

Transaction Summary
=============================================================================================================================================================================================
Install   3 Packages
Upgrade  12 Packages

2020-03-10T15:45:04Z INFO Total download size: 7.1 M
2020-03-10T15:45:04Z INFO Downloading Packages:
2020-03-10T15:45:07Z INFO ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2020-03-10T15:45:07Z INFO Total                                                                                                                                                        2.7 MB/s | 7.1 MB     00:02
2020-03-10T15:45:09Z INFO Running transaction check
2020-03-10T15:45:09Z INFO Transaction check succeeded.
2020-03-10T15:45:09Z INFO Running transaction test
2020-03-10T15:45:10Z INFO Transaction test succeeded.
2020-03-10T15:45:10Z DDEBUG timer: transaction test: 905 ms
2020-03-10T15:45:10Z INFO Running transaction
2020-03-10T15:45:11Z DDEBUG RPM transaction start.
2020-03-10T15:45:38Z DDEBUG RPM transaction over.
2020-03-10T15:45:38Z DEBUG Errors occurred during transaction.
2020-03-10T15:45:39Z DDEBUG timer: verify transaction: 822 ms
2020-03-10T15:45:39Z DDEBUG timer: transaction: 28969 ms
2020-03-10T15:45:41Z INFO
Upgraded:
  glib2-2.62.5-1.fc31.x86_64             libarchive-3.4.2-1.fc31.x86_64    libsss_idmap-2.2.3-13.fc31.x86_64    libsss_nss_idmap-2.2.3-13.fc31.x86_64    libtirpc-1.2.5-1.rc2.fc31.x86_64
  libxcrypt-4.4.15-1.fc31.x86_64         pcre-8.43-3.fc31.x86_64           pcre2-10.34-7.fc31.x86_64            sssd-client-2.2.3-13.fc31.x86_64         sudo-1.9.0-0.1.b1.fc31.x86_64
  vim-minimal-2:8.2.348-1.fc31.x86_64

Installed:
  mkpasswd-5.5.6-1.fc31.x86_64                                shared-mime-info-1.15-1.fc31.x86_64                                whois-nls-5.5.6-1.fc31.noarch

Failed:
  shadow-utils-2:4.6-16.fc31.x86_64                                                             shadow-utils-2:4.6-17.fc31.x86_64

2020-03-10T15:45:41Z DDEBUG Cleaning up.
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/whois-nls-5.5.6-1.fc31.noarch.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libarchive-3.4.2-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libsss_nss_idmap-2.2.3-13.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/mkpasswd-5.5.6-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/pcre-8.43-3.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/shadow-utils-4.6-17.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/sudo-1.9.0-0.1.b1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libsss_idmap-2.2.3-13.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/shared-mime-info-1.15-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/glib2-2.62.5-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/vim-minimal-8.2.348-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libxcrypt-4.4.15-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/sssd-client-2.2.3-13.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libtirpc-1.2.5-1.rc2.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/pcre2-10.34-7.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z SUBDEBUG
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 65, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 98, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 122, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 166, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 257, in do_transaction
    raise dnf.exceptions.Error(_('Transaction failed'))
dnf.exceptions.Error: Transaction failed
2020-03-10T15:45:41Z CRITICAL Error: Transaction failed

Here it is @fidencio

fidencio commented 4 years ago

Firstly, thanks @GabyCT for providing me a machine where I could test that. Here's what I did in order to try to reproduce your issue:

gabyopen@gabyopen:~> sudo docker run -ti --runtime kata-runtime fedora sh
sh-5.0# dnf -y update
Fedora Modular 31 - x86_64                                                                              2.2 MB/s | 5.2 MB     00:02    
Fedora Modular 31 - x86_64 - Updates                                                                    1.6 MB/s | 4.0 MB     00:02    
Fedora 31 - x86_64 - Updates                                                                            5.4 MB/s |  22 MB     00:03    
Fedora 31 - x86_64                                                                                      5.1 MB/s |  71 MB     00:13    
Last metadata expiration check: 0:00:01 ago on Tue Mar 10 16:59:06 2020.
Dependencies resolved.
========================================================================================================================================
 Package                             Architecture              Version                                 Repository                  Size
========================================================================================================================================
Upgrading:
 glib2                               x86_64                    2.62.5-1.fc31                           updates                    2.6 M
 libarchive                          x86_64                    3.4.2-1.fc31                            updates                    384 k
 libsss_idmap                        x86_64                    2.2.3-13.fc31                           updates                     46 k
 libsss_nss_idmap                    x86_64                    2.2.3-13.fc31                           updates                     53 k
 libtirpc                            x86_64                    1.2.5-1.rc2.fc31                        updates                     98 k
 libxcrypt                           x86_64                    4.4.15-1.fc31                           updates                    126 k
 pcre                                x86_64                    8.43-3.fc31                             updates                    192 k
 pcre2                               x86_64                    10.34-7.fc31                            updates                    261 k
 shadow-utils                        x86_64                    2:4.6-17.fc31                           updates                    1.2 M
 sssd-client                         x86_64                    2.2.3-13.fc31                           updates                    110 k
 sudo                                x86_64                    1.9.0-0.1.b1.fc31                       updates                    1.0 M
 vim-minimal                         x86_64                    2:8.2.348-1.fc31                        updates                    645 k
Installing dependencies:
 whois-nls                           noarch                    5.5.6-1.fc31                            updates                     34 k
Installing weak dependencies:
 mkpasswd                            x86_64                    5.5.6-1.fc31                            updates                     41 k
 shared-mime-info                    x86_64                    1.15-1.fc31                             updates                    303 k

Transaction Summary
========================================================================================================================================
Install   3 Packages
Upgrade  12 Packages

Total download size: 7.1 M
Downloading Packages:
(1/15): whois-nls-5.5.6-1.fc31.noarch.rpm                                                                74 kB/s |  34 kB     00:00    
(2/15): mkpasswd-5.5.6-1.fc31.x86_64.rpm                                                                 84 kB/s |  41 kB     00:00    
(3/15): shared-mime-info-1.15-1.fc31.x86_64.rpm                                                         427 kB/s | 303 kB     00:00    
(4/15): libarchive-3.4.2-1.fc31.x86_64.rpm                                                              991 kB/s | 384 kB     00:00    
(5/15): libsss_idmap-2.2.3-13.fc31.x86_64.rpm                                                           156 kB/s |  46 kB     00:00    
(6/15): glib2-2.62.5-1.fc31.x86_64.rpm                                                                  3.3 MB/s | 2.6 MB     00:00    
(7/15): libsss_nss_idmap-2.2.3-13.fc31.x86_64.rpm                                                       147 kB/s |  53 kB     00:00    
(8/15): libtirpc-1.2.5-1.rc2.fc31.x86_64.rpm                                                            398 kB/s |  98 kB     00:00    
(9/15): libxcrypt-4.4.15-1.fc31.x86_64.rpm                                                              1.1 MB/s | 126 kB     00:00    
(10/15): pcre-8.43-3.fc31.x86_64.rpm                                                                    1.3 MB/s | 192 kB     00:00    
(11/15): pcre2-10.34-7.fc31.x86_64.rpm                                                                  1.3 MB/s | 261 kB     00:00    
(12/15): shadow-utils-4.6-17.fc31.x86_64.rpm                                                            2.9 MB/s | 1.2 MB     00:00    
(13/15): sssd-client-2.2.3-13.fc31.x86_64.rpm                                                           281 kB/s | 110 kB     00:00    
(14/15): sudo-1.9.0-0.1.b1.fc31.x86_64.rpm                                                              2.7 MB/s | 1.0 MB     00:00    
(15/15): vim-minimal-8.2.348-1.fc31.x86_64.rpm                                                          2.4 MB/s | 645 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                   2.8 MB/s | 7.1 MB     00:02     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                1/1
  Upgrading        : vim-minimal-2:8.2.348-1.fc31.x86_64                                                                           1/27 
  Upgrading        : pcre-8.43-3.fc31.x86_64                                                                                       2/27 
  Upgrading        : glib2-2.62.5-1.fc31.x86_64                                                                                    3/27 
  Installing       : shared-mime-info-1.15-1.fc31.x86_64                                                                           4/27 
  Running scriptlet: shared-mime-info-1.15-1.fc31.x86_64                                                                           4/27 
  Upgrading        : libsss_nss_idmap-2.2.3-13.fc31.x86_64                                                                         5/27 
  Upgrading        : libsss_idmap-2.2.3-13.fc31.x86_64                                                                             6/27 
  Installing       : whois-nls-5.5.6-1.fc31.noarch                                                                                 7/27 
  Upgrading        : libxcrypt-4.4.15-1.fc31.x86_64                                                                                8/27 
  Installing       : mkpasswd-5.5.6-1.fc31.x86_64                                                                                  9/27 
  Upgrading        : shadow-utils-2:4.6-17.fc31.x86_64                                                                            10/27 
Error unpacking rpm package shadow-utils-2:4.6-17.fc31.x86_64
  Upgrading        : sssd-client-2.2.3-13.fc31.x86_64                                                                             11/27 
error: unpacking of archive failed on file /usr/bin/newgidmap;5e67c78a: cpio: cap_set_file
error: shadow-utils-2:4.6-17.fc31.x86_64: install failed

  Running scriptlet: sssd-client-2.2.3-13.fc31.x86_64                                                                             11/27 
  Upgrading        : sudo-1.9.0-0.1.b1.fc31.x86_64                                                                                12/27 
  Running scriptlet: sudo-1.9.0-0.1.b1.fc31.x86_64                                                                                12/27 
  Upgrading        : pcre2-10.34-7.fc31.x86_64                                                                                    13/27 
  Upgrading        : libtirpc-1.2.5-1.rc2.fc31.x86_64                                                                             14/27 
  Upgrading        : libarchive-3.4.2-1.fc31.x86_64                                                                               15/27 
  Running scriptlet: sssd-client-2.2.2-3.fc31.x86_64                                                                              16/27 
  Cleanup          : sssd-client-2.2.2-3.fc31.x86_64                                                                              16/27 
  Cleanup          : sudo-1.8.29-1.fc31.x86_64                                                                                    17/27 
error: shadow-utils-2:4.6-16.fc31.x86_64: erase skipped

  Cleanup          : glib2-2.62.4-2.fc31.x86_64                                                                                   18/27 
  Cleanup          : pcre-8.43-2.fc31.1.x86_64                                                                                    19/27 
  Cleanup          : vim-minimal-2:8.2.236-1.fc31.x86_64                                                                          20/27 
  Cleanup          : libxcrypt-4.4.12-1.fc31.x86_64                                                                               21/27 
  Cleanup          : libsss_idmap-2.2.2-3.fc31.x86_64                                                                             22/27 
  Cleanup          : libsss_nss_idmap-2.2.2-3.fc31.x86_64                                                                         23/27 
  Cleanup          : pcre2-10.34-5.fc31.x86_64                                                                                    24/27 
  Cleanup          : libtirpc-1.2.5-0.fc31.x86_64                                                                                 25/27 
  Cleanup          : libarchive-3.4.0-1.fc31.x86_64                                                                               26/27 
  Running scriptlet: libarchive-3.4.0-1.fc31.x86_64                                                                               26/27 
  Verifying        : mkpasswd-5.5.6-1.fc31.x86_64                                                                                  1/27 
  Verifying        : shared-mime-info-1.15-1.fc31.x86_64                                                                           2/27 
  Verifying        : whois-nls-5.5.6-1.fc31.noarch                                                                                 3/27 
  Verifying        : glib2-2.62.5-1.fc31.x86_64                                                                                    4/27 
  Verifying        : glib2-2.62.4-2.fc31.x86_64                                                                                    5/27 
  Verifying        : libarchive-3.4.2-1.fc31.x86_64                                                                                6/27 
  Verifying        : libarchive-3.4.0-1.fc31.x86_64                                                                                7/27 
  Verifying        : libsss_idmap-2.2.3-13.fc31.x86_64                                                                             8/27 
  Verifying        : libsss_idmap-2.2.2-3.fc31.x86_64                                                                              9/27 
  Verifying        : libsss_nss_idmap-2.2.3-13.fc31.x86_64                                                                        10/27 
  Verifying        : libsss_nss_idmap-2.2.2-3.fc31.x86_64                                                                         11/27 
  Verifying        : libtirpc-1.2.5-1.rc2.fc31.x86_64                                                                             12/27 
  Verifying        : libtirpc-1.2.5-0.fc31.x86_64                                                                                 13/27 
  Verifying        : libxcrypt-4.4.15-1.fc31.x86_64                                                                               14/27 
  Verifying        : libxcrypt-4.4.12-1.fc31.x86_64                                                                               15/27 
  Verifying        : pcre-8.43-3.fc31.x86_64                                                                                      16/27 
  Verifying        : pcre-8.43-2.fc31.1.x86_64                                                                                    17/27 
  Verifying        : pcre2-10.34-7.fc31.x86_64                                                                                    18/27 
  Verifying        : pcre2-10.34-5.fc31.x86_64                                                                                    19/27 
  Verifying        : shadow-utils-2:4.6-17.fc31.x86_64                                                                            20/27 
  Verifying        : shadow-utils-2:4.6-16.fc31.x86_64                                                                            21/27 
  Verifying        : sssd-client-2.2.3-13.fc31.x86_64                                                                             22/27 
  Verifying        : sssd-client-2.2.2-3.fc31.x86_64                                                                              23/27 
  Verifying        : sudo-1.9.0-0.1.b1.fc31.x86_64                                                                                24/27 
  Verifying        : sudo-1.8.29-1.fc31.x86_64                                                                                    25/27 
  Verifying        : vim-minimal-2:8.2.348-1.fc31.x86_64                                                                          26/27 
  Verifying        : vim-minimal-2:8.2.236-1.fc31.x86_64                                                                          27/27

Upgraded:
  glib2-2.62.5-1.fc31.x86_64                     libarchive-3.4.2-1.fc31.x86_64               libsss_idmap-2.2.3-13.fc31.x86_64         
  libsss_nss_idmap-2.2.3-13.fc31.x86_64          libtirpc-1.2.5-1.rc2.fc31.x86_64             libxcrypt-4.4.15-1.fc31.x86_64            
  pcre-8.43-3.fc31.x86_64                        pcre2-10.34-7.fc31.x86_64                    sssd-client-2.2.3-13.fc31.x86_64          
  sudo-1.9.0-0.1.b1.fc31.x86_64                  vim-minimal-2:8.2.348-1.fc31.x86_64         

Installed:
  mkpasswd-5.5.6-1.fc31.x86_64              shared-mime-info-1.15-1.fc31.x86_64              whois-nls-5.5.6-1.fc31.noarch             

Failed:
  shadow-utils-2:4.6-16.fc31.x86_64                                  shadow-utils-2:4.6-17.fc31.x86_64                                 

Error: Transaction failed

The important part of the error is: error: unpacking of archive failed on file /usr/bin/newgidmap;5e67c78a: cpio: cap_set_file, which seems to be possibly fixed by: https://github.com/torvalds/linux/commit/8db6c34f1dbc8e06aa016a9b829b06902c3e1340.

Mind, though, that this is a guess.

Right now I'm dropping this info here and later on I'll try to force docker to use devicemapper, which although quite slow, may be a reasonable workaround.

fidencio commented 4 years ago

Okay, I've done:

gabyopen@gabyopen:~> sudo cat /etc/systemd/system/docker.service.d/kata-containers.conf 
[Service]
Environment=""
Environment=""
ExecStart=
#ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/local/bin/kata-runtime --default-runtime=runc --storage-driver=overlay2
ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/local/bin/kata-runtime --default-runtime=runc --storage-driver=devicemapper
gabyopen@gabyopen:~> sudo systemctl daemon-reload
gabyopen@gabyopen:~> sudo systemctl restart docker
gabyopen@gabyopen:~> sudo docker run -ti --runtime kata-runtime fedora sh
Unable to find image 'fedora:latest' locally
latest: Pulling from library/fedora
5c1b9e8d7bf7: Pull complete 
Digest: sha256:c97879f8bebe49744307ea5c77ffc76c7cc97f3ddec72fb9a394bd4e4519b388
Status: Downloaded newer image for fedora:latest
sh-5.0# dnf -y update
Fedora Modular 31 - x86_64                                                                                                                                                                                                                        3.3 MB/s | 5.2 MB     00:01    
Fedora Modular 31 - x86_64 - Updates                                                                                                                                                                                                              1.0 MB/s | 4.0 MB     00:04    
Fedora 31 - x86_64 - Updates                                                                                                                                                                                                                      5.8 MB/s |  22 MB     00:03    
Fedora 31 - x86_64                                                                                                                                                                                                                                 16 MB/s |  71 MB     00:04    
Last metadata expiration check: 0:00:01 ago on Tue Mar 10 17:32:38 2020.
Dependencies resolved.
==================================================================================================================================================================================================================================================================================
 Package                                                                Architecture                                                 Version                                                                  Repository                                                     Size
==================================================================================================================================================================================================================================================================================
Upgrading:
 glib2                                                                  x86_64                                                       2.62.5-1.fc31                                                            updates                                                       2.6 M
 libarchive                                                             x86_64                                                       3.4.2-1.fc31                                                             updates                                                       384 k
 libsss_idmap                                                           x86_64                                                       2.2.3-13.fc31                                                            updates                                                        46 k
 libsss_nss_idmap                                                       x86_64                                                       2.2.3-13.fc31                                                            updates                                                        53 k
 libtirpc                                                               x86_64                                                       1.2.5-1.rc2.fc31                                                         updates                                                        98 k
 libxcrypt                                                              x86_64                                                       4.4.15-1.fc31                                                            updates                                                       126 k
 pcre                                                                   x86_64                                                       8.43-3.fc31                                                              updates                                                       192 k
 pcre2                                                                  x86_64                                                       10.34-7.fc31                                                             updates                                                       261 k
 shadow-utils                                                           x86_64                                                       2:4.6-17.fc31                                                            updates                                                       1.2 M
 sssd-client                                                            x86_64                                                       2.2.3-13.fc31                                                            updates                                                       110 k
 sudo                                                                   x86_64                                                       1.9.0-0.1.b1.fc31                                                        updates                                                       1.0 M
 vim-minimal                                                            x86_64                                                       2:8.2.348-1.fc31                                                         updates                                                       645 k
Installing dependencies:
 whois-nls                                                              noarch                                                       5.5.6-1.fc31                                                             updates                                                        34 k
Installing weak dependencies:
 mkpasswd                                                               x86_64                                                       5.5.6-1.fc31                                                             updates                                                        41 k
 shared-mime-info                                                       x86_64                                                       1.15-1.fc31                                                              updates                                                       303 k

Transaction Summary
==================================================================================================================================================================================================================================================================================
Install   3 Packages
Upgrade  12 Packages

Total download size: 7.1 M
Downloading Packages:
(1/15): whois-nls-5.5.6-1.fc31.noarch.rpm                                                                                                                                                                                                          69 kB/s |  34 kB     00:00    
(2/15): mkpasswd-5.5.6-1.fc31.x86_64.rpm                                                                                                                                                                                                           77 kB/s |  41 kB     00:00    
(3/15): shared-mime-info-1.15-1.fc31.x86_64.rpm                                                                                                                                                                                                   410 kB/s | 303 kB     00:00    
(4/15): libsss_idmap-2.2.3-13.fc31.x86_64.rpm                                                                                                                                                                                                     530 kB/s |  46 kB     00:00    
(5/15): libarchive-3.4.2-1.fc31.x86_64.rpm                                                                                                                                                                                                        1.0 MB/s | 384 kB     00:00    
(6/15): libsss_nss_idmap-2.2.3-13.fc31.x86_64.rpm                                                                                                                                                                                                 600 kB/s |  53 kB     00:00    
(7/15): libtirpc-1.2.5-1.rc2.fc31.x86_64.rpm                                                                                                                                                                                                      977 kB/s |  98 kB     00:00    
(8/15): libxcrypt-4.4.15-1.fc31.x86_64.rpm                                                                                                                                                                                                        1.2 MB/s | 126 kB     00:00    
(9/15): glib2-2.62.5-1.fc31.x86_64.rpm                                                                                                                                                                                                            4.6 MB/s | 2.6 MB     00:00    
(10/15): pcre-8.43-3.fc31.x86_64.rpm                                                                                                                                                                                                              1.6 MB/s | 192 kB     00:00    
(11/15): pcre2-10.34-7.fc31.x86_64.rpm                                                                                                                                                                                                            1.9 MB/s | 261 kB     00:00    
(12/15): shadow-utils-4.6-17.fc31.x86_64.rpm                                                                                                                                                                                                       11 MB/s | 1.2 MB     00:00    
(13/15): sssd-client-2.2.3-13.fc31.x86_64.rpm                                                                                                                                                                                                     1.1 MB/s | 110 kB     00:00    
(14/15): vim-minimal-8.2.348-1.fc31.x86_64.rpm                                                                                                                                                                                                    6.4 MB/s | 645 kB     00:00    
(15/15): sudo-1.9.0-0.1.b1.fc31.x86_64.rpm                                                                                                                                                                                                        4.1 MB/s | 1.0 MB     00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                             4.1 MB/s | 7.1 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                          1/1 
  Upgrading        : vim-minimal-2:8.2.348-1.fc31.x86_64                                                                                                                                                                                                                     1/27 
  Upgrading        : pcre-8.43-3.fc31.x86_64                                                                                                                                                                                                                                 2/27 
  Upgrading        : glib2-2.62.5-1.fc31.x86_64                                                                                                                                                                                                                              3/27 
  Installing       : shared-mime-info-1.15-1.fc31.x86_64                                                                                                                                                                                                                     4/27 
  Running scriptlet: shared-mime-info-1.15-1.fc31.x86_64                                                                                                                                                                                                                     4/27 
  Upgrading        : libsss_nss_idmap-2.2.3-13.fc31.x86_64                                                                                                                                                                                                                   5/27 
  Upgrading        : libsss_idmap-2.2.3-13.fc31.x86_64                                                                                                                                                                                                                       6/27 
  Installing       : whois-nls-5.5.6-1.fc31.noarch                                                                                                                                                                                                                           7/27 
  Upgrading        : libxcrypt-4.4.15-1.fc31.x86_64                                                                                                                                                                                                                          8/27 
  Installing       : mkpasswd-5.5.6-1.fc31.x86_64                                                                                                                                                                                                                            9/27 
  Upgrading        : shadow-utils-2:4.6-17.fc31.x86_64                                                                                                                                                                                                                      10/27 
  Upgrading        : sssd-client-2.2.3-13.fc31.x86_64                                                                                                                                                                                                                       11/27 
  Running scriptlet: sssd-client-2.2.3-13.fc31.x86_64                                                                                                                                                                                                                       11/27 
  Upgrading        : sudo-1.9.0-0.1.b1.fc31.x86_64                                                                                                                                                                                                                          12/27 
  Running scriptlet: sudo-1.9.0-0.1.b1.fc31.x86_64                                                                                                                                                                                                                          12/27 
  Upgrading        : pcre2-10.34-7.fc31.x86_64                                                                                                                                                                                                                              13/27 
  Upgrading        : libtirpc-1.2.5-1.rc2.fc31.x86_64                                                                                                                                                                                                                       14/27 
  Upgrading        : libarchive-3.4.2-1.fc31.x86_64                                                                                                                                                                                                                         15/27 
  Running scriptlet: sssd-client-2.2.2-3.fc31.x86_64                                                                                                                                                                                                                        16/27 
  Cleanup          : sssd-client-2.2.2-3.fc31.x86_64                                                                                                                                                                                                                        16/27 
  Cleanup          : shadow-utils-2:4.6-16.fc31.x86_64                                                                                                                                                                                                                      17/27 
  Cleanup          : sudo-1.8.29-1.fc31.x86_64                                                                                                                                                                                                                              18/27 
  Cleanup          : glib2-2.62.4-2.fc31.x86_64                                                                                                                                                                                                                             19/27 
  Cleanup          : pcre-8.43-2.fc31.1.x86_64                                                                                                                                                                                                                              20/27 
  Cleanup          : vim-minimal-2:8.2.236-1.fc31.x86_64                                                                                                                                                                                                                    21/27 
  Cleanup          : libxcrypt-4.4.12-1.fc31.x86_64                                                                                                                                                                                                                         22/27 
  Cleanup          : libsss_idmap-2.2.2-3.fc31.x86_64                                                                                                                                                                                                                       23/27 
  Cleanup          : libsss_nss_idmap-2.2.2-3.fc31.x86_64                                                                                                                                                                                                                   24/27 
  Cleanup          : pcre2-10.34-5.fc31.x86_64                                                                                                                                                                                                                              25/27 
  Cleanup          : libtirpc-1.2.5-0.fc31.x86_64                                                                                                                                                                                                                           26/27 
  Cleanup          : libarchive-3.4.0-1.fc31.x86_64                                                                                                                                                                                                                         27/27 
  Running scriptlet: libarchive-3.4.0-1.fc31.x86_64                                                                                                                                                                                                                         27/27 
  Verifying        : mkpasswd-5.5.6-1.fc31.x86_64                                                                                                                                                                                                                            1/27 
  Verifying        : shared-mime-info-1.15-1.fc31.x86_64                                                                                                                                                                                                                     2/27 
  Verifying        : whois-nls-5.5.6-1.fc31.noarch                                                                                                                                                                                                                           3/27 
  Verifying        : glib2-2.62.5-1.fc31.x86_64                                                                                                                                                                                                                              4/27 
  Verifying        : glib2-2.62.4-2.fc31.x86_64                                                                                                                                                                                                                              5/27 
  Verifying        : libarchive-3.4.2-1.fc31.x86_64                                                                                                                                                                                                                          6/27 
  Verifying        : libarchive-3.4.0-1.fc31.x86_64                                                                                                                                                                                                                          7/27 
  Verifying        : libsss_idmap-2.2.3-13.fc31.x86_64                                                                                                                                                                                                                       8/27 
  Verifying        : libsss_idmap-2.2.2-3.fc31.x86_64                                                                                                                                                                                                                        9/27 
  Verifying        : libsss_nss_idmap-2.2.3-13.fc31.x86_64                                                                                                                                                                                                                  10/27 
  Verifying        : libsss_nss_idmap-2.2.2-3.fc31.x86_64                                                                                                                                                                                                                   11/27 
  Verifying        : libtirpc-1.2.5-1.rc2.fc31.x86_64                                                                                                                                                                                                                       12/27 
  Verifying        : libtirpc-1.2.5-0.fc31.x86_64                                                                                                                                                                                                                           13/27 
  Verifying        : libxcrypt-4.4.15-1.fc31.x86_64                                                                                                                                                                                                                         14/27 
  Verifying        : libxcrypt-4.4.12-1.fc31.x86_64                                                                                                                                                                                                                         15/27 
  Verifying        : pcre-8.43-3.fc31.x86_64                                                                                                                                                                                                                                16/27 
  Verifying        : pcre-8.43-2.fc31.1.x86_64                                                                                                                                                                                                                              17/27 
  Verifying        : pcre2-10.34-7.fc31.x86_64                                                                                                                                                                                                                              18/27 
  Verifying        : pcre2-10.34-5.fc31.x86_64                                                                                                                                                                                                                              19/27 
  Verifying        : shadow-utils-2:4.6-17.fc31.x86_64                                                                                                                                                                                                                      20/27 
  Verifying        : shadow-utils-2:4.6-16.fc31.x86_64                                                                                                                                                                                                                      21/27 
  Verifying        : sssd-client-2.2.3-13.fc31.x86_64                                                                                                                                                                                                                       22/27 
  Verifying        : sssd-client-2.2.2-3.fc31.x86_64                                                                                                                                                                                                                        23/27 
  Verifying        : sudo-1.9.0-0.1.b1.fc31.x86_64                                                                                                                                                                                                                          24/27 
  Verifying        : sudo-1.8.29-1.fc31.x86_64                                                                                                                                                                                                                              25/27 
  Verifying        : vim-minimal-2:8.2.348-1.fc31.x86_64                                                                                                                                                                                                                    26/27 
  Verifying        : vim-minimal-2:8.2.236-1.fc31.x86_64                                                                                                                                                                                                                    27/27 

Upgraded:
  glib2-2.62.5-1.fc31.x86_64          libarchive-3.4.2-1.fc31.x86_64     libsss_idmap-2.2.3-13.fc31.x86_64   libsss_nss_idmap-2.2.3-13.fc31.x86_64   libtirpc-1.2.5-1.rc2.fc31.x86_64   libxcrypt-4.4.15-1.fc31.x86_64   pcre-8.43-3.fc31.x86_64   pcre2-10.34-7.fc31.x86_64  
  shadow-utils-2:4.6-17.fc31.x86_64   sssd-client-2.2.3-13.fc31.x86_64   sudo-1.9.0-0.1.b1.fc31.x86_64       vim-minimal-2:8.2.348-1.fc31.x86_64    

Installed:
  mkpasswd-5.5.6-1.fc31.x86_64                                                            shared-mime-info-1.15-1.fc31.x86_64                                                            whois-nls-5.5.6-1.fc31.noarch                                                           

Complete!

So, seems that the issue is related to the host kernel not being up-to-date enough so overlayfs2 could support cap_set_file.

What's the best way to proceed? I'm not sure. I'd first try to update the host kernel of the machines where you see the failure. If that doesn't help, maybe switch to using devicemapper only for installing packages may be a reasonable solution.

@GabyCT, does this (very long) answer make sense?

alicefr commented 4 years ago

@fidencio @GabyCT we have the same issue on s390x. With devicemapper it works. I moved to a more recent kernel 5.3.0-42-generic, but I still have the issue

alicefr commented 4 years ago

I might be wrong. But are we missing the cap_set_file in the guest kernel?

fidencio commented 4 years ago

I might be wrong. But are we missing the cap_set_file in the guest kernel?

I thought it was related to the host kernel. But, yes, it may be related to the guest kernel indeed. Out of curiosity. which host / guest kernel are you using, @alicefr?

alicefr commented 4 years ago

@fidencio I guess is the guest kernel because the same command works with a standard container on my host. My host is 5.3.0-42-generic and my guest is kernel is 5.4.15-71. However, I'm running on s390x and my guest kernel has been compiled with these configs https://github.com/kata-containers/packaging/pull/994. So, I might be missing the correct configs to use this capability

fidencio commented 4 years ago

@alicefr, shouldn't CONFIG_TMPFS_XATTR=y be enough to have it working? I'm far from being familiar with kernel-land, so, please, bear with me here. :-)

alicefr commented 4 years ago

@fidencio not sure. It's enabled in all the arch. So, for both case x86 and s390x. Which kernel config is your guest using?

fidencio commented 4 years ago

@alicefr, I didn't hit the issue myself, was only able to reproduce this on @GabyCT's environment, so I'll leave it to her to reply.

I'd assume whatever is the upstream that kata is using.

On my personal environment (where I haven't hit this issue), I'm playing with both host and kernel using the very same Fedora kernel.

alicefr commented 4 years ago

@fidencio I tried with fedora 31 and kata from fedora and I failed with the same error. Could you try with this command:

$ podman run  --cap-add all --security-opt label=disable -ti --runtime /usr/bin/kata-runtime fedora sh -c  'getcap -v /bin/sleep'
Failed to get capabilities of file `/bin/sleep' (Operation not supported)
fidencio commented 4 years ago

@alicefr, indeed. I can reproduce this on Fedora32 / Rawhide as well. I'm adding this to my short-term list.

alicefr commented 4 years ago

ok, my guess and I can be totally wrong is that 9p doesn't support the operation or I'm missing a config. Here my test: I changed the storage driver to devicemapper and I tried this:

$ mkdir test 
$ touch test/get-cap-test
$ docker run -v $(pwd)/test:/test --cap-add all  -ti --runtime kata-runtime fedora sh
mount 
/dev/sda on / type ext4 (rw,relatime,stripe=16)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,size=1021760k,nr_inodes=255440,mode=755)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (ro,nosuid,nodev,noexec,relatime,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,nr_inodes=255440)
kataShared on /test type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
kataShared on /etc/resolv.conf type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
kataShared on /etc/hostname type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
kataShared on /etc/hosts type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/latency_stats type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime,size=1021760k,nr_inodes=255440)
tmpfs on /sys/firmware type tmpfs (ro,relatime,size=1021760k,nr_inodes=255440)
sh-5.0# getcap bin/sleep
sh-5.0# getcap test/get-cap-test
Failed to get capabilities of file `test/get-cap-test' (Operation not supported)
fidencio commented 4 years ago

Interesting.

@alicefr, I've reproduced this using virtiofs instead of 9p. David Gilbert kindly pointed out that passing "-o xattr" to virtiofsd solves the issue when using virtiofs.

Based on that, I do assume your guess ir correct!

fidencio commented 4 years ago

Also, David also pointed that it seems to be "outside of kata" as he could reproduce that with just a virtiofs mount.

alicefr commented 4 years ago

yes, probably. Not sure how to do that for 9p

grahamwhaley commented 4 years ago

Sorry to be rather late to this - there is a common known issue with 9p to do with unlinked tmp files, that in the past has stopped things like dnf, apt etc. from working. There are two ways to try and track this down to (dis)prove that is the problem.... 1) use strace to see what failed, and can you trace it back to an open/unlink operation. 2) move the TMPDIR to a ramfs in the container, and see if that fixes it...

(we have done the TMPDIR hack using the shmem dir before - see https://github.com/kata-containers/tests/blob/master/metrics/network/network-metrics-iperf3.sh#L40

alicefr commented 4 years ago

From my preview examples, here the strace output:

sh-5.0# strace getcap get-cap-test 
execve("/usr/sbin/getcap", ["getcap", "get-cap-test"], 0x3ffde47f0a8 /* 11 vars */) = 0
brk(NULL)                               = 0x2aa1374a000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3ff9c6fb000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=12092, ...}) = 0
mmap(NULL, 12092, PROT_READ, MAP_PRIVATE, 3, 0) = 0x3ff9c600000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\2\1\0\0\0\0\0\0\0\0\0\0\3\0\26\0\0\0\1\0\0\0\0\0\0\27@"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=24848, ...}) = 0
mmap(NULL, 24888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ff9c580000
mmap(0x3ff9c585000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x3ff9c585000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\2\1\3\0\0\0\0\0\0\0\0\0\3\0\26\0\0\0\1\0\0\0\0\0\2\267P"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=3093208, ...}) = 0
mmap(NULL, 1736592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ff9c380000
mmap(0x3ff9c51e000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19d000) = 0x3ff9c51e000
mmap(0x3ff9c525000, 12176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3ff9c525000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3ff9c6f9000
mprotect(0x3ff9c51e000, 16384, PROT_READ) = 0
mprotect(0x3ff9c585000, 4096, PROT_READ) = 0
mprotect(0x2aa12082000, 4096, PROT_READ) = 0
mprotect(0x3ff9c6a6000, 4096, PROT_READ) = 0
munmap(0x3ff9c600000, 12092)            = 0
lstat("get-cap-test", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
brk(NULL)                               = 0x2aa1374a000
brk(0x2aa1376b000)                      = 0x2aa1376b000
brk(NULL)                               = 0x2aa1376b000
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
getxattr("get-cap-test", "security.capability", 0x3ffd847ea88, 24) = -1 EOPNOTSUPP (Operation not supported)
write(2, "Failed to get capabilities of fi"..., 76Failed to get capabilities of file `get-cap-test' (Operation not supported)
) = 76
exit_group(0)                           = ?
+++ exited with 0 +++
grahamwhaley commented 4 years ago

Ah, OK, I see the getxattr there now - so, ignore my previous comment ;-) As David was mentioned etc., /cc @dagrh

alicefr commented 4 years ago

So, as far as I can see from qemu code we get ENOTSUP from this function https://github.com/qemu/qemu/blob/master/hw/9pfs/9p-xattr.c#L242 and https://github.com/qemu/qemu/blob/2833ad487cfff7dc33703e4731b75facde1c561e/hw/9pfs/9p-posix-acl.c#L150 that is used if the security model is none https://github.com/qemu/qemu/blob/master/hw/9pfs/9p-xattr.c#L282 . It's what we have in Kata for 9p https://github.com/kata-containers/runtime/blob/705713b4f9bc4d1e754871d5ef1ab5e99ea71aff/virtcontainers/qemu_arch_base.go#L467

dagrh commented 4 years ago

So the -o xattr on virtiofsd is probably equivalent to security_model=passthrough on 9p - we don't do any translation.

alicefr commented 4 years ago

@dagrh I tried to change the security model to passthrough, but I still get EOPNOTSUPP :

git diff
diff --git a/virtcontainers/qemu_arch_base.go b/virtcontainers/qemu_arch_base.go
index 9d72dd09..3632e481 100644
--- a/virtcontainers/qemu_arch_base.go
+++ b/virtcontainers/qemu_arch_base.go
@@ -464,7 +464,7 @@ func generic9PVolume(volume types.Volume, nestedRun bool) govmmQemu.FSDevice {
                ID:            devID,
                Path:          volume.HostPath,
                MountTag:      volume.MountTag,
-               SecurityModel: govmmQemu.None,
+               SecurityModel: govmmQemu.PassThrough,
                DisableModern: nestedRun,
        }
 }

at least on s390x I have this behavior. Can somebody check on x86?

amshinde commented 4 years ago

Looks like the dnf update tests have been failing with virtio-fs for a while: https://github.com/kata-containers/tests/issues/2008

The recent failures have been with 9p. Something has changed to cause these failures. @GabyCT @jcvenegas Can we do a bisect to figure out when this test started failing with 9p?

alicefr commented 4 years ago

@amshinde it could be due to a change in the fedora image?

amshinde commented 4 years ago

@alicefr Yes, that could be a possible reason. Might be worth testing this with a fedora image with an older tag from a month back. @GabyCT @jcvenegas

GabyCT commented 4 years ago

@amshinde , so I did a checkout to several tags 1.9.2, 1.9.0, 1.8.5, 1.8.0 and it is not possible to perform dnf -y update, I even tried to performed the checkout with 1.7.0 and 1.7.4, however, when I tried to run the container and then I got the following

docker: Error response from daemon: OCI runtime create failed: /usr/share/defaults/kata-containers/configuration-qemu.toml: file /usr/bin/qemu-lite-system-x86_64 does not exist: unknown.

/cc @jcvenegas

fidencio commented 4 years ago

@GabyCT, if the problem is still the one about shadow-utils, I don't think a different version of podman would help.

What could help is a new fedora image, which already have the shadow-utils updated as part of the image.