Closed GabyCT closed 3 years ago
We are running the following code
$ docker run -ti --runtime kata-runtime Fedora sh
$ dnf -y update
This is also present on ubuntu
any idea of why this is happening @fidencio? thanks
Any chance to get the logs from /var/log/dnf.log
(on the container)? It'd give us a better idea of what may be the causing the issue.
2020-03-10T15:42:47Z INFO --- logging initialized ---
2020-03-10T15:42:47Z DDEBUG timer: config: 133 ms
2020-03-10T15:42:47Z DEBUG DNF version: 4.2.18
2020-03-10T15:42:47Z DDEBUG Command: dnf -y update
2020-03-10T15:42:47Z DDEBUG Installroot: /
2020-03-10T15:42:47Z DDEBUG Releasever: 31
2020-03-10T15:42:47Z DEBUG cachedir: /var/cache/dnf
2020-03-10T15:42:47Z DDEBUG Base command: update
2020-03-10T15:42:47Z DDEBUG Extra commands: ['-y', 'update']
2020-03-10T15:42:47Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2020-03-10T15:42:47Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2020-03-10T15:42:47Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
2020-03-10T15:42:47Z DEBUG User-Agent: constructed: 'libdnf (Fedora 31; container; Linux.x86_64)'
2020-03-10T15:42:47Z DEBUG repo: downloading from remote: fedora-modular
2020-03-10T15:42:52Z DEBUG fedora-modular: using metadata from Wed Oct 23 22:53:13 2019.
2020-03-10T15:42:52Z DEBUG repo: downloading from remote: updates-modular
2020-03-10T15:43:01Z DEBUG updates-modular: using metadata from Fri Mar 6 22:30:57 2020.
2020-03-10T15:43:01Z DEBUG repo: downloading from remote: updates
2020-03-10T15:43:15Z DEBUG updates: using metadata from Mon Mar 9 21:47:41 2020.
2020-03-10T15:43:15Z DEBUG repo: downloading from remote: fedora
2020-03-10T15:44:54Z DEBUG fedora: using metadata from Wed Oct 23 22:52:47 2019.
2020-03-10T15:44:54Z INFO Last metadata expiration check: 0:00:02 ago on Tue Mar 10 15:43:55 2020.
2020-03-10T15:45:00Z DDEBUG timer: sack setup: 133143 ms
2020-03-10T15:45:00Z DEBUG --> Starting dependency resolution
2020-03-10T15:45:02Z DEBUG ---> Package mkpasswd.x86_64 5.5.6-1.fc31 will be installed
2020-03-10T15:45:02Z DEBUG ---> Package shared-mime-info.x86_64 1.15-1.fc31 will be installed
2020-03-10T15:45:02Z DEBUG ---> Package whois-nls.noarch 5.5.6-1.fc31 will be installed
2020-03-10T15:45:02Z DEBUG ---> Package glib2.x86_64 2.62.4-2.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package glib2.x86_64 2.62.5-1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libarchive.x86_64 3.4.0-1.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libarchive.x86_64 3.4.2-1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libsss_idmap.x86_64 2.2.2-3.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libsss_idmap.x86_64 2.2.3-13.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libsss_nss_idmap.x86_64 2.2.2-3.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libsss_nss_idmap.x86_64 2.2.3-13.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libtirpc.x86_64 1.2.5-0.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libtirpc.x86_64 1.2.5-1.rc2.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package libxcrypt.x86_64 4.4.12-1.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package libxcrypt.x86_64 4.4.15-1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package pcre.x86_64 8.43-2.fc31.1 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package pcre.x86_64 8.43-3.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package pcre2.x86_64 10.34-5.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package pcre2.x86_64 10.34-7.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package shadow-utils.x86_64 2:4.6-16.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package shadow-utils.x86_64 2:4.6-17.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package sssd-client.x86_64 2.2.2-3.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package sssd-client.x86_64 2.2.3-13.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package sudo.x86_64 1.8.29-1.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package sudo.x86_64 1.9.0-0.1.b1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG ---> Package vim-minimal.x86_64 2:8.2.236-1.fc31 will be upgraded
2020-03-10T15:45:02Z DEBUG ---> Package vim-minimal.x86_64 2:8.2.348-1.fc31 will be an upgrade
2020-03-10T15:45:02Z DEBUG --> Finished dependency resolution
2020-03-10T15:45:02Z DDEBUG timer: depsolve: 2097 ms
2020-03-10T15:45:02Z INFO Dependencies resolved.
2020-03-10T15:45:04Z INFO =============================================================================================================================================================================================
Package Architecture Version Repository Size
=============================================================================================================================================================================================
Upgrading:
glib2 x86_64 2.62.5-1.fc31 updates 2.6 M
libarchive x86_64 3.4.2-1.fc31 updates 384 k
libsss_idmap x86_64 2.2.3-13.fc31 updates 46 k
libsss_nss_idmap x86_64 2.2.3-13.fc31 updates 53 k
libtirpc x86_64 1.2.5-1.rc2.fc31 updates 98 k
libxcrypt x86_64 4.4.15-1.fc31 updates 126 k
pcre x86_64 8.43-3.fc31 updates 192 k
pcre2 x86_64 10.34-7.fc31 updates 261 k
shadow-utils x86_64 2:4.6-17.fc31 updates 1.2 M
sssd-client x86_64 2.2.3-13.fc31 updates 110 k
sudo x86_64 1.9.0-0.1.b1.fc31 updates 1.0 M
vim-minimal x86_64 2:8.2.348-1.fc31 updates 645 k
Installing dependencies:
whois-nls noarch 5.5.6-1.fc31 updates 34 k
Installing weak dependencies:
mkpasswd x86_64 5.5.6-1.fc31 updates 41 k
shared-mime-info x86_64 1.15-1.fc31 updates 303 k
Transaction Summary
=============================================================================================================================================================================================
Install 3 Packages
Upgrade 12 Packages
2020-03-10T15:45:04Z INFO Total download size: 7.1 M
2020-03-10T15:45:04Z INFO Downloading Packages:
2020-03-10T15:45:07Z INFO ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2020-03-10T15:45:07Z INFO Total 2.7 MB/s | 7.1 MB 00:02
2020-03-10T15:45:09Z INFO Running transaction check
2020-03-10T15:45:09Z INFO Transaction check succeeded.
2020-03-10T15:45:09Z INFO Running transaction test
2020-03-10T15:45:10Z INFO Transaction test succeeded.
2020-03-10T15:45:10Z DDEBUG timer: transaction test: 905 ms
2020-03-10T15:45:10Z INFO Running transaction
2020-03-10T15:45:11Z DDEBUG RPM transaction start.
2020-03-10T15:45:38Z DDEBUG RPM transaction over.
2020-03-10T15:45:38Z DEBUG Errors occurred during transaction.
2020-03-10T15:45:39Z DDEBUG timer: verify transaction: 822 ms
2020-03-10T15:45:39Z DDEBUG timer: transaction: 28969 ms
2020-03-10T15:45:41Z INFO
Upgraded:
glib2-2.62.5-1.fc31.x86_64 libarchive-3.4.2-1.fc31.x86_64 libsss_idmap-2.2.3-13.fc31.x86_64 libsss_nss_idmap-2.2.3-13.fc31.x86_64 libtirpc-1.2.5-1.rc2.fc31.x86_64
libxcrypt-4.4.15-1.fc31.x86_64 pcre-8.43-3.fc31.x86_64 pcre2-10.34-7.fc31.x86_64 sssd-client-2.2.3-13.fc31.x86_64 sudo-1.9.0-0.1.b1.fc31.x86_64
vim-minimal-2:8.2.348-1.fc31.x86_64
Installed:
mkpasswd-5.5.6-1.fc31.x86_64 shared-mime-info-1.15-1.fc31.x86_64 whois-nls-5.5.6-1.fc31.noarch
Failed:
shadow-utils-2:4.6-16.fc31.x86_64 shadow-utils-2:4.6-17.fc31.x86_64
2020-03-10T15:45:41Z DDEBUG Cleaning up.
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/whois-nls-5.5.6-1.fc31.noarch.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libarchive-3.4.2-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libsss_nss_idmap-2.2.3-13.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/mkpasswd-5.5.6-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/pcre-8.43-3.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/shadow-utils-4.6-17.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/sudo-1.9.0-0.1.b1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libsss_idmap-2.2.3-13.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/shared-mime-info-1.15-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/glib2-2.62.5-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/vim-minimal-8.2.348-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libxcrypt-4.4.15-1.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/sssd-client-2.2.3-13.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/libtirpc-1.2.5-1.rc2.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z DDEBUG /var/cache/dnf/updates-7fc4c739b3909d9f/packages/pcre2-10.34-7.fc31.x86_64.rpm removed
2020-03-10T15:45:41Z SUBDEBUG
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 65, in main
return _main(base, args, cli_class, option_parser_class)
File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 98, in _main
return cli_run(cli, base)
File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 122, in cli_run
ret = resolving(cli, base)
File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 166, in resolving
base.do_transaction(display=displays)
File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 257, in do_transaction
raise dnf.exceptions.Error(_('Transaction failed'))
dnf.exceptions.Error: Transaction failed
2020-03-10T15:45:41Z CRITICAL Error: Transaction failed
Here it is @fidencio
Firstly, thanks @GabyCT for providing me a machine where I could test that. Here's what I did in order to try to reproduce your issue:
gabyopen@gabyopen:~> sudo docker run -ti --runtime kata-runtime fedora sh
sh-5.0# dnf -y update
Fedora Modular 31 - x86_64 2.2 MB/s | 5.2 MB 00:02
Fedora Modular 31 - x86_64 - Updates 1.6 MB/s | 4.0 MB 00:02
Fedora 31 - x86_64 - Updates 5.4 MB/s | 22 MB 00:03
Fedora 31 - x86_64 5.1 MB/s | 71 MB 00:13
Last metadata expiration check: 0:00:01 ago on Tue Mar 10 16:59:06 2020.
Dependencies resolved.
========================================================================================================================================
Package Architecture Version Repository Size
========================================================================================================================================
Upgrading:
glib2 x86_64 2.62.5-1.fc31 updates 2.6 M
libarchive x86_64 3.4.2-1.fc31 updates 384 k
libsss_idmap x86_64 2.2.3-13.fc31 updates 46 k
libsss_nss_idmap x86_64 2.2.3-13.fc31 updates 53 k
libtirpc x86_64 1.2.5-1.rc2.fc31 updates 98 k
libxcrypt x86_64 4.4.15-1.fc31 updates 126 k
pcre x86_64 8.43-3.fc31 updates 192 k
pcre2 x86_64 10.34-7.fc31 updates 261 k
shadow-utils x86_64 2:4.6-17.fc31 updates 1.2 M
sssd-client x86_64 2.2.3-13.fc31 updates 110 k
sudo x86_64 1.9.0-0.1.b1.fc31 updates 1.0 M
vim-minimal x86_64 2:8.2.348-1.fc31 updates 645 k
Installing dependencies:
whois-nls noarch 5.5.6-1.fc31 updates 34 k
Installing weak dependencies:
mkpasswd x86_64 5.5.6-1.fc31 updates 41 k
shared-mime-info x86_64 1.15-1.fc31 updates 303 k
Transaction Summary
========================================================================================================================================
Install 3 Packages
Upgrade 12 Packages
Total download size: 7.1 M
Downloading Packages:
(1/15): whois-nls-5.5.6-1.fc31.noarch.rpm 74 kB/s | 34 kB 00:00
(2/15): mkpasswd-5.5.6-1.fc31.x86_64.rpm 84 kB/s | 41 kB 00:00
(3/15): shared-mime-info-1.15-1.fc31.x86_64.rpm 427 kB/s | 303 kB 00:00
(4/15): libarchive-3.4.2-1.fc31.x86_64.rpm 991 kB/s | 384 kB 00:00
(5/15): libsss_idmap-2.2.3-13.fc31.x86_64.rpm 156 kB/s | 46 kB 00:00
(6/15): glib2-2.62.5-1.fc31.x86_64.rpm 3.3 MB/s | 2.6 MB 00:00
(7/15): libsss_nss_idmap-2.2.3-13.fc31.x86_64.rpm 147 kB/s | 53 kB 00:00
(8/15): libtirpc-1.2.5-1.rc2.fc31.x86_64.rpm 398 kB/s | 98 kB 00:00
(9/15): libxcrypt-4.4.15-1.fc31.x86_64.rpm 1.1 MB/s | 126 kB 00:00
(10/15): pcre-8.43-3.fc31.x86_64.rpm 1.3 MB/s | 192 kB 00:00
(11/15): pcre2-10.34-7.fc31.x86_64.rpm 1.3 MB/s | 261 kB 00:00
(12/15): shadow-utils-4.6-17.fc31.x86_64.rpm 2.9 MB/s | 1.2 MB 00:00
(13/15): sssd-client-2.2.3-13.fc31.x86_64.rpm 281 kB/s | 110 kB 00:00
(14/15): sudo-1.9.0-0.1.b1.fc31.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00
(15/15): vim-minimal-8.2.348-1.fc31.x86_64.rpm 2.4 MB/s | 645 kB 00:00
----------------------------------------------------------------------------------------------------------------------------------------
Total 2.8 MB/s | 7.1 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : vim-minimal-2:8.2.348-1.fc31.x86_64 1/27
Upgrading : pcre-8.43-3.fc31.x86_64 2/27
Upgrading : glib2-2.62.5-1.fc31.x86_64 3/27
Installing : shared-mime-info-1.15-1.fc31.x86_64 4/27
Running scriptlet: shared-mime-info-1.15-1.fc31.x86_64 4/27
Upgrading : libsss_nss_idmap-2.2.3-13.fc31.x86_64 5/27
Upgrading : libsss_idmap-2.2.3-13.fc31.x86_64 6/27
Installing : whois-nls-5.5.6-1.fc31.noarch 7/27
Upgrading : libxcrypt-4.4.15-1.fc31.x86_64 8/27
Installing : mkpasswd-5.5.6-1.fc31.x86_64 9/27
Upgrading : shadow-utils-2:4.6-17.fc31.x86_64 10/27
Error unpacking rpm package shadow-utils-2:4.6-17.fc31.x86_64
Upgrading : sssd-client-2.2.3-13.fc31.x86_64 11/27
error: unpacking of archive failed on file /usr/bin/newgidmap;5e67c78a: cpio: cap_set_file
error: shadow-utils-2:4.6-17.fc31.x86_64: install failed
Running scriptlet: sssd-client-2.2.3-13.fc31.x86_64 11/27
Upgrading : sudo-1.9.0-0.1.b1.fc31.x86_64 12/27
Running scriptlet: sudo-1.9.0-0.1.b1.fc31.x86_64 12/27
Upgrading : pcre2-10.34-7.fc31.x86_64 13/27
Upgrading : libtirpc-1.2.5-1.rc2.fc31.x86_64 14/27
Upgrading : libarchive-3.4.2-1.fc31.x86_64 15/27
Running scriptlet: sssd-client-2.2.2-3.fc31.x86_64 16/27
Cleanup : sssd-client-2.2.2-3.fc31.x86_64 16/27
Cleanup : sudo-1.8.29-1.fc31.x86_64 17/27
error: shadow-utils-2:4.6-16.fc31.x86_64: erase skipped
Cleanup : glib2-2.62.4-2.fc31.x86_64 18/27
Cleanup : pcre-8.43-2.fc31.1.x86_64 19/27
Cleanup : vim-minimal-2:8.2.236-1.fc31.x86_64 20/27
Cleanup : libxcrypt-4.4.12-1.fc31.x86_64 21/27
Cleanup : libsss_idmap-2.2.2-3.fc31.x86_64 22/27
Cleanup : libsss_nss_idmap-2.2.2-3.fc31.x86_64 23/27
Cleanup : pcre2-10.34-5.fc31.x86_64 24/27
Cleanup : libtirpc-1.2.5-0.fc31.x86_64 25/27
Cleanup : libarchive-3.4.0-1.fc31.x86_64 26/27
Running scriptlet: libarchive-3.4.0-1.fc31.x86_64 26/27
Verifying : mkpasswd-5.5.6-1.fc31.x86_64 1/27
Verifying : shared-mime-info-1.15-1.fc31.x86_64 2/27
Verifying : whois-nls-5.5.6-1.fc31.noarch 3/27
Verifying : glib2-2.62.5-1.fc31.x86_64 4/27
Verifying : glib2-2.62.4-2.fc31.x86_64 5/27
Verifying : libarchive-3.4.2-1.fc31.x86_64 6/27
Verifying : libarchive-3.4.0-1.fc31.x86_64 7/27
Verifying : libsss_idmap-2.2.3-13.fc31.x86_64 8/27
Verifying : libsss_idmap-2.2.2-3.fc31.x86_64 9/27
Verifying : libsss_nss_idmap-2.2.3-13.fc31.x86_64 10/27
Verifying : libsss_nss_idmap-2.2.2-3.fc31.x86_64 11/27
Verifying : libtirpc-1.2.5-1.rc2.fc31.x86_64 12/27
Verifying : libtirpc-1.2.5-0.fc31.x86_64 13/27
Verifying : libxcrypt-4.4.15-1.fc31.x86_64 14/27
Verifying : libxcrypt-4.4.12-1.fc31.x86_64 15/27
Verifying : pcre-8.43-3.fc31.x86_64 16/27
Verifying : pcre-8.43-2.fc31.1.x86_64 17/27
Verifying : pcre2-10.34-7.fc31.x86_64 18/27
Verifying : pcre2-10.34-5.fc31.x86_64 19/27
Verifying : shadow-utils-2:4.6-17.fc31.x86_64 20/27
Verifying : shadow-utils-2:4.6-16.fc31.x86_64 21/27
Verifying : sssd-client-2.2.3-13.fc31.x86_64 22/27
Verifying : sssd-client-2.2.2-3.fc31.x86_64 23/27
Verifying : sudo-1.9.0-0.1.b1.fc31.x86_64 24/27
Verifying : sudo-1.8.29-1.fc31.x86_64 25/27
Verifying : vim-minimal-2:8.2.348-1.fc31.x86_64 26/27
Verifying : vim-minimal-2:8.2.236-1.fc31.x86_64 27/27
Upgraded:
glib2-2.62.5-1.fc31.x86_64 libarchive-3.4.2-1.fc31.x86_64 libsss_idmap-2.2.3-13.fc31.x86_64
libsss_nss_idmap-2.2.3-13.fc31.x86_64 libtirpc-1.2.5-1.rc2.fc31.x86_64 libxcrypt-4.4.15-1.fc31.x86_64
pcre-8.43-3.fc31.x86_64 pcre2-10.34-7.fc31.x86_64 sssd-client-2.2.3-13.fc31.x86_64
sudo-1.9.0-0.1.b1.fc31.x86_64 vim-minimal-2:8.2.348-1.fc31.x86_64
Installed:
mkpasswd-5.5.6-1.fc31.x86_64 shared-mime-info-1.15-1.fc31.x86_64 whois-nls-5.5.6-1.fc31.noarch
Failed:
shadow-utils-2:4.6-16.fc31.x86_64 shadow-utils-2:4.6-17.fc31.x86_64
Error: Transaction failed
The important part of the error is: error: unpacking of archive failed on file /usr/bin/newgidmap;5e67c78a: cpio: cap_set_file
, which seems to be possibly fixed by: https://github.com/torvalds/linux/commit/8db6c34f1dbc8e06aa016a9b829b06902c3e1340.
Mind, though, that this is a guess.
Right now I'm dropping this info here and later on I'll try to force docker to use devicemapper, which although quite slow, may be a reasonable workaround.
Okay, I've done:
gabyopen@gabyopen:~> sudo cat /etc/systemd/system/docker.service.d/kata-containers.conf
[Service]
Environment=""
Environment=""
ExecStart=
#ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/local/bin/kata-runtime --default-runtime=runc --storage-driver=overlay2
ExecStart=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/local/bin/kata-runtime --default-runtime=runc --storage-driver=devicemapper
gabyopen@gabyopen:~> sudo systemctl daemon-reload
gabyopen@gabyopen:~> sudo systemctl restart docker
gabyopen@gabyopen:~> sudo docker run -ti --runtime kata-runtime fedora sh
Unable to find image 'fedora:latest' locally
latest: Pulling from library/fedora
5c1b9e8d7bf7: Pull complete
Digest: sha256:c97879f8bebe49744307ea5c77ffc76c7cc97f3ddec72fb9a394bd4e4519b388
Status: Downloaded newer image for fedora:latest
sh-5.0# dnf -y update
Fedora Modular 31 - x86_64 3.3 MB/s | 5.2 MB 00:01
Fedora Modular 31 - x86_64 - Updates 1.0 MB/s | 4.0 MB 00:04
Fedora 31 - x86_64 - Updates 5.8 MB/s | 22 MB 00:03
Fedora 31 - x86_64 16 MB/s | 71 MB 00:04
Last metadata expiration check: 0:00:01 ago on Tue Mar 10 17:32:38 2020.
Dependencies resolved.
==================================================================================================================================================================================================================================================================================
Package Architecture Version Repository Size
==================================================================================================================================================================================================================================================================================
Upgrading:
glib2 x86_64 2.62.5-1.fc31 updates 2.6 M
libarchive x86_64 3.4.2-1.fc31 updates 384 k
libsss_idmap x86_64 2.2.3-13.fc31 updates 46 k
libsss_nss_idmap x86_64 2.2.3-13.fc31 updates 53 k
libtirpc x86_64 1.2.5-1.rc2.fc31 updates 98 k
libxcrypt x86_64 4.4.15-1.fc31 updates 126 k
pcre x86_64 8.43-3.fc31 updates 192 k
pcre2 x86_64 10.34-7.fc31 updates 261 k
shadow-utils x86_64 2:4.6-17.fc31 updates 1.2 M
sssd-client x86_64 2.2.3-13.fc31 updates 110 k
sudo x86_64 1.9.0-0.1.b1.fc31 updates 1.0 M
vim-minimal x86_64 2:8.2.348-1.fc31 updates 645 k
Installing dependencies:
whois-nls noarch 5.5.6-1.fc31 updates 34 k
Installing weak dependencies:
mkpasswd x86_64 5.5.6-1.fc31 updates 41 k
shared-mime-info x86_64 1.15-1.fc31 updates 303 k
Transaction Summary
==================================================================================================================================================================================================================================================================================
Install 3 Packages
Upgrade 12 Packages
Total download size: 7.1 M
Downloading Packages:
(1/15): whois-nls-5.5.6-1.fc31.noarch.rpm 69 kB/s | 34 kB 00:00
(2/15): mkpasswd-5.5.6-1.fc31.x86_64.rpm 77 kB/s | 41 kB 00:00
(3/15): shared-mime-info-1.15-1.fc31.x86_64.rpm 410 kB/s | 303 kB 00:00
(4/15): libsss_idmap-2.2.3-13.fc31.x86_64.rpm 530 kB/s | 46 kB 00:00
(5/15): libarchive-3.4.2-1.fc31.x86_64.rpm 1.0 MB/s | 384 kB 00:00
(6/15): libsss_nss_idmap-2.2.3-13.fc31.x86_64.rpm 600 kB/s | 53 kB 00:00
(7/15): libtirpc-1.2.5-1.rc2.fc31.x86_64.rpm 977 kB/s | 98 kB 00:00
(8/15): libxcrypt-4.4.15-1.fc31.x86_64.rpm 1.2 MB/s | 126 kB 00:00
(9/15): glib2-2.62.5-1.fc31.x86_64.rpm 4.6 MB/s | 2.6 MB 00:00
(10/15): pcre-8.43-3.fc31.x86_64.rpm 1.6 MB/s | 192 kB 00:00
(11/15): pcre2-10.34-7.fc31.x86_64.rpm 1.9 MB/s | 261 kB 00:00
(12/15): shadow-utils-4.6-17.fc31.x86_64.rpm 11 MB/s | 1.2 MB 00:00
(13/15): sssd-client-2.2.3-13.fc31.x86_64.rpm 1.1 MB/s | 110 kB 00:00
(14/15): vim-minimal-8.2.348-1.fc31.x86_64.rpm 6.4 MB/s | 645 kB 00:00
(15/15): sudo-1.9.0-0.1.b1.fc31.x86_64.rpm 4.1 MB/s | 1.0 MB 00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 4.1 MB/s | 7.1 MB 00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : vim-minimal-2:8.2.348-1.fc31.x86_64 1/27
Upgrading : pcre-8.43-3.fc31.x86_64 2/27
Upgrading : glib2-2.62.5-1.fc31.x86_64 3/27
Installing : shared-mime-info-1.15-1.fc31.x86_64 4/27
Running scriptlet: shared-mime-info-1.15-1.fc31.x86_64 4/27
Upgrading : libsss_nss_idmap-2.2.3-13.fc31.x86_64 5/27
Upgrading : libsss_idmap-2.2.3-13.fc31.x86_64 6/27
Installing : whois-nls-5.5.6-1.fc31.noarch 7/27
Upgrading : libxcrypt-4.4.15-1.fc31.x86_64 8/27
Installing : mkpasswd-5.5.6-1.fc31.x86_64 9/27
Upgrading : shadow-utils-2:4.6-17.fc31.x86_64 10/27
Upgrading : sssd-client-2.2.3-13.fc31.x86_64 11/27
Running scriptlet: sssd-client-2.2.3-13.fc31.x86_64 11/27
Upgrading : sudo-1.9.0-0.1.b1.fc31.x86_64 12/27
Running scriptlet: sudo-1.9.0-0.1.b1.fc31.x86_64 12/27
Upgrading : pcre2-10.34-7.fc31.x86_64 13/27
Upgrading : libtirpc-1.2.5-1.rc2.fc31.x86_64 14/27
Upgrading : libarchive-3.4.2-1.fc31.x86_64 15/27
Running scriptlet: sssd-client-2.2.2-3.fc31.x86_64 16/27
Cleanup : sssd-client-2.2.2-3.fc31.x86_64 16/27
Cleanup : shadow-utils-2:4.6-16.fc31.x86_64 17/27
Cleanup : sudo-1.8.29-1.fc31.x86_64 18/27
Cleanup : glib2-2.62.4-2.fc31.x86_64 19/27
Cleanup : pcre-8.43-2.fc31.1.x86_64 20/27
Cleanup : vim-minimal-2:8.2.236-1.fc31.x86_64 21/27
Cleanup : libxcrypt-4.4.12-1.fc31.x86_64 22/27
Cleanup : libsss_idmap-2.2.2-3.fc31.x86_64 23/27
Cleanup : libsss_nss_idmap-2.2.2-3.fc31.x86_64 24/27
Cleanup : pcre2-10.34-5.fc31.x86_64 25/27
Cleanup : libtirpc-1.2.5-0.fc31.x86_64 26/27
Cleanup : libarchive-3.4.0-1.fc31.x86_64 27/27
Running scriptlet: libarchive-3.4.0-1.fc31.x86_64 27/27
Verifying : mkpasswd-5.5.6-1.fc31.x86_64 1/27
Verifying : shared-mime-info-1.15-1.fc31.x86_64 2/27
Verifying : whois-nls-5.5.6-1.fc31.noarch 3/27
Verifying : glib2-2.62.5-1.fc31.x86_64 4/27
Verifying : glib2-2.62.4-2.fc31.x86_64 5/27
Verifying : libarchive-3.4.2-1.fc31.x86_64 6/27
Verifying : libarchive-3.4.0-1.fc31.x86_64 7/27
Verifying : libsss_idmap-2.2.3-13.fc31.x86_64 8/27
Verifying : libsss_idmap-2.2.2-3.fc31.x86_64 9/27
Verifying : libsss_nss_idmap-2.2.3-13.fc31.x86_64 10/27
Verifying : libsss_nss_idmap-2.2.2-3.fc31.x86_64 11/27
Verifying : libtirpc-1.2.5-1.rc2.fc31.x86_64 12/27
Verifying : libtirpc-1.2.5-0.fc31.x86_64 13/27
Verifying : libxcrypt-4.4.15-1.fc31.x86_64 14/27
Verifying : libxcrypt-4.4.12-1.fc31.x86_64 15/27
Verifying : pcre-8.43-3.fc31.x86_64 16/27
Verifying : pcre-8.43-2.fc31.1.x86_64 17/27
Verifying : pcre2-10.34-7.fc31.x86_64 18/27
Verifying : pcre2-10.34-5.fc31.x86_64 19/27
Verifying : shadow-utils-2:4.6-17.fc31.x86_64 20/27
Verifying : shadow-utils-2:4.6-16.fc31.x86_64 21/27
Verifying : sssd-client-2.2.3-13.fc31.x86_64 22/27
Verifying : sssd-client-2.2.2-3.fc31.x86_64 23/27
Verifying : sudo-1.9.0-0.1.b1.fc31.x86_64 24/27
Verifying : sudo-1.8.29-1.fc31.x86_64 25/27
Verifying : vim-minimal-2:8.2.348-1.fc31.x86_64 26/27
Verifying : vim-minimal-2:8.2.236-1.fc31.x86_64 27/27
Upgraded:
glib2-2.62.5-1.fc31.x86_64 libarchive-3.4.2-1.fc31.x86_64 libsss_idmap-2.2.3-13.fc31.x86_64 libsss_nss_idmap-2.2.3-13.fc31.x86_64 libtirpc-1.2.5-1.rc2.fc31.x86_64 libxcrypt-4.4.15-1.fc31.x86_64 pcre-8.43-3.fc31.x86_64 pcre2-10.34-7.fc31.x86_64
shadow-utils-2:4.6-17.fc31.x86_64 sssd-client-2.2.3-13.fc31.x86_64 sudo-1.9.0-0.1.b1.fc31.x86_64 vim-minimal-2:8.2.348-1.fc31.x86_64
Installed:
mkpasswd-5.5.6-1.fc31.x86_64 shared-mime-info-1.15-1.fc31.x86_64 whois-nls-5.5.6-1.fc31.noarch
Complete!
So, seems that the issue is related to the host kernel not being up-to-date enough so overlayfs2 could support cap_set_file.
What's the best way to proceed? I'm not sure. I'd first try to update the host kernel of the machines where you see the failure. If that doesn't help, maybe switch to using devicemapper only for installing packages may be a reasonable solution.
@GabyCT, does this (very long) answer make sense?
@fidencio @GabyCT we have the same issue on s390x. With devicemapper it works. I moved to a more recent kernel 5.3.0-42-generic
, but I still have the issue
I might be wrong. But are we missing the cap_set_file in the guest kernel?
I might be wrong. But are we missing the cap_set_file in the guest kernel?
I thought it was related to the host kernel. But, yes, it may be related to the guest kernel indeed. Out of curiosity. which host / guest kernel are you using, @alicefr?
@fidencio I guess is the guest kernel because the same command works with a standard container on my host. My host is 5.3.0-42-generic
and my guest is kernel is 5.4.15-71
. However, I'm running on s390x and my guest kernel has been compiled with these configs https://github.com/kata-containers/packaging/pull/994. So, I might be missing the correct configs to use this capability
@alicefr, shouldn't CONFIG_TMPFS_XATTR=y
be enough to have it working?
I'm far from being familiar with kernel-land, so, please, bear with me here. :-)
@fidencio not sure. It's enabled in all the arch. So, for both case x86 and s390x. Which kernel config is your guest using?
@alicefr, I didn't hit the issue myself, was only able to reproduce this on @GabyCT's environment, so I'll leave it to her to reply.
I'd assume whatever is the upstream that kata is using.
On my personal environment (where I haven't hit this issue), I'm playing with both host and kernel using the very same Fedora kernel.
@fidencio I tried with fedora 31 and kata from fedora and I failed with the same error. Could you try with this command:
$ podman run --cap-add all --security-opt label=disable -ti --runtime /usr/bin/kata-runtime fedora sh -c 'getcap -v /bin/sleep'
Failed to get capabilities of file `/bin/sleep' (Operation not supported)
@alicefr, indeed. I can reproduce this on Fedora32 / Rawhide as well. I'm adding this to my short-term list.
ok, my guess and I can be totally wrong is that 9p doesn't support the operation or I'm missing a config. Here my test: I changed the storage driver to devicemapper and I tried this:
$ mkdir test
$ touch test/get-cap-test
$ docker run -v $(pwd)/test:/test --cap-add all -ti --runtime kata-runtime fedora sh
mount
/dev/sda on / type ext4 (rw,relatime,stripe=16)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,size=1021760k,nr_inodes=255440,mode=755)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (ro,nosuid,nodev,noexec,relatime,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,nr_inodes=255440)
kataShared on /test type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
kataShared on /etc/resolv.conf type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
kataShared on /etc/hostname type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
kataShared on /etc/hosts type 9p (rw,nodev,relatime,dirsync,mmap,access=client,trans=virtio)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/latency_stats type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,nr_inodes=255440,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime,size=1021760k,nr_inodes=255440)
tmpfs on /sys/firmware type tmpfs (ro,relatime,size=1021760k,nr_inodes=255440)
sh-5.0# getcap bin/sleep
sh-5.0# getcap test/get-cap-test
Failed to get capabilities of file `test/get-cap-test' (Operation not supported)
Interesting.
@alicefr, I've reproduced this using virtiofs instead of 9p. David Gilbert kindly pointed out that passing "-o xattr" to virtiofsd solves the issue when using virtiofs.
Based on that, I do assume your guess ir correct!
Also, David also pointed that it seems to be "outside of kata" as he could reproduce that with just a virtiofs mount.
yes, probably. Not sure how to do that for 9p
Sorry to be rather late to this - there is a common known issue with 9p to do with unlinked tmp files, that in the past has stopped things like dnf, apt etc. from working. There are two ways to try and track this down to (dis)prove that is the problem....
1) use strace
to see what failed, and can you trace it back to an open/unlink operation.
2) move the TMPDIR to a ramfs in the container, and see if that fixes it...
(we have done the TMPDIR hack using the shmem dir before - see https://github.com/kata-containers/tests/blob/master/metrics/network/network-metrics-iperf3.sh#L40
From my preview examples, here the strace output:
sh-5.0# strace getcap get-cap-test
execve("/usr/sbin/getcap", ["getcap", "get-cap-test"], 0x3ffde47f0a8 /* 11 vars */) = 0
brk(NULL) = 0x2aa1374a000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3ff9c6fb000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=12092, ...}) = 0
mmap(NULL, 12092, PROT_READ, MAP_PRIVATE, 3, 0) = 0x3ff9c600000
close(3) = 0
openat(AT_FDCWD, "/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\2\1\0\0\0\0\0\0\0\0\0\0\3\0\26\0\0\0\1\0\0\0\0\0\0\27@"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=24848, ...}) = 0
mmap(NULL, 24888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ff9c580000
mmap(0x3ff9c585000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x3ff9c585000
close(3) = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\2\1\3\0\0\0\0\0\0\0\0\0\3\0\26\0\0\0\1\0\0\0\0\0\2\267P"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=3093208, ...}) = 0
mmap(NULL, 1736592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ff9c380000
mmap(0x3ff9c51e000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19d000) = 0x3ff9c51e000
mmap(0x3ff9c525000, 12176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3ff9c525000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3ff9c6f9000
mprotect(0x3ff9c51e000, 16384, PROT_READ) = 0
mprotect(0x3ff9c585000, 4096, PROT_READ) = 0
mprotect(0x2aa12082000, 4096, PROT_READ) = 0
mprotect(0x3ff9c6a6000, 4096, PROT_READ) = 0
munmap(0x3ff9c600000, 12092) = 0
lstat("get-cap-test", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
brk(NULL) = 0x2aa1374a000
brk(0x2aa1376b000) = 0x2aa1376b000
brk(NULL) = 0x2aa1376b000
capget({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, NULL) = 0
getxattr("get-cap-test", "security.capability", 0x3ffd847ea88, 24) = -1 EOPNOTSUPP (Operation not supported)
write(2, "Failed to get capabilities of fi"..., 76Failed to get capabilities of file `get-cap-test' (Operation not supported)
) = 76
exit_group(0) = ?
+++ exited with 0 +++
Ah, OK, I see the getxattr there now - so, ignore my previous comment ;-) As David was mentioned etc., /cc @dagrh
So, as far as I can see from qemu code we get ENOTSUP from this function https://github.com/qemu/qemu/blob/master/hw/9pfs/9p-xattr.c#L242 and https://github.com/qemu/qemu/blob/2833ad487cfff7dc33703e4731b75facde1c561e/hw/9pfs/9p-posix-acl.c#L150 that is used if the security model is none https://github.com/qemu/qemu/blob/master/hw/9pfs/9p-xattr.c#L282 . It's what we have in Kata for 9p https://github.com/kata-containers/runtime/blob/705713b4f9bc4d1e754871d5ef1ab5e99ea71aff/virtcontainers/qemu_arch_base.go#L467
So the -o xattr on virtiofsd is probably equivalent to security_model=passthrough on 9p - we don't do any translation.
@dagrh I tried to change the security model to passthrough
, but I still get EOPNOTSUPP
:
git diff
diff --git a/virtcontainers/qemu_arch_base.go b/virtcontainers/qemu_arch_base.go
index 9d72dd09..3632e481 100644
--- a/virtcontainers/qemu_arch_base.go
+++ b/virtcontainers/qemu_arch_base.go
@@ -464,7 +464,7 @@ func generic9PVolume(volume types.Volume, nestedRun bool) govmmQemu.FSDevice {
ID: devID,
Path: volume.HostPath,
MountTag: volume.MountTag,
- SecurityModel: govmmQemu.None,
+ SecurityModel: govmmQemu.PassThrough,
DisableModern: nestedRun,
}
}
at least on s390x I have this behavior. Can somebody check on x86?
Looks like the dnf update tests have been failing with virtio-fs for a while: https://github.com/kata-containers/tests/issues/2008
The recent failures have been with 9p. Something has changed to cause these failures. @GabyCT @jcvenegas Can we do a bisect to figure out when this test started failing with 9p?
@amshinde it could be due to a change in the fedora image?
@alicefr Yes, that could be a possible reason. Might be worth testing this with a fedora image with an older tag from a month back. @GabyCT @jcvenegas
@amshinde , so I did a checkout to several tags 1.9.2, 1.9.0, 1.8.5, 1.8.0 and it is not possible to perform dnf -y update
, I even tried to performed the checkout with 1.7.0 and 1.7.4, however, when I tried to run the container and then I got the following
docker: Error response from daemon: OCI runtime create failed: /usr/share/defaults/kata-containers/configuration-qemu.toml: file /usr/bin/qemu-lite-system-x86_64 does not exist: unknown.
/cc @jcvenegas
@GabyCT, if the problem is still the one about shadow-utils, I don't think a different version of podman would help.
What could help is a new fedora image, which already have the shadow-utils updated as part of the image.
While trying to perform a
dnf -y update
on a Fedora 31 container we have the following errorFedora 31 was released some time ago, it is not clear what make the regression.