search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 375 forks source link

virtio-fs cache problem? #3006

Closed wenbinzeng closed 3 years ago

wenbinzeng commented 4 years ago

Description of problem

Steps to reproduce the problem:

  1. install docker with kata static tarball -- https://github.com/kata-containers/runtime/releases/download/1.11.3/kata-static-1.11.3-x86_64.tar.xz
  2. cd /opt/kata/share/defaults/kata-containers; ln -s configuration-qemu-virtiofs.toml configuration.toml
  3. docker run --rm -ti centos:7 bash
  4. execute the following command in container: yum install -y libmnl

Expected result

expect the package to be installed with no error

Actual result

yum reported errors: ... Running transaction Installing : libmnl-1.0.3-7.el7.x86_64 1/1 Verifying : libmnl-1.0.3-7.el7.x86_64 1/1 Traceback (most recent call last): File "/usr/bin/yum", line 29, in yummain.user_main(sys.argv[1:], exit_code=True) File "/usr/share/yum-cli/yummain.py", line 375, in user_main errcode = main(args) File "/usr/share/yum-cli/yummain.py", line 281, in main return_code = base.doTransaction() File "/usr/share/yum-cli/cli.py", line 817, in doTransaction resultobject = self.runTransaction(cb=cb) File "/usr/lib/python2.7/site-packages/yum/init.py", line 1944, in runTransaction self.verifyTransaction(resultobject, vTcb) File "/usr/lib/python2.7/site-packages/yum/init.py", line 2007, in verifyTransaction po.yumdb_info.releasever = self.conf.yumvar['releasever'] File "/usr/lib/python2.7/site-packages/yum/rpmsack.py", line 1936, in setattr self._write(attr, value) File "/usr/lib/python2.7/site-packages/yum/rpmsack.py", line 1860, in _write misc.unlink_f(fn + '.tmp') File "/usr/lib/python2.7/site-packages/yum/misc.py", line 966, in unlink_f os.unlink(filename) OSError: [Errno 5] Input/output error: '/var/lib/yum/yumdb/l/c105e6d2014edce517b7da13bb8b9a085af63d34-libmnl-1.0.3-7.el7-x86_64/releasever.tmp'

Also, the above file could not be manually removed:

[root@6bd06068b8a5 /]# ls -l /var/lib/yum/yumdb/l/c105e6d2014edce517b7da13bb8b9a085af63d34-libmnl-1.0.3-7.el7-x86_64/releasever.tmp -rw-r--r-- 2 root root 1 Aug 9 21:39 /var/lib/yum/yumdb/l/c105e6d2014edce517b7da13bb8b9a085af63d34-libmnl-1.0.3-7.el7-x86_64/releasever.tmp

[root@6bd06068b8a5 /]# rm -f /var/lib/yum/yumdb/l/c105e6d2014edce517b7da13bb8b9a085af63d34-libmnl-1.0.3-7.el7-x86_64/releasever.tmp rm: cannot remove '/var/lib/yum/yumdb/l/c105e6d2014edce517b7da13bb8b9a085af63d34-libmnl-1.0.3-7.el7-x86_64/releasever.tmp': Input/output error

If I stat the directory before executing rm, the file can be removed:

[root@6bd06068b8a5 /]# ls /var/lib/yum/yumdb/l/c105e6d2014edce517b7da13bb8b9a085af63d34-libmnl-1.0.3-7.el7-x86_64/ from_repo reason releasever.tmp

[root@6bd06068b8a5 /]# rm -f /var/lib/yum/yumdb/l/c105e6d2014edce517b7da13bb8b9a085af63d34-libmnl-1.0.3-7.el7-x86_64/releasever.tmp [root@6bd06068b8a5 /]#

Further information

  1. tested 9pfs, no problem;

  2. tested kata release 1.11.0 and 1.9.3, all have the same problem;

  3. tested virtio_fs_cache="auto", same problem;

  4. tested virtio_fs_cache="none", saw a different problem:

[root@6f4e9edbf05b /]# yum install -y libmnl error: db5 error(19) from dbenv->open: No such device error: cannot open Packages index using db5 - No such device (19) error: cannot open Packages database in /var/lib/rpm CRITICAL:yum.main:

Error: rpmdb open failed

fidencio commented 4 years ago

Adding @dagrh to the loop.

wenbinzeng commented 3 years ago

Further findings: the problem happens when backing filesystems is ext4; after switching backing filesystems to xfs the problem is gone.

ariel-adam commented 3 years ago

@jcvenegas will look into this

c3d commented 3 years ago

This could be a duplicate of #2580.

Could you try again passing -o xattr as additional args for virtiofs in your configuration?

Edit your configuration.toml file (e.g. under /usr/share/kata-containers/defaults), look for virtio_fs_extra_args, set it so that it contains something like `[ "-o", "xattr" ]

fidencio commented 3 years ago

This issue is being automatically closed as Kata Containers 1.x has now reached EOL (End of Life). This means it is no longer being maintained.

Important:

All users should switch to the latest Kata Containers 2.x release to ensure they are using a maintained release that contains the latest security fixes, performance improvements and new features.

This decision was discussed by the @kata-containers/architecture-committee and has been announced via the Kata Containers mailing list:

If you believe this issue still applies to Kata Containers 2.x, please open an issue against the Kata Containers 2.x repository, pointing to this one, providing details to allow us to migrate it.