Closed wxiaoguang closed 3 years ago
More information is discussed in https://github.com/gluster/glusterfs/issues/2606, I think these issues are highly related.
(And other issues: https://github.com/gluster/glusterfs/issues/1962)
Hello Mr @wxiaoguang
in the current state, we tested this problem with these environments and we have the same issue :
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
glusterfs version : 9.0 9.02 9.03
qemu-img version 5.2.0
ubuntu@Node-4:~$ sudo qemu-img create gluster://storage-master-3/gv0/testimage.img 1G
Formatting 'gluster://storage-master-3/gv0/testimage.img', fmt=raw size=1073741824
[2021-07-11 15:50:55.887803 +0000] I [io-stats.c:3706:ios_sample_buf_size_configure] 0-gv0: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2021-07-11 15:50:56.009168 +0000] E [MSGID: 108006] [afr-common.c:6140:__afr_handle_child_down_event] 0-gv0-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-07-11 15:51:05.896412 +0000] I [io-stats.c:4038:fini] 0-gv0: io-stats translator unloaded
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
glusterfs version :5.5.3
qemu-img version 5.2.0
debian@Node-5:~$ sudo qemu-img create gluster://storage-node-5/gv0/testimage.img 1G
Formatting 'gluster://storage-node-5/gv0/testimage.img', fmt=raw size=1073741824
[2021-07-11 15:49:50.227484] E [MSGID: 108006] [afr-common.c:5314:__afr_handle_child_down_event] 0-gv0-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
@wxiaoguang I don't think this is a bug. When fini is executed, replicate xlator (AFR) will receive child down for each of its children. When connection to all its children are lost, you will see this message. You can observe it in any gfapi program that calls glfs_fini() and can be ignored.
@wxiaoguang I don't think this is a bug. When fini is executed, replicate xlator (AFR) will receive child down for each of its children. When connection to all its children are lost, you will see this message. You can observe it in any gfapi program that calls glfs_fini() and can be ignored.
@itisravi I think the user's question is why are they coming on the terminal and not in logs.
@itisravi I think the user's question is why are they coming on the terminal and not in logs.
I think this could be due to the logfile location configured in qemu. In the source code (qemu/block/gluster.c), I do see #define GLUSTER_LOGFILE_DEFAULT "-" /* handled in libgfapi as /dev/stderr */
@itisravi @pranithk it is a bug, it takes too much time in glfs_fini
, please see details in:
https://github.com/gluster/glusterfs/issues/2606 glfs_fini takes too much time and slows down qemu-img and other clients
https://github.com/gluster/glusterfs/issues/1962 gfapi: glfs_fini fucntion consume more time reusult in qemu create image or start vm slow
and the bug doesn't exist in glusterfs-5.5 (or 4.1), I can not say which version introduced this bug.
@itisravi @pranithk it is a bug, it takes too much time in
glfs_fini
, please see details in:2606 glfs_fini takes too much time and slows down qemu-img and other clients
1962 gfapi: glfs_fini fucntion consume more time reusult in qemu create image or start vm slow
and the bug doesn't exist in glusterfs-5.5 (or 4.1), I can not say which version introduced this bug.
@wxiaoguang They are separate issues. I believe Mohit is working on the delay issue. @soumyakoduri @thotz Is it okay if we make the default /dev/null instead?
Although I agree that the log is a little misleading .... however, without this terminal log, I won't look into the delay problem, I may accept the wrong situation that qemu-img
is slow with glusterfs ....
I think showing serious errors to end users is a good choice, and if the delay issue is resolved, there will be no error outputs anymore.
Closing the issue since
Description of problem:
qemu-img always reports: "All subvolumes are down. Going offline until at least one of them comes back up."
I have tried glusterfs 9.0 cluster and 9.2 cluster in debian 10 with fresh installation.
I have tried to turn on/off
server.allow-inseucre
, it doesn't have affect.The exact command to reproduce the issue:
qemu-img qcow2 format:
qemu-img raw format:
then if we look into the gluster data directory, we can find the file is there,
The full output of the command that failed:
- The operating system / glusterfs version:
Debian 10 GlusterFS 9.0 cluster (all packages are 9.0) GlusterFS 9.2 cluster (all packages are 9.2)
Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration