Closed wxiaoguang closed 3 years ago
wxiaoguang
commented
3 years ago More information is discussed in https://github.com/gluster/glusterfs/issues/2606, I think these issues are highly related.
(And other issues: https://github.com/gluster/glusterfs/issues/1962)
Shahrokh-Af
commented
3 years ago Hello Mr @wxiaoguang
in the current state, we tested this problem with these environments and we have the same issue :
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
glusterfs version : 9.0 9.02 9.03
qemu-img version 5.2.0
ubuntu@Node-4:~$ sudo qemu-img create gluster://storage-master-3/gv0/testimage.img 1G
Formatting 'gluster://storage-master-3/gv0/testimage.img', fmt=raw size=1073741824
[2021-07-11 15:50:55.887803 +0000] I [io-stats.c:3706:ios_sample_buf_size_configure] 0-gv0: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2021-07-11 15:50:56.009168 +0000] E [MSGID: 108006] [afr-common.c:6140:__afr_handle_child_down_event] 0-gv0-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2021-07-11 15:51:05.896412 +0000] I [io-stats.c:4038:fini] 0-gv0: io-stats translator unloadedDistributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
glusterfs version :5.5.3
qemu-img version 5.2.0
debian@Node-5:~$ sudo qemu-img create gluster://storage-node-5/gv0/testimage.img 1G
Formatting 'gluster://storage-node-5/gv0/testimage.img', fmt=raw size=1073741824
[2021-07-11 15:49:50.227484] E [MSGID: 108006] [afr-common.c:5314:__afr_handle_child_down_event] 0-gv0-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
itisravi
commented
3 years ago @wxiaoguang I don't think this is a bug. When fini is executed, replicate xlator (AFR) will receive child down for each of its children. When connection to all its children are lost, you will see this message. You can observe it in any gfapi program that calls glfs_fini() and can be ignored.
pranithk
commented
3 years ago @wxiaoguang I don't think this is a bug. When fini is executed, replicate xlator (AFR) will receive child down for each of its children. When connection to all its children are lost, you will see this message. You can observe it in any gfapi program that calls glfs_fini() and can be ignored.
@itisravi I think the user's question is why are they coming on the terminal and not in logs.
itisravi
commented
3 years ago @itisravi I think the user's question is why are they coming on the terminal and not in logs.
I think this could be due to the logfile location configured in qemu. In the source code (qemu/block/gluster.c), I do see #define GLUSTER_LOGFILE_DEFAULT "-" /* handled in libgfapi as /dev/stderr */
wxiaoguang
commented
3 years ago @itisravi @pranithk it is a bug, it takes too much time in glfs_fini, please see details in:
https://github.com/gluster/glusterfs/issues/2606 glfs_fini takes too much time and slows down qemu-img and other clients
https://github.com/gluster/glusterfs/issues/1962 gfapi: glfs_fini fucntion consume more time reusult in qemu create image or start vm slow
and the bug doesn't exist in glusterfs-5.5 (or 4.1), I can not say which version introduced this bug.
pranithk
commented
3 years ago @itisravi @pranithk it is a bug, it takes too much time in
glfs_fini, please see details in:2606 glfs_fini takes too much time and slows down qemu-img and other clients
1962 gfapi: glfs_fini fucntion consume more time reusult in qemu create image or start vm slow
and the bug doesn't exist in glusterfs-5.5 (or 4.1), I can not say which version introduced this bug.
@wxiaoguang They are separate issues. I believe Mohit is working on the delay issue. @soumyakoduri @thotz Is it okay if we make the default /dev/null instead?
wxiaoguang
commented
3 years ago Although I agree that the log is a little misleading .... however, without this terminal log, I won't look into the delay problem, I may accept the wrong situation that qemu-img is slow with glusterfs ....
I think showing serious errors to end users is a good choice, and if the delay issue is resolved, there will be no error outputs anymore.
itisravi
commented
3 years ago Closing the issue since
wxiaoguang
commented
2 years ago
Description of problem:
qemu-img always reports: "All subvolumes are down. Going offline until at least one of them comes back up."
I have tried glusterfs 9.0 cluster and 9.2 cluster in debian 10 with fresh installation.
I have tried to turn on/off
server.allow-inseucre, it doesn't have affect.The exact command to reproduce the issue:
qemu-img qcow2 format:
qemu-img raw format:
then if we look into the gluster data directory, we can find the file is there,
The full output of the command that failed:
- The operating system / glusterfs version:
Debian 10 GlusterFS 9.0 cluster (all packages are 9.0) GlusterFS 9.2 cluster (all packages are 9.2)
Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration