tangyla commented 3 years ago

Description of problem: program with gfapi, if program entering debug, gfapi hang up

use gdb step debug, the ‘glfs_fini’ function will hang, also glfs_creat/glfs_lseek/glfs_close and so on.

Additional info: compile command gcc -Wall -ggdb3 -c gfapi.c && gcc -o gfapi gfapi.o -lgfapi

the test code is follow:

#include <time.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <glusterfs/api/glfs.h>

int create_file(struct glfs* glfs) {
    int ret;
    int64_t current_length;
    struct glfs_fd* fd = NULL;
    fd = glfs_creat(glfs, "test.qcow2", O_WRONLY | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR);
    fprintf(stderr, "glfs_creat fd = %p\n", fd);

    current_length = glfs_lseek(fd, 0, SEEK_END);
    fprintf(stderr, "glfs_lseek ret = %ld\n", current_length);

    if (fd) {
        ret = glfs_close(fd);
        fprintf(stderr, "glfs_close ret = %d\n", ret);
    }

    return EXIT_SUCCESS;
}

int main(int argc, char** argv) {
    int ret = 0;
    char* volume = "datastore";
    char* server = "node0";
    int port = 24007;
    struct glfs* glfs;
    time_t start, end, b_fini, e_fini;

    start = time(NULL);
    fprintf(stderr, "%s start time: %ld\n", argv[0], start);

    glfs = glfs_new(volume);
    ret = glfs_set_volfile_server(glfs, "tcp", server, port);
    fprintf(stderr, "glfs_set_volfile_server ret = %d\n", ret);
    ret = glfs_set_logging(glfs, "/dev/stderr", 4);
    fprintf(stderr, "glfs_set_logging ret = %d\n", ret);
    ret = glfs_init(glfs);
    fprintf(stderr, "glfs_init ret = %d\n", ret);
    create_file(glfs);

    b_fini = time(NULL);
    fprintf(stderr, "glfs_fini start: %ld, before consume seconds: %ld\n", b_fini, b_fini - start);
    ret = glfs_fini(glfs);
    e_fini = time(NULL);
    fprintf(stderr, "glfs_fini end: %ld, consume seconds: %ld\n", e_fini, e_fini - b_fini);
    fprintf(stderr, "glfs_fini ret = %d\n", ret);

    end = time(NULL);
    fprintf(stderr, "%s end time: %ld, consume seconds: %ld\n", argv[0], end, (end - start));

    return EXIT_SUCCESS;
}

mohit84 commented 3 years ago

Hi, I have tried to reproduce same on glusterfs-7 version on local vm, i am not facing any issue.

/usr/local/sbin/gluster --version glusterfs 7.8 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. https://www.gluster.org/ GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root@ip-172-31-10-141 ~]# ./gfapi ./gfapi start time: 1610005783 glfs_set_volfile_server ret = 0 glfs_set_logging ret = 0 glfs_init ret = 0 glfs_creat fd = 0x2028940 glfs_lseek ret = 0 glfs_close ret = 0 glfs_fini start: 1610005783, before consume seconds: 0 [2021-01-07 07:49:43.455566] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:49:43.455861] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:49:43.456022] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-2: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:49:53.320568] I [io-stats.c:4040:fini] 0-patchy: io-stats translator unloaded glfs_fini end: 1610005794, consume seconds: 11 glfs_fini ret = 0 ./gfapi end time: 1610005794, consume seconds: 11 [root@ip-172-31-10-141 ~]# ./gfapi ./gfapi start time: 1610005797 glfs_set_volfile_server ret = 0 glfs_set_logging ret = 0 glfs_init ret = 0 glfs_creat fd = 0x1389940 glfs_lseek ret = 0 glfs_close ret = 0 glfs_fini start: 1610005797, before consume seconds: 0 [2021-01-07 07:49:57.549342] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:49:57.549599] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:49:57.549758] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-2: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:50:07.415926] I [io-stats.c:4040:fini] 0-patchy: io-stats translator unloaded glfs_fini end: 1610005807, consume seconds: 10 glfs_fini ret = 0 ./gfapi end time: 1610005807, consume seconds: 10 [root@ip-172-31-10-141 ~]# ./gfapi ./gfapi start time: 1610005822 glfs_set_volfile_server ret = 0 glfs_set_logging ret = 0 glfs_init ret = 0 glfs_creat fd = 0x120f940 glfs_lseek ret = 0 glfs_close ret = 0 glfs_fini start: 1610005822, before consume seconds: 0 [2021-01-07 07:50:22.482248] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:50:22.482554] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:50:22.482729] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-2: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:50:32.340615] I [io-stats.c:4040:fini] 0-patchy: io-stats translator unloaded glfs_fini end: 1610005832, consume seconds: 10 glfs_fini ret = 0 ./gfapi end time: 1610005832, consume seconds: 10 [root@ip-172-31-10-141 ~]# ./gfapi ./gfapi start time: 1610005836 glfs_set_volfile_server ret = 0 glfs_set_logging ret = 0 glfs_init ret = 0 glfs_creat fd = 0x2317940 glfs_lseek ret = 0 glfs_close ret = 0 glfs_fini start: 1610005837, before consume seconds: 1 [2021-01-07 07:50:37.330411] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:50:37.330743] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:50:37.331111] E [MSGID: 108006] [afr-common.c:5308:afr_handle_child_down_event] 0-patchy-replicate-2: All subvolumes are down. Going offline until at least one of them comes back up. [2021-01-07 07:50:47.194427] I [io-stats.c:4040:fini] 0-patchy: io-stats translator unloaded glfs_fini end: 1610005848, consume seconds: 11 glfs_fini ret = 0 ./gfapi end time: 1610005848, consume seconds: 12

Can you share stack of the program while it is hanged.

Thanks, Mohit Agrawal

tangyla commented 3 years ago

direct execute gfapi not hanged. test this like fellow:

[root@node0 gfapi]# gdb gfapi GNU gdb (GDB) Red Hat Enterprise Linux 8.2-11.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.

For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from gfapi...done. (gdb) b create_file Breakpoint 1 at 0x400952: file gfapi.c, line 10. (gdb) r Starting program: /root/qemu/gfapi/gfapi Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-101.el8.x86_64 warning: Loadable section ".note.gnu.property" outside of ELF segments [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". warning: Loadable section ".note.gnu.property" outside of ELF segments /root/qemu/gfapi/gfapi start time: 1610012561 [New Thread 0x7ffff433c700 (LWP 22829)] [New Thread 0x7ffff303b700 (LWP 22830)] [New Thread 0x7ffff283a700 (LWP 22831)] [New Thread 0x7ffff1439700 (LWP 22832)] [New Thread 0x7ffff0c38700 (LWP 22833)] [New Thread 0x7ffff016c700 (LWP 22834)] glfs_set_volfile_server ret = 0 glfs_set_logging ret = 0 [New Thread 0x7fffef768700 (LWP 22835)] [New Thread 0x7fffeef67700 (LWP 22836)] [New Thread 0x7fffee2b2700 (LWP 22837)] glfs_init ret = 0

Thread 1 "gfapi" hit Breakpoint 1, create_file (glfs=0x603a80) at gfapi.c:10 10 struct glfs_fd* fd = NULL; Missing separate debuginfos, use: yum debuginfo-install zlib-1.2.11-16.el8_2.x86_64 (gdb) n 11 fd = glfs_creat(glfs, "test.qcow2", O_WRONLY | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR); (gdb) n ^C Thread 1 "gfapi" received signal SIGINT, Interrupt. 0x00007ffff5e6f48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) where

0 0x00007ffff5e6f48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

1 0x00007ffff6ecc2bf in syncop_lookup (subvol=subvol@entry=0x7fffe0021a30, loc=loc@entry=0x7fffffffdb80, iatt=iatt@entry=0x7fffffffdbc0, parent=parent@entry=0x0, xdata_in=xdata_in@entry=0x690628, xdata_out=xdata_out@entry=0x0) at syncop.c:1135

2 0x00007ffff7bc218e in glfs_resolve_component (fs=fs@entry=0x603a80, subvol=subvol@entry=0x7fffe0021a30, parent=parent@entry=0x7fffdc00e2f8, component=, component@entry=0x7fffdc007590 "test.qcow2", iatt=iatt@entry=0x7fffffffdd60,

force_lookup=<optimized out>) at glfs-resolve.c:405

3 0x00007ffff7bc25d3 in priv_glfs_resolve_at (fs=fs@entry=0x603a80, subvol=subvol@entry=0x7fffe0021a30, at=at@entry=0x7fffdc00e2f8, origpath=, origpath@entry=0x400c98 "test.qcow2", loc=loc@entry=0x7fffffffdf10, iatt=iatt@entry=0x7fffffffdf50,

follow=1, reval=0) at glfs-resolve.c:517

4 0x00007ffff7bc3f6b in glfs_resolve_path (fs=0x603a80, subvol=0x7fffe0021a30, origpath=0x400c98 "test.qcow2", loc=0x7fffffffdf10, iatt=0x7fffffffdf50, follow=1, reval=0) at glfs-resolve.c:640

5 0x00007ffff7bc4025 in priv_glfs_resolve (fs=fs@entry=0x603a80, subvol=subvol@entry=0x7fffe0021a30, origpath=origpath@entry=0x400c98 "test.qcow2", loc=loc@entry=0x7fffffffdf10, iatt=iatt@entry=0x7fffffffdf50, reval=reval@entry=0) at glfs-resolve.c:655

6 0x00007ffff7bb8dad in pub_glfs_creat (fs=0x603a80, path=0x400c98 "test.qcow2", flags=577, mode=384) at glfs-fops.c:810

7 0x0000000000400975 in create_file (glfs=0x603a80) at gfapi.c:11

8 0x0000000000400b18 in main (argc=1, argv=0x7fffffffe1d8) at gfapi.c:45

test environment detail:

OS: CentOS Linux release 8.2.2004 (Core)

gluster: glusterfs 7.8 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. https://www.gluster.org/ GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation.

gcc: gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5) Copyright (C) 2018 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

gdb: GNU gdb (GDB) Red Hat Enterprise Linux 8.2-11.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

tangyla commented 3 years ago

other gfapis when debug program has same problem, e.g: glfs_fini/glfs_seek and so on.

mohit84 commented 3 years ago

Please share "thread apply all bt full " and brick logs also at the same time.

tangyla commented 3 years ago

(gdb) n 11 fd = glfs_creat(glfs, "test.qcow2", O_WRONLY | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR); (gdb) ^C Thread 1 "gfapi" received signal SIGINT, Interrupt. 0x00007ffff5e6f48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) thread apply all bt full

Thread 10 (Thread 0x7fffee2b2700 (LWP 23164)):

0 0x00007ffff78db1b7 in epoll_wait () from /lib64/libc.so.6