search

gluster / glusterfs

Gluster Filesystem : Build your distributed storage in minutes
https://www.gluster.org
GNU General Public License v2.0
4.54k stars 1.07k forks source link

snapshots: plugin architecture #2834

Closed aravindavk closed 2 years ago

aravindavk commented 2 years ago

A few plugin APIs are based on below repo. New plugin methods are added that were not present in the initial code, also some of the methods like getting device, missed_snaps are removed.

https://github.com/major0/glusterfs/commits/glusterfs-snapshots

Retained the original authors. This PR contains only the code changes that are required to make the current glusterd code pluggable.

ZFS and Btrfs plugins from the above glusterfs fork are not included in this PR. Those will be sent as separate PRs with tests.

Every plugin should implement the following Struct

struct glusterd_snap_ops {
    /* Name of the Plugin */
    const char *name;

    /* To check if a plugin is supported or not */
    gf_boolean_t (*const probe)(
        char *brick_path
    );

    /* To get usage details */
    int (*const details)(
        dict_t *rsp_dict,
        glusterd_brickinfo_t *snap_brickinfo,
        char *key_prefix
    );

    /* Create the Snapshot */
    int32_t (*const create)(
        glusterd_brickinfo_t *snap_brickinfo,
        char *snapname,
        char *snap_volume_id,
        int32_t brick_num
    );

    /* Create a Clone */
    int32_t (*const clone)(
        glusterd_brickinfo_t *snap_brickinfo,
        char *snapname,
        char *snap_volume_id,
        char *clonename,
        char *clone_volume_id,
        int32_t brick_num
    );

    /* Remove a Snapshot */
    int32_t (*const remove)(
        glusterd_brickinfo_t *snap_brickinfo,
        char *snapname,
        char *snap_volume_id,
        int32_t brick_num
    );

    /* Activate and Mount a Snapshot */
    int32_t (*const activate)(
        glusterd_brickinfo_t *snap_brickinfo,
        char *snapname,
        char *snap_volume_id,
        int32_t brick_num
    );

    /* Unmount and Deactivate a Snapshot */
    int32_t (*const deactivate)(
        glusterd_brickinfo_t *snap_brickinfo,
        char *snapname,
        char *snap_volume_id,
        int32_t brick_num
    );

    /* Restore a Snapshot */
    int32_t (*const restore)(
        glusterd_brickinfo_t *snap_brickinfo,
        char *snapname,
        char *snap_volume_id,
        int32_t brick_num,
        gf_boolean_t *retain_origin_path
    );

    /* Brick Path of a Snapshot/Clone */
    int32_t (*const brick_path)(
        char *snap_mount_dir,
        int clone,
        char *snap_clone_name,
        char *snap_clone_volume_id,
        char *snap_brick_dir,
        int brick_num,
        char **snap_brick_path
    );
};

Change-Id: I3c8b8b0801d8b6f03362494e2ccf672e569071db Updates: #145 Signed-off-by: Sriram Raghunathan sriram@marirs.net.in Signed-off-by: Mark Ferrell major@homeonderanged.org Signed-off-by: Aravinda Vishwanathapura aravinda@kadalu.io

aravindavk commented 2 years ago

Some more build warnings and working on details API fixes. Will update this PR soon.

aravindavk commented 2 years ago

Update: Restore command is having an issue, looking into that. All other commands are now working after refactor.

aravindavk commented 2 years ago

/run regression

aravindavk commented 2 years ago

Update: PR is ready for review. One remaining issue I am working on is handling upgrade since new glusterd store keys introduced.

gluster-ant commented 2 years ago

1 test(s) failed ./tests/basic/volume-snapshot-clone.t

1 test(s) generated core ./tests/basic/volume-snapshot-clone.t

4 test(s) needed retry ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/basic/volume-snapshot-clone.t https://build.gluster.org/job/gh_centos7-regression/1688/

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

1 test(s) failed ./tests/basic/volume-snapshot-clone.t

1 test(s) generated core ./tests/basic/volume-snapshot-clone.t

3 test(s) needed retry ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/volume-snapshot-clone.t https://build.gluster.org/job/gh_centos7-regression/1691/

amarts commented 2 years ago

/run full regression

gluster-ant commented 2 years ago

10 test(s) failed ./tests/basic/volume-snapshot-clone.t ./tests/basic/volume-snapshot.t ./tests/bugs/snapshot/bug-1112613.t ./tests/bugs/snapshot/bug-1113975.t ./tests/bugs/snapshot/bug-1202436-calculate-quota-cksum-during-snap-restore.t ./tests/bugs/snapshot/bug-1227646.t ./tests/bugs/snapshot/bug-1279327.t ./tests/bugs/snapshot/bug-1322772-real-path-fix-for-snapshot.t ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

4 test(s) generated core ./tests/basic/volume-snapshot-clone.t ./tests/basic/volume-snapshot.t ./tests/bugs/snapshot/bug-1112613.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

16 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/volume-snapshot-clone.t ./tests/basic/volume-snapshot.t ./tests/bugs/distribute/bug-1286171.t ./tests/bugs/quota/bug-1035576.t ./tests/bugs/snapshot/bug-1112613.t ./tests/bugs/snapshot/bug-1113975.t ./tests/bugs/snapshot/bug-1202436-calculate-quota-cksum-during-snap-restore.t ./tests/bugs/snapshot/bug-1227646.t ./tests/bugs/snapshot/bug-1279327.t ./tests/bugs/snapshot/bug-1322772-real-path-fix-for-snapshot.t ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

3 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

1 test(s) failed ./tests/basic/volume-snapshot-clone.t

0 test(s) generated core

2 test(s) needed retry ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/volume-snapshot-clone.t

1 flaky test(s) marked as success even though they failed ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/1696/

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

1 test(s) failed ./tests/basic/volume-snapshot-clone.t

0 test(s) generated core

5 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/afr/ta-shd.t ./tests/basic/volume-snapshot-clone.t

3 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/1710/

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

1 test(s) failed ./tests/bugs/snapshot/bug-1113975.t

0 test(s) generated core

8 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/afr/self-heald.t ./tests/basic/ctime/ctime-rep-heal.t ./tests/basic/quick-read-with-upcall.t ./tests/bugs/replicate/do-not-reopen-fd.t ./tests/bugs/snapshot/bug-1113975.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/1783/

aravindavk commented 2 years ago

/run full regression

amarts commented 2 years ago

error is smoke due to a warning while compiling.

22:15:43 /home/jenkins/root/workspace/gh_fedora-smoke/xlators/mgmt/glusterd/src/glusterd-snapshot.c: In function ‘glusterd_snapshot_umount’:
22:15:43 /home/jenkins/root/workspace/gh_fedora-smoke/xlators/mgmt/glusterd/src/glusterd-snapshot.c:2520:1: error: label ‘out’ defined but not used [-Werror=unused-label]
22:15:43  2520 | out:
22:15:43       | ^~~
22:15:43 /home/jenkins/root/workspace/gh_fedora-smoke/xlators/mgmt/glusterd/src/glusterd-snapshot.c: In function ‘glusterd_bricks_snapshot_restore’:
22:15:43 /home/jenkins/root/workspace/gh_fedora-smoke/xlators/mgmt/glusterd/src/glusterd-snapshot.c:9118:1: error: label ‘out’ defined but not used [-Werror=unused-label]
22:15:43  9118 | out:
22:15:43       | ^~~
aravindavk commented 2 years ago

/run regression

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

1 test(s) failed ./tests/bugs/snapshot/bug-1202436-calculate-quota-cksum-during-snap-restore.t

0 test(s) generated core

5 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/hardlink-limit.t ./tests/basic/quick-read-with-upcall.t ./tests/bugs/snapshot/bug-1202436-calculate-quota-cksum-during-snap-restore.t https://build.gluster.org/job/gh_centos7-regression/1788/

aravindavk commented 2 years ago

/run full regression

gluster-ant commented 2 years ago

4 test(s) failed ./tests/bugs/snapshot/bug-1202436-calculate-quota-cksum-during-snap-restore.t ./tests/bugs/snapshot/bug-1322772-real-path-fix-for-snapshot.t ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

1 test(s) generated core ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

11 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/distribute/phase1-migration-fops.t ./tests/bugs/glusterd/brick-order-check-add-brick.t ./tests/bugs/glusterd/bug-1696046.t ./tests/bugs/snapshot/bug-1202436-calculate-quota-cksum-during-snap-restore.t ./tests/bugs/snapshot/bug-1322772-real-path-fix-for-snapshot.t ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t

aravindavk commented 2 years ago

4 test(s) failed ./tests/bugs/snapshot/bug-1202436-calculate-quota-cksum-during-snap-restore.t

This one is now fixed. Working on fixing other failures.

./tests/bugs/snapshot/bug-1322772-real-path-fix-for-snapshot.t ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

This is now fixed.

1 test(s) generated core ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

Remaining:

./tests/bugs/snapshot/bug-1322772-real-path-fix-for-snapshot.t ./tests/bugs/snapshot/bug-1597662.t

aravindavk commented 2 years ago

Update: Found a valid issue that causes mount failure of restored volume on Glusterd restart. Updated the PR, but still needs some changes to make it work. Will refresh the PR with the update tomorrow.

aravindavk commented 2 years ago

/run full regression

gluster-ant commented 2 years ago

12 test(s) failed ./tests/basic/glusterd/arbiter-volume-probe.t ./tests/basic/glusterd/thin-arbiter-volume-probe.t ./tests/bugs/glusterd/bug-1699339.t ./tests/bugs/glusterd/mgmt-handshake-and-volume-sync-post-glusterd-restart.t ./tests/bugs/glusterd/optimized-basic-testcases-in-cluster.t ./tests/bugs/glusterd/remove-brick-in-cluster.t ./tests/bugs/glusterd/reset-brick-and-daemons-follow-quorum.t ./tests/bugs/glusterd/snapshot-operations.t ./tests/bugs/glusterd/sync-post-glusterd-restart.t ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t ./tests/line-coverage/glusterd-coverage.t

0 test(s) generated core

17 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync.t ./tests/basic/glusterd/arbiter-volume-probe.t ./tests/basic/glusterd/thin-arbiter-volume-probe.t ./tests/bugs/glusterd/bug-1699339.t ./tests/bugs/glusterd/mgmt-handshake-and-volume-sync-post-glusterd-restart.t ./tests/bugs/glusterd/optimized-basic-testcases-in-cluster.t ./tests/bugs/glusterd/remove-brick-in-cluster.t ./tests/bugs/glusterd/reset-brick-and-daemons-follow-quorum.t ./tests/bugs/glusterd/snapshot-operations.t ./tests/bugs/glusterd/sync-post-glusterd-restart.t ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t ./tests/line-coverage/glusterd-coverage.t

3 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t

aravindavk commented 2 years ago

/run full regression

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

2 test(s) failed ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

0 test(s) generated core

5 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/bugs/snapshot/bug-1597662.t ./tests/bugs/snapshot/bug-1618004-fix-memory-corruption-in-snap-import.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

0 test(s) failed

1 test(s) generated core ./tests/basic/fencing/afr-lock-heal-advanced.t https://build.gluster.org/job/gh_centos7-regression/1808/

aravindavk commented 2 years ago

/run full regression

gluster-ant commented 2 years ago

6 test(s) needed retry ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/00-geo-rep/georep-basic-dr-rsync.t ./tests/bugs/replicate/do-not-reopen-fd.t ./tests/bugs/rpc/bug-954057.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t

gluster-ant commented 2 years ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 382273c5f..c8a16710e 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -920,9 +920,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
-                                       : (transport == 1) ? "rdma"
-                                                          : "tcp,rdma"));
+        cli_out("Transport-type: %s",
+                ((transport == 0) ? "tcp"
+                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
aravindavk commented 2 years ago

/run full regression

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 382273c5f..c8a16710e 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -920,9 +920,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
-                                       : (transport == 1) ? "rdma"
-                                                          : "tcp,rdma"));
+        cli_out("Transport-type: %s",
+                ((transport == 0) ? "tcp"
+                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
diff --git a/xlators/mgmt/glusterd/src/glusterd-utils.c b/xlators/mgmt/glusterd/src/glusterd-utils.c
index 9fff62f72..967d32664 100644
--- a/xlators/mgmt/glusterd/src/glusterd-utils.c
+++ b/xlators/mgmt/glusterd/src/glusterd-utils.c
@@ -2255,8 +2255,8 @@ retry:
                         "*-posix.health-check-interval=0", NULL);

         /* Snapshot Bricks are read only, disable the changelog xlator */
-        runner_add_args(&runner, "--xlator-option",
-                        "*-changelog.changelog=off", NULL);
+        runner_add_args(&runner, "--xlator-option", "*-changelog.changelog=off",
+                        NULL);
     }

     runner_add_arg(&runner, "--brick-port");
aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 382273c5f..c8a16710e 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -920,9 +920,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
-                                       : (transport == 1) ? "rdma"
-                                                          : "tcp,rdma"));
+        cli_out("Transport-type: %s",
+                ((transport == 0) ? "tcp"
+                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
gluster-ant commented 2 years ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 382273c5f..c8a16710e 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -920,9 +920,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
-                                       : (transport == 1) ? "rdma"
-                                                          : "tcp,rdma"));
+        cli_out("Transport-type: %s",
+                ((transport == 0) ? "tcp"
+                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

1 test(s) failed ./tests/basic/ec/ec-data-heal.t

0 test(s) generated core

5 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/ec/ec-data-heal.t

3 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/1868/

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

1 test(s) failed ./tests/bugs/glusterd/brick-order-check-add-brick.t

0 test(s) generated core

2 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/bugs/glusterd/brick-order-check-add-brick.t https://build.gluster.org/job/gh_centos7-regression/1877/

aravindavk commented 2 years ago

/run full regression

gluster-ant commented 2 years ago

0 test(s) failed

1 test(s) generated core ./tests/basic/fencing/afr-lock-heal-advanced.t

6 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/bugs/glusterd/bug-1696046.t ./tests/features/worm.t

3 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t

gluster-ant commented 2 years ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 382273c5f..c8a16710e 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -920,9 +920,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
-                                       : (transport == 1) ? "rdma"
-                                                          : "tcp,rdma"));
+        cli_out("Transport-type: %s",
+                ((transport == 0) ? "tcp"
+                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
aravindavk commented 2 years ago

/run regression

aravindavk commented 2 years ago

/run regression

gluster-ant commented 2 years ago

1 test(s) failed ./tests/00-geo-rep/00-georep-verify-non-root-setup.t

0 test(s) generated core

15 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_core_multiplex-limit-issue-151.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_distribute_bug-1122443.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_glusterd_quorum-value-check.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/features_lock-migration_lkmigration-set-option.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/00-georep-verify-non-root-setup.t

14 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_core_multiplex-limit-issue-151.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_distribute_bug-1122443.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_glusterd_quorum-value-check.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/features_lock-migration_lkmigration-set-option.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/1905/