search

gluster / glusterfs

Gluster Filesystem : Build your distributed storage in minutes
https://www.gluster.org
GNU General Public License v2.0
4.54k stars 1.07k forks source link

io-stats: Introduce FOP_HITS_PER_SEC in profile data #3735

Closed mohit84 closed 1 year ago

mohit84 commented 1 year ago

1) Currently, glusterfs profile does not provide any metrics to measure what is the fops handling rate per second? Sometimes a user/developer can be interested to monitor this value. 2) During testing i have observed the client profile is printing latency in microsec even time difference is captured in nanosecond.

Solution: 1) Analyze the FOP_HITS_PER_SEC during profile capture based on the last time duration interval divided by a total number of fop hits during that duration. 2) Correct the format to print the latency in nanosecond.

Fixes: #3734 Change-Id: Id0f043493854724d32d81dbe09e254d4d19ffdbb Signed-off-by: Mohit Agrawal moagrawa@redhat.com

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index ffafdda20..38d44c922 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
gluster-ant commented 1 year ago

1 test(s) failed ./tests/basic/afr/afr-read-hash-mode.t

1 test(s) generated core ./tests/basic/afr/afr-read-hash-mode.t

4 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/afr/afr-read-hash-mode.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2724/

mohit84 commented 1 year ago

/run regression

mohit84 commented 1 year ago

/run full regression

mohit84 commented 1 year ago

/run regression

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index ffafdda20..38d44c922 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/bugs/distribute/bug-882278.t

0 test(s) generated core

5 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/basic/hardlink-limit.t ./tests/bugs/distribute/bug-882278.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2728/

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/bugs/distribute/bug-882278.t

0 test(s) generated core

4 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/bugs/distribute/bug-882278.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2729/

mohit84 commented 1 year ago

The test case ./tests/bugs/distribute/bug-882278.t is failing because of not being able to resolve a hostname, I think we need to add an entry in /etc/hosts. I have updated the same to the infra team.

mscherer commented 1 year ago

/run regression

mscherer commented 1 year ago

Related to https://github.com/gluster/project-infrastructure/issues/170

mscherer commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/basic/distribute/file-create.t

0 test(s) generated core

9 test(s) needed retry ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/basic/afr/ta-shd.t ./tests/basic/distribute/file-create.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2734/

mohit84 commented 1 year ago

/run regression

mohit84 commented 1 year ago

/run full regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/00-geo-rep/georep-basic-tarssh-ec.t

0 test(s) generated core

11 test(s) needed retry ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/bug-1600145.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/georep-basic-tarssh-ec.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2737/

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/00-geo-rep/georep-basic-tarssh-ec.t

0 test(s) generated core

9 test(s) needed retry ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/georep-basic-tarssh-ec.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2738/

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/00-geo-rep/georep-basic-tarssh-ec.t

0 test(s) generated core

10 test(s) needed retry ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/georep-basic-tarssh-ec.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2739/

mscherer commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/00-geo-rep/georep-basic-tarssh-ec.t

0 test(s) generated core

9 test(s) needed retry ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/georep-basic-tarssh-ec.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2742/

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/bugs/rpc/bug-884452.t

0 test(s) generated core

8 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-tarssh-arbiter.t ./tests/bugs/glusterd/brick-order-check-add-brick.t ./tests/bugs/glusterfs-server/bug-887145.t ./tests/bugs/quota/bug-1035576.t ./tests/bugs/rpc/bug-884452.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2748/

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/basic/ec/ec-data-heal.t

0 test(s) generated core

3 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/basic/ec/ec-data-heal.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2749/

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/basic/distribute/spare_file_rebalance.t

0 test(s) generated core

6 test(s) needed retry ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/distribute/spare_file_rebalance.t

5 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2785/

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

1 test(s) failed ./tests/basic/distribute/spare_file_rebalance.t

0 test(s) generated core

5 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/afr/sparse-file-self-heal.t ./tests/basic/distribute/file-create.t ./tests/basic/distribute/spare_file_rebalance.t https://build.gluster.org/job/gh_centos7-regression/2792/

gluster-ant commented 1 year ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index e94ace427..6760123bc 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index e94ace427..6760123bc 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 45555f343..ce5c471f3 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
mohit84 commented 1 year ago

/run regression

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 813552c50..240f6bd66 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
mohit84 commented 1 year ago

/recheck smoke

gluster-ant commented 1 year ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 813552c50..240f6bd66 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);
xhernandez commented 1 year ago

I've done a quick test of the patch and it doesn't seem to work:

Brick: gf:/root/bricks/test_1
-----------------------------
Cumulative Stats:
   Block Size:                  4b+              131072b+ 
 No. of Reads:                    0                     8 
No. of Writes:                    1                     8 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 ns       0.00 ns       0.00 ns              5     RELEASE
      0.00       0.00 ns       0.00 ns       0.00 ns              6  RELEASEDIR
      0.67  201960.00 ns  201960.00 ns  201960.00 ns              1    SETXATTR
      0.83  251334.00 ns  251334.00 ns  251334.00 ns              1     SETATTR
      1.04   78542.25 ns   71584.00 ns   84417.00 ns              4     INODELK
      1.06   79563.00 ns   69709.00 ns   87334.00 ns              4    FINODELK
      1.30  195209.50 ns  192210.00 ns  198209.00 ns              2     OPENDIR
      1.71   86062.83 ns   59709.00 ns  107876.00 ns              6     ENTRYLK
      2.03  610753.00 ns  610753.00 ns  610753.00 ns              1       MKDIR
      2.47  148500.80 ns  130584.00 ns  181167.00 ns              5      STATFS
      2.76  118768.57 ns   80376.00 ns  159251.00 ns              7       FLUSH
      3.09  310446.00 ns  269710.00 ns  331585.00 ns              3        OPEN
      4.07  204452.33 ns  177793.00 ns  241209.00 ns              6    GETXATTR
      4.69  353324.50 ns  218043.00 ns  547294.00 ns              4    FXATTROP
      5.32  801212.50 ns  691295.00 ns  911130.00 ns              2      CREATE
      8.74  292570.74 ns  208751.00 ns  467252.00 ns              9        READ
     17.28  578178.67 ns  261418.00 ns  946422.00 ns              9       WRITE
     42.93  404081.34 ns  211668.00 ns  569711.00 ns             32      LOOKUP

  Start time: 1663841511 seconds
    End time: 1663841917 seconds
   Data Read: 1048576 bytes
Data Written: 1048581 bytes
  Throughput: 0 fops/s

Interval 1 Stats:
   Block Size:             131072b+ 
 No. of Reads:                    8 
No. of Writes:                    8 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 ns       0.00 ns       0.00 ns              3     RELEASE
      0.83   74917.00 ns   59709.00 ns   90125.00 ns              2     ENTRYLK
      0.90   81250.50 ns   75167.00 ns   87334.00 ns              2    FINODELK
      2.50  151389.67 ns  130584.00 ns  181167.00 ns              3      STATFS
      2.92  132646.75 ns   80376.00 ns  159251.00 ns              4       FLUSH
      3.57  323980.50 ns  242584.00 ns  405377.00 ns              2    FXATTROP
      3.60  217736.67 ns  189667.00 ns  241209.00 ns              3    GETXATTR
      3.64  330814.00 ns  330043.00 ns  331585.00 ns              2        OPEN
      3.81  691295.00 ns  691295.00 ns  691295.00 ns              1      CREATE
     14.51  292570.74 ns  208751.00 ns  467252.00 ns              9        READ
     27.22  617773.75 ns  386794.00 ns  946422.00 ns              8       WRITE
     36.50  441777.07 ns  256293.00 ns  522711.00 ns             15      LOOKUP

  Start time: 1663841720 seconds
    End time: 1663841917 seconds
   Data Read: 1048576 bytes
Data Written: 1048576 bytes
  Throughput: 0 fops/s

Throughput shouldn't be 0. Start and end times should be shown in a more user-friendly way IMO.

Also there are some inconsistencies with the data shown. For example: end time is always the time of the last fop processed, but start time is not the time of the first fop processed, so the throughput will be different depending whether the fops have been sent at the beginning of the interval or the end, even if the number of fops is the same.

mohit84 commented 1 year ago

I've done a quick test of the patch and it doesn't seem to work:

Brick: gf:/root/bricks/test_1
-----------------------------
Cumulative Stats:
   Block Size:                  4b+              131072b+ 
 No. of Reads:                    0                     8 
No. of Writes:                    1                     8 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 ns       0.00 ns       0.00 ns              5     RELEASE
      0.00       0.00 ns       0.00 ns       0.00 ns              6  RELEASEDIR
      0.67  201960.00 ns  201960.00 ns  201960.00 ns              1    SETXATTR
      0.83  251334.00 ns  251334.00 ns  251334.00 ns              1     SETATTR
      1.04   78542.25 ns   71584.00 ns   84417.00 ns              4     INODELK
      1.06   79563.00 ns   69709.00 ns   87334.00 ns              4    FINODELK
      1.30  195209.50 ns  192210.00 ns  198209.00 ns              2     OPENDIR
      1.71   86062.83 ns   59709.00 ns  107876.00 ns              6     ENTRYLK
      2.03  610753.00 ns  610753.00 ns  610753.00 ns              1       MKDIR
      2.47  148500.80 ns  130584.00 ns  181167.00 ns              5      STATFS
      2.76  118768.57 ns   80376.00 ns  159251.00 ns              7       FLUSH
      3.09  310446.00 ns  269710.00 ns  331585.00 ns              3        OPEN
      4.07  204452.33 ns  177793.00 ns  241209.00 ns              6    GETXATTR
      4.69  353324.50 ns  218043.00 ns  547294.00 ns              4    FXATTROP
      5.32  801212.50 ns  691295.00 ns  911130.00 ns              2      CREATE
      8.74  292570.74 ns  208751.00 ns  467252.00 ns              9        READ
     17.28  578178.67 ns  261418.00 ns  946422.00 ns              9       WRITE
     42.93  404081.34 ns  211668.00 ns  569711.00 ns             32      LOOKUP

  Start time: 1663841511 seconds
    End time: 1663841917 seconds
   Data Read: 1048576 bytes
Data Written: 1048581 bytes
  Throughput: 0 fops/s

Interval 1 Stats:
   Block Size:             131072b+ 
 No. of Reads:                    8 
No. of Writes:                    8 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 ns       0.00 ns       0.00 ns              3     RELEASE
      0.83   74917.00 ns   59709.00 ns   90125.00 ns              2     ENTRYLK
      0.90   81250.50 ns   75167.00 ns   87334.00 ns              2    FINODELK
      2.50  151389.67 ns  130584.00 ns  181167.00 ns              3      STATFS
      2.92  132646.75 ns   80376.00 ns  159251.00 ns              4       FLUSH
      3.57  323980.50 ns  242584.00 ns  405377.00 ns              2    FXATTROP
      3.60  217736.67 ns  189667.00 ns  241209.00 ns              3    GETXATTR
      3.64  330814.00 ns  330043.00 ns  331585.00 ns              2        OPEN
      3.81  691295.00 ns  691295.00 ns  691295.00 ns              1      CREATE
     14.51  292570.74 ns  208751.00 ns  467252.00 ns              9        READ
     27.22  617773.75 ns  386794.00 ns  946422.00 ns              8       WRITE
     36.50  441777.07 ns  256293.00 ns  522711.00 ns             15      LOOKUP

  Start time: 1663841720 seconds
    End time: 1663841917 seconds
   Data Read: 1048576 bytes
Data Written: 1048576 bytes
  Throughput: 0 fops/s

Throughput shouldn't be 0. Start and end times should be shown in a more user-friendly way IMO.

Also there are some inconsistencies with the data shown. For example: end time is always the time of the last fop processed, but start time is not the time of the first fop processed, so the throughput will be different depending whether the fops have been sent at the beginning of the interval or the end, even if the number of fops is the same.

If total_fops < (end_time - start_time) then throughput will be 0.

mohit84 commented 1 year ago

Start time

I will check more about start time also.

mohit84 commented 1 year ago

I've done a quick test of the patch and it doesn't seem to work:

Brick: gf:/root/bricks/test_1
-----------------------------
Cumulative Stats:
   Block Size:                  4b+              131072b+ 
 No. of Reads:                    0                     8 
No. of Writes:                    1                     8 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 ns       0.00 ns       0.00 ns              5     RELEASE
      0.00       0.00 ns       0.00 ns       0.00 ns              6  RELEASEDIR
      0.67  201960.00 ns  201960.00 ns  201960.00 ns              1    SETXATTR
      0.83  251334.00 ns  251334.00 ns  251334.00 ns              1     SETATTR
      1.04   78542.25 ns   71584.00 ns   84417.00 ns              4     INODELK
      1.06   79563.00 ns   69709.00 ns   87334.00 ns              4    FINODELK
      1.30  195209.50 ns  192210.00 ns  198209.00 ns              2     OPENDIR
      1.71   86062.83 ns   59709.00 ns  107876.00 ns              6     ENTRYLK
      2.03  610753.00 ns  610753.00 ns  610753.00 ns              1       MKDIR
      2.47  148500.80 ns  130584.00 ns  181167.00 ns              5      STATFS
      2.76  118768.57 ns   80376.00 ns  159251.00 ns              7       FLUSH
      3.09  310446.00 ns  269710.00 ns  331585.00 ns              3        OPEN
      4.07  204452.33 ns  177793.00 ns  241209.00 ns              6    GETXATTR
      4.69  353324.50 ns  218043.00 ns  547294.00 ns              4    FXATTROP
      5.32  801212.50 ns  691295.00 ns  911130.00 ns              2      CREATE
      8.74  292570.74 ns  208751.00 ns  467252.00 ns              9        READ
     17.28  578178.67 ns  261418.00 ns  946422.00 ns              9       WRITE
     42.93  404081.34 ns  211668.00 ns  569711.00 ns             32      LOOKUP

  Start time: 1663841511 seconds
    End time: 1663841917 seconds
   Data Read: 1048576 bytes
Data Written: 1048581 bytes
  Throughput: 0 fops/s

Interval 1 Stats:
   Block Size:             131072b+ 
 No. of Reads:                    8 
No. of Writes:                    8 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 ns       0.00 ns       0.00 ns              3     RELEASE
      0.83   74917.00 ns   59709.00 ns   90125.00 ns              2     ENTRYLK
      0.90   81250.50 ns   75167.00 ns   87334.00 ns              2    FINODELK
      2.50  151389.67 ns  130584.00 ns  181167.00 ns              3      STATFS
      2.92  132646.75 ns   80376.00 ns  159251.00 ns              4       FLUSH
      3.57  323980.50 ns  242584.00 ns  405377.00 ns              2    FXATTROP
      3.60  217736.67 ns  189667.00 ns  241209.00 ns              3    GETXATTR
      3.64  330814.00 ns  330043.00 ns  331585.00 ns              2        OPEN
      3.81  691295.00 ns  691295.00 ns  691295.00 ns              1      CREATE
     14.51  292570.74 ns  208751.00 ns  467252.00 ns              9        READ
     27.22  617773.75 ns  386794.00 ns  946422.00 ns              8       WRITE
     36.50  441777.07 ns  256293.00 ns  522711.00 ns             15      LOOKUP

  Start time: 1663841720 seconds
    End time: 1663841917 seconds
   Data Read: 1048576 bytes
Data Written: 1048576 bytes
  Throughput: 0 fops/s

Throughput shouldn't be 0. Start and end times should be shown in a more user-friendly way IMO.

Also there are some inconsistencies with the data shown. For example: end time is always the time of the last fop processed, but start time is not the time of the first fop processed, so the throughput will be different depending whether the fops have been sent at the beginning of the interval or the end, even if the number of fops is the same.

DONE

mohit84 commented 1 year ago

/run regression

gluster-ant commented 1 year ago

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 608dbbf94..e845721a7 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);

-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;

         GF_FREE(local->get_vol.volname);