Traffic limitation in transfering data between different node_mgrs

[jd-zhang]

Issue migrated from trac ticket # 497 www.kunlunbase.com

component: node manager | priority: major

2022-03-11 14:55:55: snow@zettadb.com created the issue

---

Data transfer between different node_mgrs is the fundementally function.To achive the gole of using physical resource precisly,we implemented the Traffic limitaion mechanism.\ == How To\ BRPC( HTTP ProgressiveAttachment )is used as the chassis of the network framework, the mechanism we implementd is base on it.\ === Server side We created a File service which residens in the BRPC server which means the file-service listen the same unique Port. \ The file data which is required by client is sent by chunked stream after the HTTP header is recived by the client. That means the client side need revcive data stream continually until End-Of-Data is detected. ==== principle of the traffic control The standard of estimating the network traffic is widly accepted in Second Level, so we control the number of sent bytes in second interval.\ pseudocode is described below:\

Traffic control:
  {{{#!c++

    for (;;) {
    ret = read(fd, buff, SEND_BUFFER_SIZE(1024B));
    if (ret < 0) {
      Error break;
    } else if (ret == 0) {
      Finish break;
    } else {
      for (;;) {
        if (timer.timeout(n microsecond)) {
          // do check
          inner_check_counter++;
          if (inner_check_counter >= (1000000 / n)) {
            // traffic control under seconde level
            inner_check_counter = 0;
            bytes_send_counter = 0;
          }
          if (bytes_send_counter >= bytes_per_sec) {
            // stop and wait in the rest of the one seconde long
            sleep(((1000000 /n) - inner_check_counter) * n);
            inner_check_counter = 0;
            bytes_send_counter = 0;
            continue;
          }
        }
        Write(buff, ret) < 0) ;
        bytes_send_counter += ret;
        finish break;
      }
    }
  }

}}} === Client side We implementd a new tool residens in the util folders called download_file，which tell server the speed to send data.\ the parameters is listed below:\

  {{{#!c++
    -out_filename (Downloaded File name) type: string default: ""
    -out_prefix (Downloaded File path prefix) type: string default: ""
    -output_override (Whether override the output file if destination has the
      same name or not) type: bool default: false
    -traffic_limit (Download traffic limit, default is 5242880(5 MB))
      type: int64 default: 5242880
    -url (URL of the request reource) type: string default: ""

}}}

Here is how the traffic control performing.\

1. node_mgr1 want to the file F1 residens on the node_mgr2. \
2. node_mgr1 use download_file tool to download file F1 from node_mgr2 and specify the speed by traffic_limit para.\
3. node_mgr2 send file F1 in the speed of traffic_limit. \
4. node_mgr1 recived F1.

[jd-zhang]

2022-03-11 15:02:53: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 15:18:03: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 15:18:52: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 15:29:12: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 15:32:57: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 15:33:30: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 15:35:30: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 15:48:15: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 16:16:23: snow@zettadb.com edited the issue description

[jd-zhang]

2022-03-11 16:16:51: snow@zettadb.com changed title from Traffic limit in transfering data between different node_mgrs to Traffic limitation in transfering data between different node_mgrs

[jd-zhang]

2022-03-11 16:18:03: snow@zettadb.com changed component from DBA GUI to node manager

[jd-zhang]

2022-03-11 16:18:03: snow@zettadb.com

[jd-zhang]

2022-03-11 16:18:03: snow@zettadb.com set owner to all

[jd-zhang]

2022-03-11 16:18:03: snow@zettadb.com set resolution to finished

[jd-zhang]

2022-03-11 16:18:03: snow@zettadb.com changed severity from trivial to normal

[jd-zhang]

2022-03-11 16:18:03: snow@zettadb.com changed status from new to closed

[jd-zhang]

2022-03-11 16:18:03: snow@zettadb.com changed type from defect to feature

[jd-zhang]

2022-03-11 16:23:24: snow@zettadb.com removed resolution (was finished)

[jd-zhang]

2022-03-11 16:23:24: snow@zettadb.com changed status from closed to reopened

[jd-zhang]

2022-03-14 10:07:05: snow@zettadb.com changed owner from all to snow

[jd-zhang]

2022-03-14 10:07:05: snow@zettadb.com changed status from reopened to accepted

[jd-zhang]

2022-04-06 18:21:11: snow@zettadb.com