SimaticResearchActivity / FBAE

Framework for Broadcast Algorithms Evaluation
GNU Affero General Public License v3.0
0 stars 1 forks source link

For sending messages with Tcp communication layer, use 2 writes instead of 1 when messages are big #36

Closed simatic closed 2 weeks ago

simatic commented 5 months ago

In current Tcp::send(), we systematically do the following code to limit the number of write system calls:

    auto sWithLength = oStream.str();
    sWithLength.append(msg);
    boost::asio::write(*rank2sock[r], boost::asio::buffer(sWithLength.data(), sWithLength.length()));

This is suboptimal when size of msg is big (over 10'000 bytes?) as sWithLength.append(msg); allocates a new big chunk of memory and does a copy from msg to sWithLength. So we propose the following modification:

    auto sWithLength = oStream.str();
    if (msg.length() < ConstexprToBeDeterminedEmpirically) {
       sWithLength.append(msg);
       boost::asio::write(*rank2sock[r], boost::asio::buffer(sWithLength.data(), sWithLength.length()));
   } else {
       boost::asio::write(*rank2sock[r], boost::asio::buffer(sWithLength.data(), sWithLength.length()));
       boost::asio::write(*rank2sock[r], boost::asio::buffer(msg.data(), msg.length()));
   }

Note: The difference of performance has been observed during an experiment made with 2 participants on a 1 Gbps network (python3 launch_fbae.py /netfs/inf/simatic/FBAE/ /netfs/inf/simatic/FBAE/results/ -a B -c t -f 8000 -n 40000 -s 8192 -S /netfs/inf/simatic/FBAE/sites_2_b313.json -w 10 -m 1000000):

simatic commented 5 months ago

This issue has to be handled in conjunction with issue #44 .