How to run the program on two terminals for WAN network simulation?

[Bankee-256]

Hi! I am trying to run your program on two terminals to adapt to WAN network conditions. I've used the tc command to limit the network bandwidth, but it seems that running bazel-bin/examples/upsi/upsi alone doesn’t rely on the network. Do you have any recommendations on how to run this program on two terminals to better simulate and adapt to a WAN environment?

[ShallMate]

You can start the network by calling "SetupBrpcWorld" instead of "SetupWorld". This will start the local ports 10086 and 10087. Note that the traffic volume does not seem to be accurate when using SetupBrpcWorld (so we do not use SetupBrpcWorld in this repo), but it is enough to simulate a WAN using "tc". The traffic volume is shown in the value when calling "SetupWorld". @Bankee-256

[ShallMate]

@Bankee-256 The following indicates that the UPSI in a real network has been successful.

[Bankee-256]

Thanks for the guidance! I was able to run the program under WAN conditions. However, when I set the bandwidth to 200Mbit and the delay to 40ms, I noticed that the UPSI time feels unusually long, as highlighted in the red box in the attached image. Could you help me understand what might be causing this extended runtime?

[ShallMate]

It's normal in my local area. I don't know what went wrong at the moment because the information you provided is limited. I will try debugging to find out why you are having problems. @Bankee-256

[Bankee-256]

I’m running the program in a Docker container (shallmate/upsi). I made a modification in main.cc, replacing SetupWorld with SetupBrpcWorld.

When building, I initially encountered an issue using bazel build --linkopt=-ldl //..., so I switched to bazel build --linkopt="-L/usr/lib64" --linkopt="-ldl" --cxxopt="-std=c++17" //... --experimental_cc_shared_library, which allowed the build to complete successfully.

After applying the tc command to set bandwidth and delay as discussed, I ran ./bazel-bin/examples/upsi/upsi. However, the UPSI time in the output appears quite high.

[ShallMate]

void RunUPSI() { const uint64_t num = 1 << 22; const uint64_t addnum = 1 << 8; const uint64_t subnum = 1 << 8; SPDLOG_INFO("|X| = |Y|: {}", num); SPDLOG_INFO("|X^+| = |Y^+|: {}", addnum); SPDLOG_INFO("|X^-| = |Y^-|: {}", subnum); size_t bin_size = num; size_t weight = 3; size_t ssp = 40; okvs::Baxos baxos; yacl::crypto::Prg prng(yacl::crypto::FastRandU128()); uint128_t seed; prng.Fill(absl::MakeSpan(&seed, 1)); SPDLOG_INFO("items_num:{}, bin_size:{}", num, bin_size); baxos.Init(num, bin_size, weight, ssp, okvs::PaxosParam::DenseType::GF128, seed);

SPDLOG_INFO("baxos.size(): {}", baxos.size());

std::vector X = CreateRangeItems(addnum, num); std::vector Y = CreateRangeItems(addnum, num); std::vector Xadd = CreateRangeItems(0, addnum); std::vector Yadd = CreateRangeItems(0, addnum); std::vector Xsub = CreateRangeItems(num - subnum, subnum); std::vector Ysub = CreateRangeItems(num - subnum, subnum); auto lctxs = yacl::link::test::SetupBrpcWorld(2); // setup network //lctxs[0]->SetRecvTimeout(600000); //lctxs[1]->SetRecvTimeout(600000); std::cout<<"Base PSI start"<<std::endl; auto start_time_base = std::chrono::high_resolution_clock::now(); std::future<std::vector> rr22_sender = std::async( std::launch::async, [&] { return BasePsiSend(lctxs[0], X, baxos); }); std::future<std::vector> rr22_receiver = std::async( std::launch::async, [&] { return BasePsiRecv(lctxs[1], Y, baxos); }); auto psi_result_sender = rr22_sender.get(); auto psi_result = rr22_receiver.get(); auto end_time_base = std::chrono::high_resolution_clock::now(); std::chrono::duration duration_base = end_time_base - start_time_base; std::cout << "Base PSI time: " << duration_base.count() << " seconds" << std::endl; std::set intersection_sender(psi_result_sender.begin(), psi_result_sender.end()); std::set intersection_receiver(psi_result.begin(), psi_result.end()); std::cout<<"Base PSI intersection size = "<<intersection_receiver.size()<<std::endl; if (intersection_sender == intersection_receiver) { std::cout << "The base PSI finish." << std::endl; } else { std::cout << "The base PSI error." << std::endl; } auto bytesToMB = [](size_t bytes) -> double { return static_cast(bytes) / (1024 * 1024); };

auto sender_stats = lctxs[0]->GetStats(); auto receiver_stats = lctxs[1]->GetStats(); std::cout << "Base PSI Sender sent bytes: " << bytesToMB(sender_stats->sent_bytes.load()) << " MB" << std::endl; std::cout << "Base PSI Sender received bytes: " << bytesToMB(sender_stats->recv_bytes.load()) << " MB" << std::endl; std::cout << "Base PSI Receiver sent bytes: " << bytesToMB(receiver_stats->sent_bytes.load()) << " MB" << std::endl; std::cout << "Base PSI Receiver received bytes: " << bytesToMB(receiver_stats->recv_bytes.load()) << " MB" << std::endl; std::cout << "Base PSI Total Communication: " << bytesToMB(receiver_stats->sent_bytes.load()) + bytesToMB(receiver_stats->recv_bytes.load()) << " MB" << std::endl; auto start_time = std::chrono::high_resolution_clock::now(); EcdhReceiver yaddreceiver; EcdhSender yaddsender; yaddsender.UpdatePRFs(absl::MakeSpan(X)); EcdhReceiver xaddreceiver; EcdhSender xaddsender; xaddsender.UpdatePRFs(absl::MakeSpan(Y)); auto end_time = std::chrono::high_resolution_clock::now(); std::chrono::duration duration = end_time - start_time; std::cout << "Setup time: " << duration.count() << " seconds" << std::endl; size_t c1 = sender_stats->sent_bytes.load(); size_t c2 = sender_stats->recv_bytes.load(); size_t c3 = receiver_stats->sent_bytes.load(); size_t c4 = receiver_stats->recv_bytes.load(); auto start_time1 = std::chrono::high_resolution_clock::now(); std::future<std::vector> upsisender = std::async(std::launch::async, [&] { return UPsiSend(lctxs[0], Y, Yadd, Ysub, yaddreceiver, xaddsender, intersection_sender); });

std::future<std::vector> upsireceiver = std::async(std::launch::async, [&] { return UPsiRecv(lctxs[1], X, Xadd, Xsub, xaddreceiver, yaddsender, intersection_receiver); }); auto upsi_result_sender = upsisender.get(); auto upsi_result_receiver = upsireceiver.get(); auto end_time1 = std::chrono::high_resolution_clock::now(); std::chrono::duration duration1 = end_time1 - start_time1; std::set upsi_intersection_sender(upsi_result_sender.begin(), upsi_result_sender.end()); std::set upsi_intersection_receiver(upsi_result_receiver.begin(), upsi_result_receiver.end()); if (upsi_result_sender == upsi_result_receiver) { std::cout << "The uPSI finish." << std::endl; } else { std::cout << "The uPSI error." << std::endl; } std::cout << "UPSI time: " << duration1.count() << " seconds" << std::endl; auto sender_stats1 = lctxs[0]->GetStats(); auto receiver_stats1 = lctxs[1]->GetStats(); size_t c5 = sender_stats1->sent_bytes.load() - c1; size_t c6 = sender_stats1->recv_bytes.load() - c2; size_t c7 = receiver_stats1->sent_bytes.load() - c3; size_t c8 = receiver_stats1->recv_bytes.load() - c4; std::cout << "UPSI Sender sent bytes: " << bytesToMB(c5) << " MB" << std::endl; std::cout << "UPSI Sender received bytes: " << bytesToMB(c6) << " MB" << std::endl; std::cout << "UPSI Receiver sent bytes: " << bytesToMB(c7) << " MB" << std::endl; std::cout << "UPSI Receiver received bytes: " << bytesToMB(c8) << " MB" << std::endl; std::cout << "UPSI Total Communication: " << bytesToMB(c5) + bytesToMB(c6) << " MB" << std::endl; }

You can try changing RunUPSI to the above. I think it may solve your problem. @Bankee-256

[Bankee-256]

Thank you for the RunUPSI function! I made the suggested modifications, changing static_cast(bytes) to static_cast<double>(bytes) and std::chrono::duration to std::chrono::duration<double>. However, I’m still encountering the same issue with high UPSI times. I suspect that network delay may be contributing to the problem, but I'm unsure of the best way to address or mitigate this. Any additional guidance would be greatly appreciated!

[ShallMate]

Hello, you can clone the latest upsi code into the container and recompile it. I believe it will work properly. Previously, it seemed that the remote PSU implementation was inconsistent with my local one. The specific method is to delete the upsi in the examples directory, then clone it again in the examples directory, and recompile it. @Bankee-256

[Bankee-256]

Thank you again for your help! The code runs well with an initial set size $\geq 2^{20}$. However, when the initial set size is below $2^{20}$, the issue reoccurs, especially when network delay is added. I’m still encountering the same timeout error:

terminate called after throwing an instance of 'yacl::IoError'
  what():  [yacl/link/transport/channel.cc:427] Get data timeout, key=root:P2P-37:0->1
Aborted (core dumped)