kevinbcarpenter / restful-with-billions

Repository for abstract and code related to a talk.
MIT License
20 stars 1 forks source link

Seg fault in `restful_functional_test` if server not responding #5

Open oschonrock opened 11 months ago

oschonrock commented 11 months ago

For example, if server is not started at all... There should be a more graceful way to fail.

~/c/restful-with-billions/bin/Release$ ./restful_functional_test 
Segmentation fault (core dumped)
oschonrock commented 11 months ago

Interestingly some high concurrency load testing has shown that when the server gets overloaded and the client times out with a "Read" error, then the client would also segfault without a modification similar to the linked pull request.

The way to reproduce this for me was:

  1. comment out all operations but testPost
  2. TESTCOUNT = 100
  3. DELAY = 0

Then spawn a large number of clients:

for i in {1..100}; do bin/Debug/restful_functional_test &  done 

These numbers may need adjusting depending on your hardware.

And you will find that the server stores less than the expected 10,000 TXs and some of the clients segfault.

if we modify the code in the linked pull request along these lines:

auto testPost(httplib::Client &cli, detail::cardInfo ci) -> detail::cardInfo {
  json payload = ci;

  while (true) { // keep trying until success, maybe consider a max count
    auto res = cli.Post("/sale", payload.dump(), contentType);
    if (res) {
      detail::cardInfo rci = json::parse(res->body)[0];
      std::cout << "POST/CREATE - Status: " << res->status << std::endl
                << "Guid: " << rci.guid << std::endl
                << "Body: " << res->body << std::endl;

      return rci;
    } else {
      // server busy.. wait a bit
      std::this_thread::sleep_for(std::chrono::microseconds(10));
    }
  }
}

Then all 10,000 TXs get recorded and the clients don't segfault.

ie the cause of the segfaults was simply the accessing of res->body without checking if (bool)res was true, ie no error.