`grapesy-kvstore` server cannot support the Java client (in JSON mode)

[edsko]

In #168 we add support for JSON, both in the main library and in the kvstore benchmark. Running

cabal run grapesy-kvstore -- --json --client

against the Java reference works just fine, but running the grapesy server against the reference client does not. When we slow down the client and only send a few requests, it works just fine, but when the client is operating at full speed, after the first bunch of TCP packets everything just stops.

To reproduce, you will need the gRPC Java reference from https://github.com/edsko/kvstore/tree/edsko/revive-gson , and then modify KvRunner.java to only run the client or the server.

It is possible that this is another manifestation of https://github.com/well-typed/grapesy/issues/162#issuecomment-2183070579 .

[FinleyMcIlwaine]

Reproduced on macOS

[FinleyMcIlwaine]

Confirmed that https://github.com/kazu-yamamoto/http2/pull/126 does not fix on macOS.

[FinleyMcIlwaine]

I was able to reproduce this with grapesy on its own, with and without --json. The grapesy server just freezes when we flood it with many requests very quickly. The java client does this because it does not wait for a response before executing the next request. Therefore, on startup, it is in a very tight loop of doing a bunch of creates before knownKeys has been populated:

void doClientWork(AtomicBoolean done) throws InterruptedException {
    ...
    while (/* not done and no errors */) {
      // Pick a random CRUD action to take.
      int command = random.nextInt(4);
      if (command == 0) {
        doCreate(channel, errors);
        continue;
      }
      synchronized (knownKeys) {
        // If we don't know about any keys, retry with a new random action.
        if (knownKeys.isEmpty()) {
          continue;
        }
      }
    ...

If the server takes long enough to respond, it completely exhausts the limiter semaphore with 100 creates at the beginning of the run. It does this when I run it against the grapesy server.

We can emulate this behavior in the grapesy kvstore client with something like this in Client.hs:

    withConnection params server $ \conn -> do
      ...
      mapConcurrently_
        ( \_ -> do
            doCreate kvstore knownKeys
            modifyIORef' statsVar incNumCreate
        )
        [0 .. 100]

This client causes the server to freeze up, with and without -N, even without --json! With -N and --json I am seeing it freeze up with as few as 32 concurrent creates.

[edsko]

This is due to https://github.com/kazu-yamamoto/network-control/pull/4 . Closing this, but we do then urgently need to do https://github.com/well-typed/grapesy/issues/145#issuecomment-2203382627 .