microsoft / garnet

Garnet is a remote cache-store from Microsoft Research that offers strong performance (throughput and latency), scalability, storage, recovery, cluster sharding, key migration, and replication features. Garnet can work with existing Redis clients.
https://microsoft.github.io/garnet/
MIT License
10.37k stars 525 forks source link

Publish does not work after a given data size if there are subscribers #787

Closed qcz closed 1 week ago

qcz commented 1 week ago

Describe the bug

I'm trying out the capabilities and limits of Garnet and whether I can replace Redis with it. Using the following PageSize configurations:

{
    "PageSize": "128m",
    "PubSubPageSize": "128m",
}

I can set keys with fairly large values and I can even publish fairly large pub/sub messages. Except if somebody is subscribed to the channel I am publishing to. Then if the message is longer than 128 * 1024 - 11 (131 061) bytes long, I get the following error:

ERR Garnet Exception: Failed to write to response buffer

In the log of Garnet I see the following:

01::34::57 crit: Session[0] [127.0.0.1:55402] [00EE8998] ProcessMessages threw a GarnetException: Garnet.common.GarnetException: Failed to write to response buffer
    at Garnet.common.GarnetException.Throw(String message, LogLevel logLevel)
    at Garnet.server.RespServerSession.Publish(Byte*& keyPtr, Int32 keyLength, Byte*& valPtr, Int32 valLength, Byte*& inputPtr, Int32 sid)
    at Garnet.server.SubscribeBroker`3.Broadcast(Byte[] key, Byte* valPtr, Int32 valLength, Boolean ascii)
    at Garnet.server.RespServerSession.NetworkPUBLISH()    at Garnet.server.RespServerSession.ProcessMessages()
    at Garnet.server.RespServerSession.TryConsumeMessages(Byte* reqBuffer, Int32 bytesReceived)

Note: I can get rid of this error by compiling an own version of Garnet and increasing the network buffer related limits in NetworkBufferSettings.cs, BufferSizeUtils.cs and MaxSizeSettings.cs. As I do not fully grasp the networking related code of Garnet, I don't know the consequences of changing them to new, fairly large numbers. However I don't think these changes should be required by an end user or we should be able to change these limits via configuration.

Steps to reproduce the bug

  1. Use the following configuration:
    {
    "MemorySize": "4g",
    "PageSize": "128m",
    "PubSubPageSize": "128m",
    "IndexSize": "128m",
    "DisableObjects": true,
    "CompactionType" : "Lookup",
    "LogLevel": "Information",
    }
  2. Publish a message using StackExchange.Redis to channel "test" containg 256 000 bytes. (With no subscribers). It works.
  3. Subscribe to the test channel (with e.g. redis-cli)
  4. Publish a message using StackExchange.Redis to channel "test" containg 256 000 bytes. (With no subscribers). It fails with the error above.

Expected behavior

The publish should work fine with messages allowed by the page size. Redis does not have any problems with these data sizes.

Screenshots

No response

Release version

v1.0.36

IDE

Visual Studio 2022

OS version

Windows 11 Pro

Additional context

No response

badrishc commented 1 week ago

Can you check the linked PR if it fixes the issue?