microsoft / garnet

Garnet is a remote cache-store from Microsoft Research that offers strong performance (throughput and latency), scalability, storage, recovery, cluster sharding, key migration, and replication features. Garnet can work with existing Redis clients.
https://microsoft.github.io/garnet/
MIT License
10.37k stars 525 forks source link

Publish (still) does not work after a given data size if there are subscribers #790

Closed qcz closed 1 week ago

qcz commented 1 week ago

Describe the bug

I've reported a pub/sub related issue in #787. I've tested the fixed version and while the error message itself is fixed, publishing large messages (larger than 131 061 bytes) still don't work:

Steps to reproduce the bug

  1. Use the following configuration:

    {
    "MemorySize": "4g",
    "PageSize": "128m",
    "PubSubPageSize": "128m",
    "IndexSize": "128m",
    "DisableObjects": true,
    "CompactionType" : "Lookup",
    "LogLevel": "Information",
    }
  2. Subscribe using StackExchange.Redis or redis-cli:

var redis = ConnectionMultiplexer.Connect(ConfigurationOptions.Parse($"localhost:6379"));
redis.GetSubscriber().Subscribe(RedisChannel.Literal("test")).OnMessage((msg) =>
{
    byte[]? data = msg.Message;
    Console.WriteLine($"Received {data?.Length ?? 0} bytes");
});
  1. Use the following code to publish to redis:
var db = connectionMultiplexer.GetDatabase();
db.Publish(RedisChannel.Literal("test"), RandomNumberGenerator.GetBytes(1024)); // works
db.Publish(RedisChannel.Literal("test"), RandomNumberGenerator.GetBytes(127 * 1024)); // still works
db.Publish(RedisChannel.Literal("test"), RandomNumberGenerator.GetBytes(1024 * 1024)); // does not work
db.Publish(RedisChannel.Literal("test"), RandomNumberGenerator.GetBytes(1024)); // does not work, even if it is not larger than 131 061 bytes, because it was sent after a large message

Expected behavior

All pub/sub messages arrive which fits into PubSubPageSize and received by the subscriber.

Screenshots

No response

Release version

main branch (652b49f2a192cde004e6a35c27b9738f014e2cf0)

IDE

Visual Studio 2022

OS version

Windows 11 Pro

Additional context

No response

badrishc commented 1 week ago

OK, tracked the issue. Please check the linked PR.