search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.49k stars 1.38k forks source link

Extreme JetStream write performance degredation when subject counts are limited #4043

Closed kung-foo closed 1 year ago

kung-foo commented 1 year ago

Defect

Writes to JetStream seem to slow down drastically when there is a constrained number of subjects and there is a size/count/ttl limit applied.

For example, with a stream limited to 100k messages and writing 1m messages

test MsgsPerSec
single subject 52414
all unique subjects 30191
16 subjects 37574
256 subjects 8984
1024 subjects: 2417

Versions of nats-server and affected client libraries used:

Tested with 2.9.6 and 2.10.0-beta.30 (via nats cli)

OS/Container environment:

Steps or code to reproduce the issue:

$ export NATS_CONTEXT=nats_development
$ rm -rf ~/.local/share/nats/nats_development/jetstream
$ nats server run --jetstream --verbose
$ nats stream add --defaults --max-msgs=100000 --subjects 'foo.>' --max-bytes=1GB TEST
$ nats bench --pub=1 --js foo.bar --stream TEST --msgs=1000000 --size=200 --pubbatch=1024 --csv /dev/stdout --multisubject

Expected result:

Running with different --multisubjectmax values should result in similar performace

Actual result:

RunID,ClientID,MsgCount,MsgBytes,MsgsPerSec,BytesPerSec,DurationSecs
single subject,P0,2000000,200000000,52414,10482974.154080,19.078555
--multisubjectmax=0,P0,2000000,200000000,30191,6038270.572961,33.122067
--multisubjectmax=16,JoCx5tctLCR9yhX1yC7zzt,P0,2000000,200000000,37574,7514918.051133,26.613730
--multisubjectmax=256,P0,2000000,200000000,8984,1796898.424830,111.302897
--multisubjectmax=1024,P0,2000000,200000000,2417,483465.501359,413.679982
kung-foo commented 1 year ago

Possibly related to https://github.com/nats-io/nats-server/issues/3948

derekcollison commented 1 year ago

What does nats stream info show for the stream in question when the performance is the slowest?

kung-foo commented 1 year ago 
❯ nats stream info TEST
Information for Stream TEST created 2023-04-12 21:29:22

             Subjects: foo.>
             Replicas: 1
              Storage: Memory

Options:

            Retention: Limits
     Acknowledgments: true
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false

Limits:

     Maximum Messages: 100,000
  Maximum Per Subject: unlimited
        Maximum Bytes: 1.0 GiB
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

State:

             Messages: 100,000
                Bytes: 22 MiB
             FirstSeq: 462,655 @ 2023-04-12T19:31:53 UTC
              LastSeq: 562,654 @ 2023-04-12T19:32:30 UTC
     Active Consumers: 0
   Number of Subjects: 4,096
derekcollison commented 1 year ago

ok yes I understand what the issue is, apologies for needing to jog my memory..

Not sure there is much we can do for 2.9.16, but are looking to improve for 2.10.