Flawky `Maximum Per Subject` retention policy

[ggarri]

Observed behavior

Hi team, We are facing an issue on jetstreams with the retention policy of max-msg-per-subject. We are setting a stream with 2k as max message per subject

Information for Stream notifications created 2023-12-07 06:15:34

              Subjects: notifications.>
              Replicas: 5
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: 2,000
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: nats-prod
                Leader: nats-prod-1
               Replica: nats-prod-0, current, seen 1ms ago
               Replica: nats-prod-2, current, seen 1ms ago
               Replica: nats-prod-3, current, seen 1ms ago
               Replica: nats-prod-4, current, seen 1ms ago

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 139,079
         Last Sequence: 139,078 @ 2023-12-08 08:52:12 UTC
      Active Consumers: 6

That is working well for a period of time but at a certain point the retention policy stops applying, and our storage starts growing to infinitive, as I can show below:

│      3 Subjects in stream notifications      │
├──────────────────────────────────────────────────┬─────────┤
│ Subject                                          │ Count   │
├──────────────────────────────────────────────────┼─────────┤
│ notifications.subjectOne                         │ 47,664  │
│ notifications.subjectTwo                         │ 47,687  │
│ notifications.subjectThree                       │ 828,103 │

Then, after one of our NATs instance restart or if we perform an update on the max-msg limit to something like 20k (any other value also works) messages per subjects goes down to 2k again and remain like that, at least for now. Is it this a know bug? is there any remediation we can set to prevent that? Thanks in advance for your help :slightly_smiling_face: (edited)

Expected behavior

Retention policy per subject should be strickly applied

Server and client version

nats:2.10.4-alpine

Host environment

Linux

Steps to reproduce

Run nats with multiple streams(40-50) with 100msg/s

[derekcollison]

Consider upgrading to the latest 2.10.7 which has some fixes on lookup per subject etc.

[derekcollison]

Any updates here? ok to close?

[ggarri]

It seems to be buggy in the last version, so we switched to restrict limits based on stream instead of subject to guarantee our flat storage growth.

[Jarema]

@ggarri what was the last version you tested this on? Could you share some more details about your setup and how stream is utilized so we can try reproducing it?