JetStream Max Store Size calculation should add back current store size

[caleblloyd]

Right now the JetStream Max Store Size calculation only takes into account Available disk. So each time you write data to JetStream and restart the server, it goes down:

https://github.com/nats-io/nats-server/blob/6e6ce3a6f65aed54bb7dea60d2a8ed42046a7627/server/disk_avail.go#L30C1-L33

Example:

I start with a 10GB disk and it calculates:

[16] 2023/05/09 15:49:46.241840 [INF] ---------------- JETSTREAM ----------------
[16] 2023/05/09 15:49:46.241842 [INF]   Max Memory:      0 B
[16] 2023/05/09 15:49:46.241845 [INF]   Max Storage:     7.25 GB
[16] 2023/05/09 15:49:46.241847 [INF]   Store Directory: "/data/jetstream"
[16] 2023/05/09 15:49:46.241848 [INF] -------------------------------------------

Then, I write 5GB of data to a stream. Now when I restart the server it calculates:

[15] 2023/05/09 16:24:36.663491 [INF] ---------------- JETSTREAM ----------------
[15] 2023/05/09 16:24:36.663494 [INF]   Max Memory:      0 B
[15] 2023/05/09 16:24:36.663497 [INF]   Max Storage:     3.37 GB
[15] 2023/05/09 16:24:36.663498 [INF]   Store Directory: "/data/jetstream"
[15] 2023/05/09 16:24:36.663499 [INF] -------------------------------------------

The 5GB stream is now over the Max Storage limit and cannot be written to anymore, although I could consume from it.

Also the 75% calculation is undocumented, JS Docs say Up to 1TB if available

Might be nice to have a few more knobs to turn, such as being able to set % of Disk and Fixed Overhead

[derekcollison]

We calculate based on available so I think that is correct, plus this is more for easy setup etc but does not reflect reality where the user should set this number in their config.

But do note that it will protect the system correctly in my opinion on restarts.

[bruth]

@caleblloyd Is this still a relevant issue?