search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.49k stars 1.38k forks source link

Jetstream store cannot be created even if already exists in jetstream_store #3641

Closed siredmar closed 1 year ago

siredmar commented 1 year ago

Defect

Create and restore of an already present stream doesn't work if disk storage is low - even if the stream occupies its max size.

Precondition:

Runs on an embedded Linux device with limited disk resources. The local jetstream store already contains a stream (with 4 GB max size) completely full. Free disk space where the jetstream store is located is also around 4 GB. Performed a reboot and the nats-server is started again.

What happened?

During startup of the nats-server it complains about insufficient storage resources available: Looking at the source code of nats-server, if not defined, it calculates the max_storage by looking at the free disk space and take 75% of it. This explains the 2.94 GB max_storage (75% of ~4 GB is ~3 GB).

[87] 2022/11/16 12:55:12.039413 [INF] ---------------- JETSTREAM ----------------
[87] 2022/11/16 12:55:12.039448 [INF]   Max Memory:      1.45 GB
[87] 2022/11/16 12:55:12.039461 [INF]   Max Storage:     2.94 GB
[87] 2022/11/16 12:55:12.039469 [INF]   Store Directory: "/store/jetstream"
[87] 2022/11/16 12:55:12.039476 [INF]   Domain:          mydomain
[87] 2022/11/16 12:55:12.039482 [INF] -------------------------------------------
[87] 2022/11/16 12:55:12.039753 [INF] Standalone server started in clustered mode do not support extending domains
[87] 2022/11/16 12:55:12.039788 [INF] Manually disable standalone mode by setting the JetStream Option "extension_hint: will_extend"
[87] 2022/11/16 12:55:12.042931 [WRN]   Error recreating stream "mystream": insufficient storage resources available (10047)

Versions of nats-server and affected client libraries used:

NATS server version: 2.8.2-alpine3.15

Expected Result

I'd expect that the startup process takes into account that there already might be defined streams in the jetstream_store directory. So the pre-check during creating and restoring of the stream should check not only the max_storage, but also if the data of the stream to be created is already present. Thus, max_storage cannot be trusted as the only source of truth.

derekcollison commented 1 year ago

Can the issue be reproduced with a current version of the server? Version 2.9.6?

derekcollison commented 1 year ago

Any updates?

siredmar commented 1 year ago

I'm currently not able to test this with the latest version.

derekcollison commented 1 year ago

ok will close for now. Feel free to re-open if needed.