search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.51k stars 1.38k forks source link

Object Store times out (python SDK, CLI) #5622

Open mxchinegod opened 2 months ago

mxchinegod commented 2 months ago

Observed behavior

nats ls object my_object_store hangs on CLI, python SDK with my timeout calls off the request accordingly.

Expected behavior

list a bunch of PDFs I placed in the aforementioned object store bucket

Server and client version

0.1.1
nats:2.10.10-alpine

Host environment

Kubernetes cluster of two machines behind DNS.

Screenshot 2024-07-07 at 5 29 32 PM

Steps to reproduce

  1. Create object store with python SDK.
  2. Upload a bunch of PDFs with python SDK, named after hashes of their fullpaths.
  3. Download PDFs using python SDK to local disk on different machine (or same as the one used to upload, for testing purposes)
  4. experience timeouts in the object store commands, attempts to send data to normal js also times out in python SDK
wallyqs commented 2 months ago

do you see warnings in the server logs? are you able to reproduce this on v2.10.17?

mxchinegod commented 2 months ago

Nothing useful in the logs; although 

[7] 2024/07/07 21:52:06.054867 [INF] JetStream cluster new consumer leader for '$G > OBJ_my_object_store > DAcMLn1x'
[7] 2024/07/07 21:54:55.798334 [INF] JetStream cluster new consumer leader for '$G > OBJ_my_object_store > LD13MRJl'
[7] 2024/07/07 22:00:49.253497 [INF] JetStream cluster new consumer leader for '$G > OBJ_my_object_store > ooYy6C6X'
[7] 2024/07/07 22:00:49.310463 [INF] JetStream cluster new consumer leader for '$G > OBJ_my_object_store > spNBidYe'
[7] 2024/07/07 22:03:40.162075 [INF] JetStream cluster new consumer leader for '$G > OBJ_my_object_store > IGozDsOV'

that's pretty frequent.

aside from that, I am able to reproduce the issue when I change to 2.10.17-alpine