nats-io / nats.rs

Rust client for NATS, the cloud native messaging system.
Apache License 2.0
980 stars 159 forks source link

Hard to debug slow consumers with Rust client #1261

Open nazar-pc opened 1 month ago

nazar-pc commented 1 month ago

Proposed change

sid generated by subscription doesn't seem to be exposed anywhere publicly. Without it debugging things like this is very hard:

2024-05-13T10:03:05.357440Z  INFO async_nats: event: slow consumers for subscription 1

It would be more helpful if it was printing subscription subject or something, but sid generated internally that is not possible to access (except using std::fmt::Debug that will print it along with all other things).

Use case

Debugging slow consumers and general logging

Contribution

No response

nazar-pc commented 1 month ago

On a similar note I have a pool of connections to NATS, I have no idea what connection it belongs to either. Just like I have no idea what connections correspond to messages like this:

2024-05-10T23:05:19.230236Z  INFO async_nats: event: connected

Or how to map them to applications and connections with logs like this:

[1] 2024/05/13 09:12:18.908182 [INF] 172.20.30.1:51700 - cid:35 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 66 chunks of 4194972 total bytes.
Jarema commented 1 month ago

Adding the subject does not solve the problem in its entirety, as you might have many subscription for it.

One of the solutions is to add sid info to every subscriber, but that forces users to find and match subscriptions. The best solution seems to be able to hook up a callback/events to a specific subscription/consumers. That however is different approach from how other libraries are doing this.

We are discussing if we can address it on more cross-library general level with solution like above.

nazar-pc commented 1 month ago

Adding the subject does not solve the problem in its entirety, as you might have many subscription for it.

Agree, but it does narrow down things significantly.

The best solution seems to be able to hook up a callback/events to a specific subscription/consumers. That however is different approach from how other libraries are doing this.

Does seem to be the most accurate indeed.

We are discussing if we can address it on more cross-library general level with solution like above.

Nice, anything that helps matching connections/subscriptions with logs on both library and NATS server level will be highly appreciated. I recall I was able to specify client name in NATS streaming a few years ago in TypeScript, but situation seems to be significantly different here with clients seemingly identified by CID.