Relay hardware configuration

[VegeBun-csj]

Are there any recommended hardware configurations？ This is my configuration and grafana. Is there a memory leak？

[scsibug]

It would be great to see some metrics of the actual load (REQ/sec, EVENT/sec); as well as the database engine and size. The full config file helps too, since there are numerous settings in there that affect load. Using prometheus/grafana against the /metrics endpoint on the relay is a great way to pull in those relay metrics.

I'm not aware of any memory leaks, I'm running 0.8.7 right now with a 9GB database, for about 8 days with no issue. Over the last 24 hours, memory usage has been very stable, starting at 11.4GB active, ending at 11.2GB (min 10.9GB, max 11.6GB) - that's for Linux as a whole, so the relay is just part of that.

My main heavily-loaded relay runs on a dedicated 6-core AMD 64GB server with NVMe storage.

If this is a new relay with an empty database, then I agree something is wrong.

[VegeBun-csj]

Hi, sir. I improve the hardware, especially the cpu and mem, but the memory remains raising by the time. You can see the fig. It's very strange.

[VegeBun-csj]

@scsibug I still think the nostr-rs has a memory leak. i'm using the 0.8.8 with this grafana monitor

This is still a situation where the number of requests is very small and the memory continues to expand. Your recommended 64GB is also too large. I don't think the requirements for nostr should be as high as this. After all, it's just a simple server.

[scsibug]

64GB isn't recommended, it's just what I happen to be using. CPU core count just happened to come along with more memory than necessary. I've run for weeks with no issues and plenty of free memory - Note that i've also actually got something like a 20GB+ database now as I write this. We probably do want SQLite to have as much of that in memory as possible.

It's possible what you're seeing is fragmentation, not leaks. We should try swapping out the allocator for jemalloc and see what that behavior is like, perhaps that will keep things more constant. Will try that out soon-ish.

[scsibug]

OK, commit 80c459c36c88c9d90e313e032f545f2d67bfb294 switches to the jemalloc allocator; which should be better for heap fragmentation. I followed the simple advice here: https://www.svix.com/blog/heap-fragmentation-in-rust-applications/

Just deployed this to nostr-pub.wellorder.net, so I'll keep an eye on memory usage there. I wasn't able to duplicate the behavior @VegeBun-csj was seeing, but if you want to try with this latest commit as well, perhaps you can give some feedback as to whether it looks better.

I guess one other question would be, are you loading a lot of events into the database during the time period you are graphing? Obviously, if the database size is increasing quickly, I'd expect memory usage to go up as well, and that could look like a leak.

[VegeBun-csj]

OK, commit 80c459c switches to the jemalloc allocator; which should be better for heap fragmentation. I followed the simple advice here: https://www.svix.com/blog/heap-fragmentation-in-rust-applications/

Just deployed this to nostr-pub.wellorder.net, so I'll keep an eye on memory usage there. I wasn't able to duplicate the behavior @VegeBun-csj was seeing, but if you want to try with this latest commit as well, perhaps you can give some feedback as to whether it looks better.

I guess one other question would be, are you loading a lot of events into the database during the time period you are graphing? Obviously, if the database size is increasing quickly, I'd expect memory usage to go up as well, and that could look like a leak.

Thanks for your improvement. For the last question, my relay has only small request num, so it should't happen so increasing in mem.

[scsibug]

I am still tracking this - I am going to do some tests with small VMs running large databases and try to get a handle on the behavior (maybe try and see if the OOM killer will ever get involved). I think that we may want to tweak or make tweak-able the SQLite memory-map settings. SQLite mmap is (I assume) more efficient than using the linux page cache; but I'm not sure that it can't cause memory pressure issues in the long term. Just for monitoring purposes I'd rather have this show up under 'cached' memory instead of active/inactive.

[0xtrr]

hey @VegeBun-csj, do you have any tips/guides for setting up such a Grafana dashboard? I believe my relay gets a good load of traffic so would be nice to be able to surveil the resources a bit more. I have also had problems where my VM stops running/freezes because of an out of memory error (32 GB RAM) so I have suspected some kind of memory leak myself. I have not looked into it yet though.

[VegeBun-csj]

Hey bro, @0xtrr. I'm using the node explorer panel in Grafana. I recently upgraded to 0.8.9 (are you currently using 0.8.9?). It seems like the memory is not increasing as steeply as before, but the memory cache is still increasing. Our relay is currently not experiencing much traffic, so I need to continue monitoring its status. For comparison, I set up another relay using the Notream framework, and its memory usage is very low and its performance is stable. Therefore, we need to continue monitoring the status of Nostr-rs. If you are still experiencing memory leaks with 32GB of memory, it may indeed be a code issue. (Consider upgrading to 0.8.9) Below is my Grafana monitoring of nostr-rs. Red line is the previous version(0.8.8) and blue line is current version(0.8.9).

I believe that Rust's network services should not have such problems, because TypeScript can achieve such low memory usage. Rust itself is a high-performance programming language. This is the nostream relay's monitor:

You can see that the data is basically stable.

[VegeBun-csj]

@0xtrr hey bro, We can continue to monitor this issue together :)

[scsibug]

I suspect issues will be localized on sqlite and its caching behavior, probably not so much tokio/networking.

[VegeBun-csj]

This problem still exists

[scsibug]

I've just pushed another commit (04db2203bb646e4f42bddb3d0c63fdc76eccf1ca) which goes back to the standard allocator, and reduces the SQLite mmap settings from a max of 16GB, to 4GB.

I'm running this on nostr-pub.wellorder.net right now as a test to see if that reduces the active/inactive memory, and moves more to the linux page cache. If that goes well, I'll probably try reducing the mmap size to zero, and just force the page cache to be used. That is less efficient, but should improve memory pressure.

Will check back in here in about 48 hours to report any changes.

[VegeBun-csj]

I've just pushed another commit (04db220) which goes back to the standard allocator, and reduces the SQLite mmap settings from a max of 16GB, to 4GB.

I'm running this on nostr-pub.wellorder.net right now as a test to see if that reduces the active/inactive memory, and moves more to the linux page cache. If that goes well, I'll probably try reducing the mmap size to zero, and just force the page cache to be used. That is less efficient, but should improve memory pressure.

Will check back in here in about 48 hours to report any changes.

anything update? :)

[scsibug]

I will post new commits and details tomorrow, but yes, I have had some success. Running with mmap set now to zero, i have dramatically reduced the memory used by the relay with no noticeable performance impact.

[VegeBun-csj]

@scsibug Sounds great, looking forward to your attention to detail and solution.

[scsibug]

Here is a snippet of my memory graph for the operating system on the nostr-pub.wellorder.net host (which is fairly busy, with a ~20GB database), showing a prolonged period of very stable usage. A few notable things going on with this configuration:

• I'm not using jemalloc (just to keep things simple).
• I've got systemd set with a MemoryHigh setting of 20GB for the relay.
• I've set sqlite mmap to 0.
• The relay has been running for about 40 hours.
• 325K websocket connections, which created 5M subscriptions, submitted 400k+ events, and sent out 360M events to clients.
• Swap is unused.
• I couldn't notice any difference in overall system performance with the lack of sqlite mmap.

The sqlite mmap change is in commit 2bcddf8bbfe11a2db6419303fb8935e41687558d, which I think is going to give the host more flexibility with the page cache to manage memory pressure.

I am still planning to do experiments with low-memory situations in some VMs and see how much memory is truly needed to maintain respectable performance with large databases.

If you can give the latest commit a try as well, I'd be curious on anyone else's results. My server has 64GB of memory, and has well over half sitting unused, which is a significant difference from a couple weeks ago.

[VegeBun-csj]

Sounds great, i will start and monitor with this latest commit.

[scsibug]

Just pushed a few more changes to help memory-constrained servers. I have been doing some testing in a small VM (2GB RAM) with a 20GB database file. With a little benchmark script to simulate a few clients sending REQs non-stop, that is enough to duplicate conditions that would lead to memory exhaustion.

The idle timeout (to release sqlite connections) has been reduced from 10min to 10sec. This helps clean up memory faster. I've also changed the idle connection pool to default to zero connections, so on really idle relays that may help too.

Finally, I've re-enabled jemalloc as it did seem to be helpful in returning memory to the OS after the connections were cleaned up.

[scsibug]

Closing; I think we've made enough improvements at this point. More work to do obviously, but the current release seems stable for reasonably sized databases over reasonable amounts of time.