search

LemmyNet / activitypub-federation-rust

High-level Rust library for the Activitypub protocol
GNU Affero General Public License v3.0
420 stars 45 forks source link

Add a no limit option to the config #45

Closed cetra3 closed 1 year ago

cetra3 commented 1 year ago

This is an extension to an earlier PR which allows you to set the retry & worker counts to 0 to have basically unlimited tasks. This does mean that if the server can't keep up with the outgoing requests then it will probably hit RAM limits (i.e, out of memory will occur).

Nutomic commented 1 year ago

Thanks!

phiresky commented 1 year ago

Just saw this PR. I want to say that this PR is not a good solution with the current queue, because not just does the join_set keep all running tasks in memory, it also keeps all finished tasks in memory if you don't set a limit.

Edit: I see that the join set is disabled when worker_count is 0. So that's better than I thought then, with worker count 0 shutdown() is broken but it doesn't have the kind of memory leak i thought.

cetra3 commented 1 year ago

Yeah we don't bother with the join set in the case it's unlimited for that reason. This does mean that the shutdown method doesn't work in that case. However it's not a public method, only really used in testing