Join the recovery threads.

[mojoX911]

Problem

In the Maker server, we sometimes launch the recovery thread. But we don't join them after completion. They should be joined.

Ex: https://github.com/citadel-tech/coinswap/blob/bab961479c780ea7ee8402772e83b942e5c443d0/src/maker/api.rs#L478-L483

Approach.

• Make a globally accessible thread pool in the Maker struct definition.
• Add the recovery thread as well as all other thread in this thread pool.
• Join all the threads from this pool at shutdown.

[Shourya742]

Hey @mojoX911, I was thinking about the issue with the unjoined recovery threads in the Maker server. Instead of manually joining them, we could use scoped threads to handle this more cleanly. Using scoped threads would ensure that any threads we spawn would be guaranteed to complete within the scope they're created, and we wouldn’t need to manually track or join them at shutdown. What do you thoughts on this?

[mojoX911]

If it works and keeps the behavior intact I am fine with it.

My only concern is, as the recovery thread takes a very long time to return (144 blocks of waiting), it will surely last longer than the calling function. I don't know if in case of scope threads it will make the calling function blocking on the spawned thread? If it does then it would hang the rest of the app until recovery finishes.

Whereas currently we have an async type behavior, where recovery can keep happening in the background, while all other app APIs remain operational.

[KnowWhoami]

I don't know if in case of scope threads it will make the calling function blocking on the spawned thread?

Yes, the scoped threads will block the calling function -> so IMO, we should not them in case, when we are spawning normal threads which are meant to do their task asynchronously. But there are several instance where we are waiting just after spawning a normal thread -> so in that case we can use these scoped ones.

[mojoX911]

But there are several instance where we are waiting just after spawning a normal thread

Seems like all the threads in start_maker_server can be scoped? @Shourya742 ?

[KnowWhoami]

Seems like all the threads in start_maker_server can be scoped?

No It can't be because:

• scoped threads are used to create temporary threads that must complete thier work before a certain point in the process i.e the process will wait for their conclusion.

• In start_maker_server api -> we spawn various threads like checking Bitcoin Core Connection , rpc_server etc which must run asyncrounsly

• So using scoped threads for these things will halt our process since they will conclude only after the maker server is flagged to shutdown.

But there are several instance where we are waiting just after spawning a normal thread -> so in that case we can use these scoped ones.

Upon further reflection, I concluded that the choice between normal threads and scoped threads depends on the number of threads being used:

• Single thread:
  In cases like this, where we only need a single thread, threads are not strictly necessary. We can achieve the desired behavior in blocking mode without introducing additional complexity.

• Multiple threads:
  When working with multiple threads, using scoped threads is preferable. They provide better control and ensure that all threads are properly joined, reducing potential issues with thread safety and cleanup.

[mojoX911]

I would say for now, we handle the joining of the recovery thread the basic way. Add it to a global threadpool which always joins at main server scope. Solves the problem at hand.

I am turning this into 1.0 as getting it done is not that hard. And without this, we can currently have dangling recovery threads, or we won't see panics from the recovery. Thats pretty bad.

later on, we can explore scoped threads more and track it via another issue. Redoing the server architecture better is in scope for 1.1.