search

autonomys / subspace

Subspace Network reference implementation
https://subspace.network
382 stars 243 forks source link

Optimize farm and cache identification #2900

Open nazar-pc opened 4 months ago

nazar-pc commented 4 months ago

Currently farming cluster operates on farms and caches without any knowledge of "farmer".

This works and is flexible, but results in a bit too many notifications from identification messages, which https://github.com/subspace/subspace/pull/2899 improve a little bit.

It might be beneficial to also associate farms with farmers (and similarly for caches), just some ephemeral identifier, such that we can reduce identification broadcast from one per farm to one per farmer most of the time.

Farm identification as such should likely be replaced with a stream request such that farmer can send a stream of responses back to controller at a sustainable rate rather than bursting potentially thousands of messages all at once that might be dropped by NATS server due to buffer limits.

tediou5 commented 3 months ago

let me work on this. Its looks easy

tediou5 commented 2 months ago

@nazar-pc I have an experimental implementation repo, but here i have a problem.

In the https://github.com/autonomys/subspace/pull/2945#issuecomment-2255576651, we mentioned support quick farmer restarts. But while coding, I found out that the farm's fingerprints include randomly generated ids each time. So how do we implement this feature? (I've already finished the rest of farmer, so you can take a look at it in advance

nazar-pc commented 1 month ago

But while coding, I found out that the farm's fingerprints include randomly generated ids each time. So how do we implement this feature?

Check how we generate fingerprints right now. Essentially fingerprint of the farmer would be a generalization of the fingerprints of individual farms. The idea is to be able to tell whether it is the same farmer with the same set of farms in the same state or if some changes have happened to it during restart.

tediou5 commented 1 month ago

@nazar-pc What do you think about me generating the fingerprint through the hash of plotted_sectors.

My current implementation is generate fingerprints from a randomly farmer ID for each startup and total_sectors_count for each farm , The concrete realisation is here.

Or I can support quick restarts in another pr?

nazar-pc commented 1 month ago

Have you see what is currently used as fingerprint? You can literally do the same thing and hash it together.