spike: Miner Selection Criteria for Validators

[Luka-Loncar]

Currently the approach we use to select miners takes a random set of UIDs and does not include a #64 that is required to ensure that every miner doesn't respond to every request on the network.

Outcomes

• [x] #92

Acceptance criteria

• [ ] We know a Miner selection criteria to adopt
• [ ] We have a follow-up card tracking the implementation

[mudler]

This is currently blocked by #64

[hide-on-bush-x]

Hey @mudler is there any reason this is blocked by #64 ? I have the feeling like this two things can go in parallel

[mudler]

Hey @mudler is there any reason this is blocked by #64 ? I have the feeling like this two things can go in parallel

we shouldn't let pick up miners which are blacklisted, hence why we need first to understand how we can handle blacklisting of miners before diving into this one which might depend on logic that we have yet still to discover

[hide-on-bush-x]

Oh, no no, there is a little difference Blacklist: Miners blocking validators requests Selection criteria: Validators selecting what miners to call

Thats why I believe this can be done in parallel

[mudler]

Oh, no no, there is a little difference Blacklist: Miners blocking validators requests Selection criteria: Validators selecting what miners to call

Thats why I believe this can be done in parallel

Sorry, could you expand? it's not entirely clear to me what do you mean here - from the description of the card, we want to clearly avoid picking up miners given a blacklist:

Currently the approach we use to select miners takes a random set of UIDs and does not include a https://github.com/masa-finance/masa-bittensor/issues/64 that is required to ensure that every miner doesn't respond to every request on the network.

[hide-on-bush-x]

Yeah, description seems to be a little bit off

Miners have a blacklist logic to define which validator ignore based on a vpermit ( by default ): https://github.com/masa-finance/masa-bittensor/blob/main/neurons/miner.py#L87

Validators on the other side have this selection criteria, which is used to pick the best miners given some criteria, this ticket is to explore posibilities around that, say "only calling miners that have more than xyz score on chain" "Call miners based on a whitelist" That logic will go somewhere in the forwarder close to the UID definition here: https://github.com/masa-finance/masa-bittensor/blob/main/masa/validator/forwarder.py#L31

Both things are similar but not dependant and whatever we find out on the blacklist spike does not impact this task

[mudler]

got it, thanks @hide-on-bush-x!

right, this doesn't look really a blocker then, and we can always go back at this if we want to include some sorta of logic around blacklisting miners when validators are going to pick these.

[hide-on-bush-x]

Thanks @mudler !

[juanmanso]

Moved out of On Blocked and taking the lead for these 2 hours and doing hand-off to @hide-on-bush-x near daily meeting

[juanmanso]

Since miners are described by their metadata, the first thing we need to inspect is the Metagraph to understand which params are key in the evaluation of Miner's eligibility (link to API docs).

List of potential relevant Metagraph elements

## Key components of Metagraph Object for Miners - `ranks` – Neuron rankings as per the Yuma Consensus algorithm, influencing their incentive distribution and network authority ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.R)). - `trust` – Scores indicating the reliability of neurons, mainly miners, within the network’s operational context ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.T)). - `incentive` – Rewards allocated to neurons, particularly miners, for their network contributions ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.I)). - `dividends` – Rewards received primarily by validators as part of the incentive mechanism ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.D)). - `active` – Status indicating whether a neuron is actively participating in the network. - `weights` – Inter-neuronal weights set by each neuron, influencing network dynamics ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.W)). ### Lower importance but could be useful - `n` (torch.nn.Parameter) – The total number of neurons in the network, reflecting its size and complexity. - `consensus` – Scores reflecting each neuron’s alignment with the network’s collective decisions ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.C)). - `stake` – Represents the cryptocurrency staked by neurons, impacting their influence and earnings within the network ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.S)). - `bonds` – Represents speculative investments by neurons in others, part of the reward mechanism ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.B)). - `uids` – Unique identifiers for each neuron, essential for network operations.

Potential INDIVIDUAL Selection Criteria

Below we will list the set of individual components that could be leveraged for a decision. It is clear that a weighted sum might be the final approach but to understand how important these are, we shall analyse them in a vacuum.

1. Based on miner's active status (listening to active)
2. Based on trust value since that represents the inter-peer weights and thus is a measure of past behavior and contribution
3. Based on weights that is proportional to mean performance against other miners a. ranks based decisions are discarded IMO since they are directly proportional to the weights and also do not have a reference in terms of performance between them. I'd say is a truncated version of looking into weights and thus less informative.
4. By looking at incentive and dividends of miners, we could infer the recent activity of it. So, if we are interested in adding diversity to the responses, considering performant miners with rising influence in the network, we might want to consider using these. a. Using stake could be useful since miners with higher staking numbers mean that they've been participating for longer tempos and they are not eager to cash-out rapidly. Although that could also flag inactivity and underperformance
5. [To Be Further Researched] Bonds are a measure of trust within individuals. so it could be an alternative way of adding diversity.

[juanmanso]

How do other subnets select miners?

• SN 1 / Prompting -> Random

• SN 8 / Trading Network -> Random: UUID Tracker + No Priorization (Usage)

• SN 9 / Pretraining -> Seems by UID: 1 eval per epoch + keeps track of pending uids to eval + tracked by filepath

• SN 15 / Blockchain Insights -> Top miners are defined as the ones available (filter (validators + without IP + not serving(aka not hooked to the network)) + ping to check availability) and sorted by score eligibility factoring both trust and incentive values.

• SN 16 / 3D Gen -> Tasks are appended and taken by Miners when available querying validators filtered by serving status, min stake and cooldown periods.

• SN 20 / BitAgent -> Notebook on docs selecting by Incentive but code might show it's by random selection or given by Synapse Object.

• SN 22 / Meta Search -> Random or All on get_uids.

• SN 27 / Compute -> Random but looks like it filters those axons that share IP. I infer that they dismiss users that have multiple neurons.

• SN 32 / It's AI -> Random

Conclusion from this analysis

• Almost every other relevant Subnet (Emissions >~ 1%) is doing random selection
• Those that keep it safe, consider if axons (miners) are available and/or active
• When selecting between top miners, incentive and trust data is taking into account to sort and/or select
• Some others swap selection so that Miners take tasks from validators which are queried based on availability, stake and cooldown periods

[juanmanso]

Moving it to in review since research looks to be over and its up to us to determine what should be our priority in Miner Selection and take action

[mudler]

@juanmanso can you please fill-in https://github.com/masa-finance/masa-bittensor/issues/92 with the implementation details so we can pick it up in the next planning session? thank you!

[juanmanso]

@juanmanso can you please fill-in #92 with the implementation details so we can pick it up in the next planning session? thank you!

@mudler I understood that this ticket needs to define the criteria. However, I just exposed different points of view/implementations but did not choose any. I'd say that depends on the goal of the subnet and the incentive mechanism to choose which selection criteria to take.

Personally, I would choose a random sampling of uids with some minimum filtering to guarantee that selected ones are operational miners but still keep it random. Although if we want to promote activity by checking incentives and/or trust, we could go down that path as well.

Since this is more of a business decision than a technical one from my POV, I left it for review of stakeholders or anyone in position to make the decision. If it's up to me the decision, then I'll complete #92 myself and close this ticket