Initial implementation of miner selection criteria

teslashibe commented 2 weeks ago

Description

Since miners are described by their metadata, the first thing we need to inspect is the Metagraph to understand which params are key in the evaluation of Miner's eligibility (link to API docs).

Individual elements to determine Miner's status

List of potential relevant Metagraph elements

### Key components of Metagraph Object for Miners - `ranks` – Neuron rankings as per the Yuma Consensus algorithm, influencing their incentive distribution and network authority ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.R)). - `trust` – Scores indicating the reliability of neurons, mainly miners, within the network’s operational context ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.T)). - `incentive` – Rewards allocated to neurons, particularly miners, for their network contributions ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.I)). - `dividends` – Rewards received primarily by validators as part of the incentive mechanism ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.D)). - `active` – Status indicating whether a neuron is actively participating in the network. - `weights` – Inter-neuronal weights set by each neuron, influencing network dynamics ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.W)). #### Lower importance but could be useful - `n` (torch.nn.Parameter) – The total number of neurons in the network, reflecting its size and complexity. - `consensus` – Scores reflecting each neuron’s alignment with the network’s collective decisions ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.C)). - `stake` – Represents the cryptocurrency staked by neurons, impacting their influence and earnings within the network ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.S)). - `bonds` – Represents speculative investments by neurons in others, part of the reward mechanism ([tensor getter](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.B)). - `uids` – Unique identifiers for each neuron, essential for network operations.

Below we will list the set of individual components that could be leveraged for a decision. It is clear that a weighted sum might be the final approach but to understand how important these are, we shall analyse them in a vacuum.

Based on miner's active status (listening to active)
Based on trust value since that represents the inter-peer weights and thus is a measure of past behavior and contribution
Based on weights that is proportional to mean performance against other miners a. ranks based decisions are discarded IMO since they are directly proportional to the weights and also do not have a reference in terms of performance between them. I'd say is a truncated version of looking into weights and thus less informative.
By looking at incentive and dividends of miners, we could infer the recent activity of it. So, if we are interested in adding diversity to the responses, considering performant miners with rising influence in the network, we might want to consider using these. a. Using stake could be useful since miners with higher staking numbers mean that they've been participating for longer tempos and they are not eager to cash-out rapidly. Although that could also flag inactivity and underperformance
[To Be Further Researched] Bonds are a measure of trust within individuals. so it could be an alternative way of adding diversity.

How do others do it?

Click to expand

- SN 1 / Prompting -> [Random](https://github.com/macrocosm-os/prompting/blob/082c52a4e4667e7044596480fdc59d6c8582f04b/prompting/forward.py#L185) - SN 8 / Trading Network -> Random: [UUID Tracker](https://github.com/taoshidev/proprietary-trading-network/blob/185dc688b129e496d478c1e84982e2cb869ee552/vali_objects/uuid_tracker.py) + [No Priorization](https://github.com/taoshidev/proprietary-trading-network/blob/185dc688b129e496d478c1e84982e2cb869ee552/neurons/validator.py#L272) ([Usage](https://github.com/taoshidev/proprietary-trading-network/blob/185dc688b129e496d478c1e84982e2cb869ee552/neurons/validator.py#L191)) - SN 9 / Pretraining -> Seems by UID: [1 eval per epoch](https://github.com/macrocosm-os/pretraining/blob/1285929220125e8c9eeaaa8b3a6892c1c8a7ab70/utilities/miner_iterator.py#L9) + [keeps track of pending uids to eval](https://github.com/macrocosm-os/pretraining/blob/1285929220125e8c9eeaaa8b3a6892c1c8a7ab70/neurons/validator.py#L509) + [tracked by filepath ](https://github.com/macrocosm-os/pretraining/blob/1285929220125e8c9eeaaa8b3a6892c1c8a7ab70/neurons/validator.py#L114) - SN 15 / Blockchain Insights -> [Top miners](https://github.com/blockchain-insights/blockchain-data-subnet/blob/c6b8d4e7e9607d0f5ed2bac24857ab1f11cf56fb/neurons/validators/utils/uids.py#L41) are defined as the ones available (filter ([validators](https://github.com/blockchain-insights/blockchain-data-subnet/blob/c6b8d4e7e9607d0f5ed2bac24857ab1f11cf56fb/neurons/validators/utils/uids.py#L31) + [without IP](https://github.com/blockchain-insights/blockchain-data-subnet/blob/c6b8d4e7e9607d0f5ed2bac24857ab1f11cf56fb/neurons/validators/utils/uids.py#L35) + [not serving](https://github.com/blockchain-insights/blockchain-data-subnet/blob/c6b8d4e7e9607d0f5ed2bac24857ab1f11cf56fb/neurons/validators/utils/uids.py#L24)(aka [not hooked to the network](https://docs.bittensor.com/python-api/html/autoapi/bittensor/axon/index.html#bittensor.axon.axon.serve))) + [ping](https://github.com/blockchain-insights/blockchain-data-subnet/blob/c6b8d4e7e9607d0f5ed2bac24857ab1f11cf56fb/neurons/validators/utils/uids.py#L68C39-L68C48) to check availability) and sorted by [score eligibility](https://github.com/blockchain-insights/blockchain-data-subnet/blob/c6b8d4e7e9607d0f5ed2bac24857ab1f11cf56fb/neurons/validators/utils/uids.py#L78) factoring both [trust](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.T) and [incentive](https://docs.bittensor.com/python-api/html/autoapi/bittensor/metagraph/index.html#bittensor.metagraph.metagraph.I) values. - SN 16 / 3D Gen -> [Tasks are appended](https://github.com/404-Repo/three-gen-subnet/blob/89d15392a6fc7c84e156b342035763726f6d78ef/neurons/validator/__init__.py#L165) and taken by Miners when available [querying validators](https://github.com/404-Repo/three-gen-subnet/blob/89d15392a6fc7c84e156b342035763726f6d78ef/neurons/miner/validator_selector.py#L29) filtered by [serving status](https://github.com/404-Repo/three-gen-subnet/blob/89d15392a6fc7c84e156b342035763726f6d78ef/neurons/miner/validator_selector.py#L40), [min stake](https://github.com/404-Repo/three-gen-subnet/blob/89d15392a6fc7c84e156b342035763726f6d78ef/neurons/miner/validator_selector.py#L41) and [cooldown periods](https://github.com/404-Repo/three-gen-subnet/blob/89d15392a6fc7c84e156b342035763726f6d78ef/neurons/miner/validator_selector.py#L42). - SN 20 / BitAgent -> [Notebook on docs](https://github.com/RogueTensor/bitagent_subnet/blob/c676ab4b33a6d087d65f4c7cf347cb49fc9c4b80/docs/tasks.ipynb#L92) selecting by Incentive but code might show it's by [random selection](https://github.com/RogueTensor/bitagent_subnet/blob/c676ab4b33a6d087d65f4c7cf347cb49fc9c4b80/bitagent/validator/forward.py#L51) or given by [Synapse Object](https://github.com/RogueTensor/bitagent_subnet/blob/c676ab4b33a6d087d65f4c7cf347cb49fc9c4b80/bitagent/validator/forward.py#L69). - SN 22 / Meta Search -> [Random or All ](https://github.com/Datura-ai/smart-scrape/blob/4a8007f5d77d60739778735f9c800966535af2df/datura/__init__.py#L91)on [get_uids](https://github.com/Datura-ai/smart-scrape/blob/4a8007f5d77d60739778735f9c800966535af2df/neurons/validators/validator.py#L165). - SN 27 / Compute -> Random but looks like it [filters those axons that share IP.](https://github.com/neuralinternet/compute-subnet/blob/2a2f51ec9f93ac77ada4346cf3a44b08a2dc295c/neurons/validator.py#L374) I infer that they dismiss users that have multiple neurons. - SN 32 / It's AI -> [Random](https://github.com/It-s-AI/llm-detection/blob/be4417f83e2e2762a28b50b61178962bac8e50af/detection/validator/forward.py#L87)

Summary of references

Almost every other relevant Subnet (Emissions >~ 1%) is doing random selection
Those that keep it safe, consider if axons (miners) are available and/or active
When selecting between top miners, incentive and trust data is taking into account to sort and/or select
Some others swap selection so that Miners take tasks from validators which are queried based on availability, stake and cooldown periods

Additional Notes

There are different points of view/implementations but did not choose any. Depending on the goal of the subnet and the incentive mechanism we might take different selection criteria.

Personally, I would choose a random sampling of uids with some minimum filtering/sanitizing to guarantee that selected ones are operational miners but still keep it random. Although if we want to promote activity by checking incentives and/or trust, we could go down that path as well. That however might skew the network and instead of optimizing their miners for the task, users might optimize to get selections rather than perform better tasks (which depending on incentive mechanism they could differ).

Since this is more of a business decision than a technical one I'd say for the implementation to properly start, we may need that decision first.

Acceptance criteria

[ ] We have an implementation in the codebase of the worker selection, based on random sampling
[ ] sampling should filter nodes which are available and operational
[ ] The worker selection heuristic should be random for now
[ ] ~Explore potential alternatives that takes into account incentives and trust~ Exploration was made on spike, this is implementation task only

mudler commented 2 weeks ago

This card needs to be triaged during planning with the results of https://github.com/masa-finance/masa-bittensor/issues/63

juanmanso commented 1 week ago

Moving it to backlog to triage

juanmanso commented 1 week ago

Awaiting Stakeholders and Roadmap Owners for definition. Based on that, this task can be:

Defined to have a bound implementation
Be put on backlog for development

Luka-Loncar commented 1 day ago

@hide-on-bush-x - I've seen that @juanmanso wrote that he resolved some issues here but we aren't sure if these fixes are the ones that you have encountered.

Can we get the confirmation that the fix is ok, or fix it? Since this is part of our release v0.5.

hide-on-bush-x commented 1 day ago

Hey @Luka-Loncar, yeah it's related, we already synced with @juanmanso and we are looking at it together, we'll have this merged by daily time today

juanmanso commented 22 hours ago

This is somewhat blocked by errors related out of bounds indexing.

Related by #127

However, after some testing with @obasilakis the issue persists (link to slack thread).

Needs further debugging

masa-finance / masa-bittensor