search

impel-intelligence / dippy-bittensor-subnet

MIT License
46 stars 18 forks source link

Miners have found a way to register the same model hash on a different key under a new block. #70

Closed Nadejde closed 2 weeks ago

Nadejde commented 2 weeks ago

Some examples:

These showed up much much later this morning and just picked up the same model has as the ones I uploaded las night. and are scored on the board.

121 5EecJA3qFgPCeP4aDWCBt82XrLc2CGJwfC4DaqDTdt3jfp7Y 0.4121 ✅COMPLETED chatml 5408010054771362223 20241005-ciux-c-12 2024-10-05T21:26:29.530601+00:00 4fab1b23fcabf30e762a754cc792f19ea95eb813d4f6b3711db58149deef6b98

and

137 5H3PtrPVw3eM8bXRiLfrxPmWmhkkd1cuzNtgkphMiQvAuTMP 0.4134 ✅COMPLETED chatml 6328808364566147717 20241005-ciux-c-13 2024-10-05T21:25:35.982864+00:00 0c8b120e962cb9aea7c37847070afae873a89cf32541b67135ad72ecdc232b22

image

and they did not get evaluated this morning they just got the same score that my model already got last night

model hash: 4fab1b23fcabf30e762a754cc792f19ea95eb813d4f6b3711db58149deef6b98 original block: 3,979,253 then it shows up again on block: 3,981,607

Happened again on 2024-10-08:

169 5DiHDkhxU5EzVDQzHrUT5D7aQNtFW3JGjwxKc1h5MvGGAdd5 0.4076 ✅COMPLETED llama3 12212680012525209568 20-03102305-02Haymak 2024-10-03T20:30:48.367599+00:00 5c7f7c6477eb51e0fde35e0859f63a5340cb75a48ac87fb311542d9a2b41136b 0.2484 0.9218 OhaymakingO 0.9766 0.8171 0.8373 0.3662 3,997,282 2024-10-08T09:35:46.72736+00:00 209 5E4dvUNJhKDW9JqsbY1F63xTe18iSr6RYUjZVuFUPdLXDBu5 0.4091 ✅COMPLETED llama3 17307090214359259512 18-03102248-SloTomatos04 2024-10-03T20:03:16.669589+00:00 e45ba96b94d772e9456bb2f7a070618aceb726c8a6d2e78d8f072cc3af95932d 0.2484 0.9225 slotomatos 0.9766 0.816 0.8373 0.3682 3,997,280 2024-10-08T09:37:07.638935+00:00

model hash: 5c7f7c6477eb51e0fde35e0859f63a5340cb75a48ac87fb311542d9a2b41136b original block: 3,964,487 shows up again without evaluation at block: 3,997,282

on different hotkey different coldkey and now you have the same model twice on the board.

donaldknoller commented 2 weeks ago

Note: the issue described here can be resolved in part by addressing https://github.com/impel-intelligence/dippy-bittensor-subnet/issues/67 .

To provide a detailed breakdown:

The "same" repository is technically not the "same" according to the code as the hash value is different (the components of the hash value include other data among other fields such as competition_id and git_commit as seen here: https://github.com/impel-intelligence/dippy-bittensor-subnet/blob/main/model/data.py#L15).

One implementation that can address the above concern is to add the following:

  1. Explicitly associate the commit data to a hotkey for unique identification
  2. Add an additional check for specifically the repository namespace + repository name
Vo1ks commented 2 weeks ago

169 5DiHDkhxU5EzVDQzHrUT5D7aQNtFW3JGjwxKc1h5MvGGAdd5 0.4076 ✅COMPLETED llama3 12212680012525209568 20-03102305-02Haymak 2024-10-03T20:30:48.367599+00:00 5c7f7c6477eb51e0fde35e0859f63a5340cb75a48ac87fb311542d9a2b41136b 0.2484 0.9218 OhaymakingO 0.9766 0.8171 0.8373 0.3662 3,997,282 2024-10-08T09:35:46.72736+00:00 209 5E4dvUNJhKDW9JqsbY1F63xTe18iSr6RYUjZVuFUPdLXDBu5 0.4091 ✅COMPLETED llama3 17307090214359259512 18-03102248-SloTomatos04 2024-10-03T20:03:16.669589+00:00 e45ba96b94d772e9456bb2f7a070618aceb726c8a6d2e78d8f072cc3af95932d 0.2484 0.9225 slotomatos 0.9766 0.816 0.8373 0.3682 3,997,280 2024-10-08T09:37:07.638935+00:00

What does this have to do with my model, which was uploaded on 2024-10-03, has my hotkeys, its own block and was rated in the queue after 18 hours of waiting? I think you've got something mixed up!

P.S. Sorry, I understood what you wrote. Indeed, some other hotkey was loaded for my model. My 171 and 240, but 169 and 209 are not mine, but at the same time this is my model! O_o screen1 screen2

donaldknoller commented 2 weeks ago

Note: We have added a partial hotfix for this:

Although the minerboard may reflect a different state from that of the validators, the important aspect of emissions has been addressed in the meantime while a more comprehensive fix is in progress.

There will be an additional check to be added as part of a larger update to the commit / ModelId string format

donaldknoller commented 2 weeks ago

This behavior has been patched as of https://github.com/impel-intelligence/dippy-bittensor-subnet/pull/72