Open cirquit opened 1 year ago
So I found the issue - adding additional validators to the DHT is necessary to parse LocalTrainingProgress.
from hivemind.dht.schema import (
BytesWithPublicKey,
RSASignatureValidator,
SchemaValidator)
from pydantic import BaseModel, StrictBool, StrictFloat, confloat, conint
class LocalTrainingProgress(BaseModel):
peer_id: bytes
epoch: conint(ge=0, strict=True)
samples_accumulated: conint(ge=0, strict=True)
samples_per_second: confloat(ge=0.0, strict=True)
time: StrictFloat
client_mode: StrictBool
class TrainingProgressSchema(BaseModel):
progress: Dict[BytesWithPublicKey, Optional[LocalTrainingProgress]]
run_id = (...get run_id)
dht = (...init dht)
signature_validator = RSASignatureValidator(None)
local_public_key = signature_validator.local_public_key
dht.add_validators(
[
SchemaValidator(TrainingProgressSchema, prefix=f"{run_id}"),
signature_validator,
]
)
metadata, expiration = dht.get(key=f"{run_id}_progress", return_future=False)
I'm planning to create a pull request to update the documentation with a full example to access the GlobalTrainingProgress. You're welcome to either keep the issue open for me to reference it or close it.
Hi,
I can't seem to be able to read the training information (like here) out of the DHT that was created by hivemind.
I can connect to the DHT and run the following:
However, when training with hivemind, I can't seem to be able to get the data with two different behaviors after calling the
get
function after each other.Only the second call shows some actual training progress data, but not complete (1 out of 4 peers) and not in a way that allows me to access it compared to the documentation.
It seems that there is some issue with the
get
call being run asynchronously and not being able to decode the returning LocalTrainingProgress.How does the tutorial data
get/store
differ from what hivemind does with the LocalTrainingProgress?First call to
get
Second call to
get