yeagerai / genlayer-simulator

MIT License
12 stars 5 forks source link

SIM-BE-Remove randomization of validators configuration #439

Open cristiam86 opened 1 week ago

cristiam86 commented 1 week ago

When setting up heurist, it randomizes the configuration, but I don’t think max_tokens is a good parameter to randomise

denishacquin commented 1 week ago

Should we then decide of a flat value @cristiam86 ?

Also, is this on creation of a new validator, or when installing the simulator?

cristiam86 commented 1 week ago

This is in the backend when we create a new validator. We are randomizing the parameters (see: backend/node/create_nodes and backend/node/defaults.json) Making all these values random is wrong. We should aim for the optimal values where we can. More research is needed to understand which values can be random and which ones can be optimized

denishacquin commented 1 week ago

If I'm correct, this is only when using the init command genlayer init --numValidators 10

There's no randomization happening when creating a validator manually from the UI.

Do we still want to remove the randomization of config if that is the case?

cristiam86 commented 1 week ago

For this issue we are always talking about the randomization of config, not the providers and models.

denishacquin commented 1 week ago

Sorry if I wasn't clear. What I mean is, it seems we only randomize the config when calling explicitly those endpoints:

And from what I understand, these are only used in tests and in debug_simulator.py (which will be removed).

The create_validator endpoint (the one used when we create a validator via the frontend) does not randomize the config.

If I'm correct about the above, do we still need to change something?

Happy to discuss it if necessary.

AgustinRamiroDiaz commented 1 week ago

@denishacquin I think it might be a good idea to wait for https://github.com/yeagerai/genlayer-simulator/issues/359, since all of this configuration will change after that refactor

cristiam86 commented 5 days ago

@denishacquin @AgustinRamiroDiaz it seems to me that what Agustín did here: https://github.com/yeagerai/genlayer-simulator/pull/475/files#diff-c0c0177862bf06ec410fcbcd543e48045c6e9fc17c2120ff2e93e402b6037fb2R1 fixes this issue

denishacquin commented 5 days ago

Indeed, let's close this when we merge #475