Models should be reevaluated after 24~48 hours later.

[torquedrop]

If a model is trained on a very small, specific dataset, it can lead to overfitting. The current validation process relies heavily on data generated in the last 48 hours. To address this, we could train the model using data from the previous 5 to 10 days, which tends to result in higher emissions.

This approach isn't fair to developers of general-purpose models, and it could result in ineffective models. To prevent this, newly submitted models should be regularly reevaluated using newer datasets.

There are two key benefits to this:

• If a developer doesn't update their model regularly, the model will have lower emissions after reevaluation.
• If a developer updates their model frequently, they won't receive emissions for extended periods with a score of zero.

I suggest implementing 2 or 3 validation processes during regular operations if the subnet is busy. However, if the subnet is not busy, all available validation processes should be utilized.

[donaldknoller]

Thanks for your suggestion. Please note that a variation of this has already been mentioned here which covers the more pertinent issue at hand.