Closed bpiwowar closed 1 year ago
Re-opening: the current solution is not optimal, so it is better to switch to solution 2 (Add a mechanism to have several tasks to be run)
Fixed in bcaa1eff7683236d502e0a79d5919a51c2eba702 with experimaestro 0.28 that includes initialization tasks.
When a model is trained, one might want to access one of its submodules in another task, e.g.
In that case, the encoder is random... and this is quite unexpected.
Potential solutions:
Add a mechanism in experimaestro that allows linking the sub-models to the main one - when initialized, the
encoder
will require themodel
to be loaded from the checkpoint. Caveats: as with any implicit process, this makesxpmir
more complex and might have other side effects.Add a mechanism to have several tasks to be run (and in that case, the output will be just the path), instead of just one, i.e.
Important caveat: it breaks the current HuggingFace uploading/downloading mechanism and we need to include
load_model
each time we want to use the pre-trained model (which can begin to be cumbersome when various models are combined since we need to track down this). For HF, the solution would be to have this when loadingThird solution, do something a bit in between
where
model_loader
is a specialexperimaestro
object (ConfigFactory
) that does whatever is needed and returns the "encoder" submodel.