Downloading a model and saving it locally using AutoModel.from_pretrained loses the model specialization (detailed description in #213) . After loading these broken models, using them for prediction returns bad results.
To solve the testing issues, we need to make the prediction udfs deterministic
For this, we need to be able to download a fixed version of a model and set the seed inside the prediction udf.
This means we need a version parameter in download udf, prediction udf and model upload cli
The version should be part of the name of the uploaded model file, such that users can upload multiple versions
We need a seed parameter in the prediction udfs
Prerequisites: #216, #217, #218, #219, #220
This ticket adds the seed parameter and new tests:
[ ] add an optional "seed" parameter in the prediction udfs
[ ] add the "task" , and "version" parameters to the prediction udfs
[ ] change the tests for the prediction udfs accordingly.
[ ] use the new "version" and optional "seed" parameter for a new test which downloads a model , and invokes a prediction udf with pinned model version and seed, checking if the output stays consistent.
Downloading a model and saving it locally using AutoModel.from_pretrained loses the model specialization (detailed description in #213) . After loading these broken models, using them for prediction returns bad results.
To solve the testing issues, we need to make the prediction udfs deterministic
Prerequisites: #216, #217, #218, #219, #220
This ticket adds the seed parameter and new tests:
[ ] add an optional "seed" parameter in the prediction udfs
[ ] add the "task" , and "version" parameters to the prediction udfs
[ ] change the tests for the prediction udfs accordingly.
[ ] use the new "version" and optional "seed" parameter for a new test which downloads a model , and invokes a prediction udf with pinned model version and seed, checking if the output stays consistent.