ADS is the Oracle Data Science Cloud Service's python SDK supporting, model ops (train/eval/deploy), along with running workloads on Jobs and Pipeline resources.
In the initial implementation of the aqua-evaluate project, default model parameters were hard coded within the project. Given the diversity of LLM frameworks, which may support varying parameters under the Open AI spec, this setup could lead to compatibility issues. To enhance flexibility and adaptability, it is proposed to implement a pass-through mechanism. This mechanism would allow parameters from the client side to be directly forwarded to the LLM inferencing framework, enabling the framework to handle the parameters and generate errors.
As a temporary solution, this PR addressing a list of mandatory service parameters that will be added on top of user-provided parameters during evaluation.
Description
In the initial implementation of the aqua-evaluate project, default model parameters were hard coded within the project. Given the diversity of LLM frameworks, which may support varying parameters under the Open AI spec, this setup could lead to compatibility issues. To enhance flexibility and adaptability, it is proposed to implement a pass-through mechanism. This mechanism would allow parameters from the client side to be directly forwarded to the LLM inferencing framework, enabling the framework to handle the parameters and generate errors.
As a temporary solution, this PR addressing a list of mandatory service parameters that will be added on top of user-provided parameters during evaluation.
In the next iteration, these parameters will be moved to the service containers' config as outlined in the dedicated ticket - https://jira.oci.oraclecorp.com/browse/ODSC-60520.