This pull request brings several enhancements and bug fixes to the langtest module, focusing on model configuration and handling. The key updates include:
Model Configuration and Handling Improvements:
Configuration Types Added:
Introduced DatasetConfig and ModelConfig types to define configurations for datasets and models in langtest/types.py.
Class Initializer Changes:
Updated the Harness class initializer to accept ModelConfig and DatasetConfig types for the model and data parameters.
Modified the __init__ method in Harness to handle the model_type attribute.
Model Loading Enhancements:
Enhanced the load_model method in llm_modelhandler.py to support the model_type parameter, including chat models.
Updated transformers_modelhandler.py to manage model_type in the load_model and predict methods, with added support for chat models.
Utility and Miscellaneous Updates:
Utility Improvements:
Added a CHAT_MODEL_CLASSES mapping in utils.py for associating hub names with chat model classes.
Applied type annotations in config_utils.py for DEFAULTS_CONFIG.
Prompt Handling Updates:
Modified prompts.py to support prompt retrieval for the "transformers" hub.
These changes enhance the flexibility and robustness of the langtest module, making it easier to work with various model configurations and types.
This pull request brings several enhancements and bug fixes to the
langtest
module, focusing on model configuration and handling. The key updates include:Model Configuration and Handling Improvements:
Configuration Types Added:
DatasetConfig
andModelConfig
types to define configurations for datasets and models inlangtest/types.py
.Class Initializer Changes:
Harness
class initializer to acceptModelConfig
andDatasetConfig
types for themodel
anddata
parameters.__init__
method inHarness
to handle themodel_type
attribute.Model Loading Enhancements:
load_model
method inllm_modelhandler.py
to support themodel_type
parameter, including chat models.transformers_modelhandler.py
to managemodel_type
in theload_model
andpredict
methods, with added support for chat models.Utility and Miscellaneous Updates:
Utility Improvements:
CHAT_MODEL_CLASSES
mapping inutils.py
for associating hub names with chat model classes.config_utils.py
forDEFAULTS_CONFIG
.Prompt Handling Updates:
prompts.py
to support prompt retrieval for the "transformers" hub.These changes enhance the flexibility and robustness of the
langtest
module, making it easier to work with various model configurations and types.