This PR updates the model names in neuron_explainer to use the recently updated version of GPT-4, gpt-4-turbo (which currently points to gpt-4-turbo-2024-04-09)
The reasons for this change are:
Better explanations/scoring: I noticed slightly improved MLP explanation scoring when using the new gpt-4-turbo-2024-04-09 model vs. the currently used gpt-4 (currently points to gpt-4-0613) and gpt-4-0125-preview models. Specifically, I observed higher confidence scores for accurate explanations and fewer instances of high scores for poor explanations.
Here is a quick model comparison test with gpt-2-small using the mlp_neuron dataset at 1:1
More cost-effective: The gpt-4-turbo-2024-04-09 model is cheaper than the currently used gpt-4-0613 model (1/3 the input cost and 1/2 the output cost according to https://openai.com/pricing).
If this proposed model change would disrupt any TDB use cases, the following alternatives may be worth considering instead:
Allow users to set or override the default model via environment variable, e.g. EXPLAINER_MODEL or similar.
Display the active explainer/scoring model name in the TDB UI and/or add "model name" attribution for scored explanations in the local cached explanation directory.
Use the version-specific model names (gpt-4-turbo-2024-04-09 instead of gpt-4-turbo) to prevent potential inconsistencies due to continuous model upgrades.
This PR updates the model names in
neuron_explainerto use the recently updated version of GPT-4,gpt-4-turbo(which currently points togpt-4-turbo-2024-04-09)The reasons for this change are:
gpt-4-turbo-2024-04-09model vs. the currently usedgpt-4(currently points togpt-4-0613) andgpt-4-0125-previewmodels. Specifically, I observed higher confidence scores for accurate explanations and fewer instances of high scores for poor explanations.gpt-4-turbo-2024-04-09model is cheaper than the currently usedgpt-4-0613model (1/3 the input cost and 1/2 the output cost according to https://openai.com/pricing).If this proposed model change would disrupt any TDB use cases, the following alternatives may be worth considering instead:
Allow users to set or override the default model via environment variable, e.g.
EXPLAINER_MODELor similar.Display the active explainer/scoring model name in the TDB UI and/or add "model name" attribution for scored explanations in the local cached explanation directory.
Use the version-specific model names (
gpt-4-turbo-2024-04-09instead ofgpt-4-turbo) to prevent potential inconsistencies due to continuous model upgrades.Open to all feedback and suggestions, thank you.
Model info: https://platform.openai.com/docs/models/continuous-model-upgrades
gpt-4-turbocurrently points togpt-4-turbo-2024-04-09gpt-4currently points togpt-4-0613