dosubot[bot]
commented
4 months ago To resolve the ValueError in the prompt_helper module caused by the default settings of DASHSCOPE_MODEL_META, you need to ensure that the context_window is greater than the sum of num_prompt_tokens and num_output. The current default settings have context_window equal to num_output, which leaves no space for num_prompt_tokens.
Here's how you can adjust the DASHSCOPE_MODEL_META settings to avoid this issue:
- Update the
DASHSCOPE_MODEL_METAdictionary: Ensure thatcontext_windowis sufficiently larger thannum_outputto accommodate the prompt tokens.
DASHSCOPE_MODEL_META = {
DashScopeGenerationModels.QWEN_TURBO: {
"context_window": 1024 * 8,
"num_output": 1024 * 6, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
DashScopeGenerationModels.QWEN_PLUS: {
"context_window": 1024 * 32,
"num_output": 1024 * 30, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
DashScopeGenerationModels.QWEN_MAX: {
"context_window": 1024 * 8,
"num_output": 1024 * 6, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
DashScopeGenerationModels.QWEN_MAX_1201: {
"context_window": 1024 * 8,
"num_output": 1024 * 6, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
DashScopeGenerationModels.QWEN_MAX_LONGCONTEXT: {
"context_window": 1024 * 30,
"num_output": 1024 * 28, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
}- Modify the
DashScopeclass to validatemodel_name: Ensure that themodel_nameused is valid and exists in theDASHSCOPE_MODEL_METAdictionary.
class DashScope(CustomLLM):
# ... other code ...
@property
def metadata(self) -> LLMMetadata:
if self.model_name not in DASHSCOPE_MODEL_META:
raise ValueError(f"Model name '{self.model_name}' is not valid. Please choose from {list(DASHSCOPE_MODEL_META.keys())}.")
DASHSCOPE_MODEL_META[self.model_name]["num_output"] = (
self.max_tokens or DASHSCOPE_MODEL_META[self.model_name]["num_output"]
)
return LLMMetadata(
model_name=self.model_name, **DASHSCOPE_MODEL_META[self.model_name]
)
# ... other code ...- Ensure environment variables are set correctly: Make sure the environment variables
DASHSCOPE_WORKSPACE_ID,DASHSCOPE_API_KEY, andDASHSCOPE_BASE_URLare correctly set, as these are required for theDashScopeCloudIndexclass to function properly.
class DashScopeCloudIndex(BaseManagedIndex):
"""DashScope Cloud Platform Index."""
def __init__(
self,
name: str,
nodes: Optional[List[BaseNode]] = None,
transformations: Optional[List[TransformComponent]] = None,
timeout: int = 60,
workspace_id: Optional[str] = None,
api_key: Optional[str] = None,
base_url: Optional[str] = DASHSCOPE_DEFAULT_BASE_URL,
show_progress: bool = False,
callback_manager: Optional[CallbackManager] = None,
**kwargs: Any,
) -> None:
"""Initialize the Platform Index."""
self.name = name
self.transformations = transformations or []
if nodes is not None:
raise ValueError(
"DashScopeCloudIndex does not support nodes on initialization"
)
self.workspace_id = workspace_id or os.environ.get("DASHSCOPE_WORKSPACE_ID")
self._api_key = api_key or os.environ.get("DASHSCOPE_API_KEY")
self._base_url = os.environ.get("DASHSCOPE_BASE_URL", None) or base_url
self._headers = {
"Content-Type": "application/json",
"Accept-Encoding": "utf-8",
"X-DashScope-WorkSpace": self.workspace_id,
"Authorization": "Bearer " + self._api_key,
"X-DashScope-OpenAPISource": "CloudSDK",
}
self._timeout = timeout
self._show_progress = show_progress
self._service_context = None
self._callback_manager = callback_manager or Settings.callback_managerBy making these adjustments, you should be able to avoid the ValueError in the prompt_helper module [1][2].
To continue talking to Dosu, mention @dosu.
Bug Description
The dashscope llm module has the following default settings:
where context_window is equal to num_output.
This default setting can always cause ValueError in prompt_helper
since self.context_window = self.num_output, no space left for num_prompt_tokens.
Version
0.10.55
Steps to Reproduce
Relevant Logs/Tracbacks