Open PhantomGrapes opened 1 month ago
To resolve the ValueError
in the prompt_helper
module caused by the default settings of DASHSCOPE_MODEL_META
, you need to ensure that the context_window
is greater than the sum of num_prompt_tokens
and num_output
. The current default settings have context_window
equal to num_output
, which leaves no space for num_prompt_tokens
.
Here's how you can adjust the DASHSCOPE_MODEL_META
settings to avoid this issue:
DASHSCOPE_MODEL_META
dictionary: Ensure that context_window
is sufficiently larger than num_output
to accommodate the prompt tokens.DASHSCOPE_MODEL_META = {
DashScopeGenerationModels.QWEN_TURBO: {
"context_window": 1024 * 8,
"num_output": 1024 * 6, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
DashScopeGenerationModels.QWEN_PLUS: {
"context_window": 1024 * 32,
"num_output": 1024 * 30, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
DashScopeGenerationModels.QWEN_MAX: {
"context_window": 1024 * 8,
"num_output": 1024 * 6, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
DashScopeGenerationModels.QWEN_MAX_1201: {
"context_window": 1024 * 8,
"num_output": 1024 * 6, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
DashScopeGenerationModels.QWEN_MAX_LONGCONTEXT: {
"context_window": 1024 * 30,
"num_output": 1024 * 28, # Adjusted to leave space for prompt tokens
"is_chat_model": True,
},
}
DashScope
class to validate model_name
: Ensure that the model_name
used is valid and exists in the DASHSCOPE_MODEL_META
dictionary.class DashScope(CustomLLM):
# ... other code ...
@property
def metadata(self) -> LLMMetadata:
if self.model_name not in DASHSCOPE_MODEL_META:
raise ValueError(f"Model name '{self.model_name}' is not valid. Please choose from {list(DASHSCOPE_MODEL_META.keys())}.")
DASHSCOPE_MODEL_META[self.model_name]["num_output"] = (
self.max_tokens or DASHSCOPE_MODEL_META[self.model_name]["num_output"]
)
return LLMMetadata(
model_name=self.model_name, **DASHSCOPE_MODEL_META[self.model_name]
)
# ... other code ...
DASHSCOPE_WORKSPACE_ID
, DASHSCOPE_API_KEY
, and DASHSCOPE_BASE_URL
are correctly set, as these are required for the DashScopeCloudIndex
class to function properly.class DashScopeCloudIndex(BaseManagedIndex):
"""DashScope Cloud Platform Index."""
def __init__(
self,
name: str,
nodes: Optional[List[BaseNode]] = None,
transformations: Optional[List[TransformComponent]] = None,
timeout: int = 60,
workspace_id: Optional[str] = None,
api_key: Optional[str] = None,
base_url: Optional[str] = DASHSCOPE_DEFAULT_BASE_URL,
show_progress: bool = False,
callback_manager: Optional[CallbackManager] = None,
**kwargs: Any,
) -> None:
"""Initialize the Platform Index."""
self.name = name
self.transformations = transformations or []
if nodes is not None:
raise ValueError(
"DashScopeCloudIndex does not support nodes on initialization"
)
self.workspace_id = workspace_id or os.environ.get("DASHSCOPE_WORKSPACE_ID")
self._api_key = api_key or os.environ.get("DASHSCOPE_API_KEY")
self._base_url = os.environ.get("DASHSCOPE_BASE_URL", None) or base_url
self._headers = {
"Content-Type": "application/json",
"Accept-Encoding": "utf-8",
"X-DashScope-WorkSpace": self.workspace_id,
"Authorization": "Bearer " + self._api_key,
"X-DashScope-OpenAPISource": "CloudSDK",
}
self._timeout = timeout
self._show_progress = show_progress
self._service_context = None
self._callback_manager = callback_manager or Settings.callback_manager
By making these adjustments, you should be able to avoid the ValueError
in the prompt_helper
module [1][2].
To continue talking to Dosu, mention @dosu.
Bug Description
The dashscope llm module has the following default settings:
where context_window is equal to num_output.
This default setting can always cause ValueError in prompt_helper
since self.context_window = self.num_output, no space left for num_prompt_tokens.
Version
0.10.55
Steps to Reproduce
Relevant Logs/Tracbacks