Open JamesScharf opened 7 months ago
I am also getting the same exact error when trying to use "Getting Started" example notebook. Here is my code
bedrock = dspy.Bedrock(model='anthropic.claude-v2', region_name="us-west-2") colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts') dspy.settings.configure(lm=bedrock, rm=colbertv2_wiki17_abstracts)
Evaluation
`from dspy.evaluate.evaluate import Evaluate
evaluate_on_hotpotqa = Evaluate(devset=devset, num_threads=1, display_progress=True, display_table=5)
metric = dspy.evaluate.answer_exact_match evaluate_on_hotpotqa(compiled_rag, metric=metric)`
I see. Does Bedrock support something else in place of n
and max_tokens
? Both are necessary in DSPy.
i.e., num_completions
and maximum_output_tokens
If the former isn't supported, we can do it with a loop... but that's more expensive.
Anyway can fix now by just ignoring n and max_tokens in the request
function for Bedrock @JamesScharf if they can't be replaced.
Otherwise, can fix by replacing these values on the fly with the correct kwarg names
Re: n
, I don't believe that Bedrock has any equivalent. So, we'd have to loop I think.
The equivalent to max_new_tokens
is max_tokens_to_sample
; there is no equivalent for max_tokens
@okhat It depends with which model one is using with Bedrock. If using Claude then as @JamesScharf correctly points out ismax_tokens_to_sample
if using Titan Models it is maxTokenCount
. I am kind of curious if using something like LiteLLM or langChain would abstract these model/provider specific nuances.
I'd be hesitant to use LiteLLM for this abstraction. I've used its Sagemaker wrapper and it did not seem to support temperature.
I'll make a PR with the following updates to Bedrock--let me know if I'm missing anything:
n
as a loop since n
is not currently supported by Bedrock (to my knowledge)max_tokens
to max_tokens_to_sample
Bedrock's parameters are provider specific and Langchain makes no attempt to unify them (which will anyways be a bad idea). So I am not sure how much introduction of Langchain help.
Hello @JamesScharf,
The issue persists for me. As a workaround, I implemented a removal of the n key from kwargs in the following manner: if "n" in kwargs: if self._batch_n: del kwargs["n"] llm_out = self._simple_api_call(formatted_prompt=formatted_prompt, **kwargs) is that correct?
Good catch, would you mind making a PR? I am away from my computer for a few days and can't do so myself.
I'm encountering a permission issues for the PR
@JamesScharf I have a new error with bedrock botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: max_tokens_to_sample: range: 1..1,000,000
@tech4life87 I got the same error using Litellm: openai.BadRequestError: Error code: 400 - {'error': {'message': 'BedrockException - BedrockException - An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: extraneous key [drop_params] is not permitted, please reformat your input and try again.', 'type': None, 'param': None, 'code': 400}}
I have a PR #772 that tackles this issue. Almost done...
AWS's Bedrock runtime does not accept the following parameters, however they are still being passed to the LM at some point:
n
andmax_tokens
At what point is this being done? I haven't specified either value in my code which uses DSPy.
Here's the full error from boto3: