Two MIME type issues in the Amazon BedrockRuntime client

alex23lemm commented 4 months ago

Hi paws team, thanks for all of your work and effort to constantly improve the paws package while adding clients for newly available AWS services.

I assume I found two MIME-type related bugs in the recently released function bedrockruntime_invoke_model() of the Amazon BedrockRuntime client.

In short, I am trying to convert the following Python/Boto3 example, which is part of the Anthropic Bedrock API documentation here, to R/paws:

import boto3
import json

bedrock = boto3.client(service_name="bedrock-runtime")
body = json.dumps(
    {
        "prompt": "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:",
        "max_tokens_to_sample": 100,
        "anthropic_version": "bedrock-2023-05-31"
    }
)

response = bedrock.invoke_model(body=body, modelId="anthropic.claude-v2:1")

response_body = json.loads(response.get("body").read())
print(response_body.get("completion"))

The Python code works as expected on my local machine and returns:

Sure, here's a silly space joke:

Why can't astronauts tell good jokes timing jokes?
Because of the long pauses between the punch lines!

The R equivalent of the first part above looks as follows:

library(paws)
library(jsonlite)

body <- list(prompt = "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:", 
             max_tokens_to_sample = 100,
             anthropic_version = "bedrock-2023-05-31") |>
  toJSON(auto_unbox = TRUE)

response <- bedrock$invoke_model(body = body, modelId = "anthropic.claude-v2:1")

Issue 1: The invoke_model() call - in which I did not set the MIME types explicitly - results in the error below. This seems to be a bug as the default value should be application/json based on the documentation.

Error: ValidationException (HTTP 400). The provided Accept Type is invalid or not supported for this model

Setting the MIME type for the contentType and accept parameters to application/json explicitly solves this issue and the client receives the response from the server:

bedrock <- bedrockruntime(
  region = "us-east-1"
)

body <- list(prompt = "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:", 
             max_tokens_to_sample = 100,
             anthropic_version = "bedrock-2023-05-31") |>
  toJSON(auto_unbox = TRUE)

response <- bedrock$invoke_model(body = body,
                        accept = "application/json",
                        contentType = "application/json",
                        modelId = "anthropic.claude-v2:1")

Issue 2: The received and parsed response is in binary format but should be JSON instead:

> response
$body
  [1] 7b 22 63 6f 6d 70 6c 65 74 69 6f 6e 22 3a 22 20 57 68 79 20 63 61 6e 27 74 20 61 73 74 72 6f 6e 61
 [34] 75 74 73 20 74 65 6c 6c 20 6a 6f 6b 65 73 20 74 69 6d 69 6e 67 3f 20 42 65 63 61 75 73 65 20 69 74
 [67] 27 73 20 61 6c 6c 20 61 62 6f 75 74 20 74 68 65 20 64 65 6c 69 76 65 72 79 21 22 2c 22 73 74 6f 70
[100] 5f 72 65 61 73 6f 6e 22 3a 22 73 74 6f 70 5f 73 65 71 75 65 6e 63 65 22 2c 22 73 74 6f 70 22 3a 22
[133] 5c 6e 5c 6e 48 75 6d 61 6e 3a 22 7d

$contentType
logical(0)

Current workaround: I need to call rawToChar() explicitly to convert the response to JSON:

> response$body |> rawToChar() |> fromJSON() |> {\(x) x[["completion"]]}()
[1] " Why can't astronauts tell jokes timing? Because it's all about the delivery!"

Thanks for looking into this!

DyfanJones commented 4 months ago

Sorry about that @alex23lemm and thanks for raising this issue. It looks like we will need a custom handler function for bedrockruntime. I will have a little look at botocore to see if they have something there :)

DyfanJones commented 4 months ago

My guess is we are missing these default content-type and accept header:

RestJSONSerializer: https://github.com/boto/botocore/blob/48c1bcfdd752f5fa965451e58103ccebad3e165b/botocore/serialize.py#L671-L700

add default accept header: https://github.com/boto/botocore/blob/48c1bcfdd752f5fa965451e58103ccebad3e165b/botocore/handlers.py#L673-L676

DyfanJones commented 4 months ago

Hi @alex23lemm,

I believe I have an initial implementation.

remotes::install_github("DyfanJones/paws/paws.common", ref = "application/json")

library(paws)
client <- bedrockruntime(config(credentials(profile = "paws")))

body <- list(prompt = "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:") |>
  jsonlite::toJSON(auto_unbox = TRUE)

response <- client$invoke_model(
  body = body,
  modelId = "ai21.j2-ultra"
)

jsonlite::fromJSON(rawToChar(response$body))
#> $id
#> [1] 1234
#> 
#> $prompt
#> $prompt$text
#> [1] "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:"
#> 
#> $prompt$tokens
#>    generatedToken.token generatedToken.logprob generatedToken.raw_logprob
#> 1           <|newline|>            -4.71495771                -4.71495771
#> 2           <|newline|>            -0.06756511                -0.06756511
#> 3                ▁Human            -9.61753464                -9.61753464
#> 4                     :            -5.56536293                -5.56536293
#> 5              ▁Tell▁me            -7.38631439                -7.38631439
#> 6              ▁a▁funny            -6.16015053                -6.16015053
#> 7           ▁joke▁about            -4.98311758                -4.98311758
#> 8          ▁outer▁space            -8.84491539                -8.84491539
#> 9           <|newline|>            -0.02789355                -0.02789355
#> 10          <|newline|>            -2.34469557                -2.34469557
#> 11           ▁Assistant            -5.89183331                -5.89183331
#> 12                    :            -0.01577717                -0.01577717
#>    topTokens textRange.start textRange.end
#> 1         NA               0             1
#> 2         NA               1             2
#> 3         NA               2             7
#> 4         NA               7             8
#> 5         NA               8            16
#> 6         NA              16            24
#> 7         NA              24            35
#> 8         NA              35            47
#> 9         NA              47            48
#> 10        NA              48            49
#> 11        NA              49            58
#> 12        NA              58            59
#> 
#> 
#> $completions
#>                                                                   data.text
#> 1  They say the earth is round,\n\nBut it's square,\nThey think it's flat\n
#>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                data.tokens
#> 1 ▁They▁say, ▁the▁earth, ▁is, ▁round, ,, <|newline|>, <|newline|>, ▁But, ▁it's, ▁square, ,, <|newline|>, ▁They▁think, ▁it's, ▁flat, <|newline|>, -7.92391395568848, -1.7487964630127, -0.306633919477463, -0.0392366088926792, -0.931648254394531, -1.87512421607971, -0.141698807477951, -0.738928914070129, -2.99068760871887, -0.54417484998703, -2.74556589126587, -1.28033459186554, -9.45336151123047, -0.948258876800537, -0.110951364040375, -2.02394270896912, -7.92391395568848, -1.7487964630127, -0.306633919477463, -0.0392366088926792, -0.931648254394531, -1.87512421607971, -0.141698807477951, -0.738928914070129, -2.99068760871887, -0.54417484998703, -2.74556589126587, -1.28033459186554, -9.45336151123047, -0.948258876800537, -0.110951364040375, -2.02394270896912, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 0, 9, 19, 22, 28, 29, 30, 31, 34, 39, 46, 47, 48, 58, 63, 68, 9, 19, 22, 28, 29, 30, 31, 34, 39, 46, 47, 48, 58, 63, 68, 69
#>   finishReason.reason finishReason.length
#> 1              length                  16

^{Created on 2024-02-20 with reprex v2.1.0}

DyfanJones commented 4 months ago

Are you sure the returning object should be a json? When you say it should return json do you mean a Json string? Or a json but in bytes similar to what Boto3 is doing.

import boto3
import json

bedrock = boto3.Session(profile_name="paws").client(service_name="bedrock-runtime")
body = json.dumps(
    {
        "prompt": "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:"
    }
)

response = bedrock.invoke_model(body=body, modelId="ai21.j2-ultra")
obj = response.get("body").read()

type(obj)
#> bytes

This is basically what paws is returning. It is returning a raw vector that is in json format.

library(paws)
client <- bedrockruntime(config(credentials(profile = "paws")))

body <- list(prompt = "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:") |>
  jsonlite::toJSON(auto_unbox = TRUE)

response <- client$invoke_model(
  body = body,
  modelId = "ai21.j2-ultra"
)

yyjsonr::read_json_raw(response$body)
#> $id
#> [1] 1234
#> 
#> $prompt
#> $prompt$text
#> [1] "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:"
#> 
#> $prompt$tokens
#>                                           generatedToken topTokens textRange
#> 1      <|newline|>, -4.71495771408081, -4.71495771408081      NULL      0, 1
#> 2  <|newline|>, -0.0675651058554649, -0.0675651058554649      NULL      1, 2
#> 3           ▁Human, -9.61753463745117, -9.61753463745117      NULL      2, 7
#> 4                :, -5.56536293029785, -5.56536293029785      NULL      7, 8
#> 5         ▁Tell▁me, -7.38631439208984, -7.38631439208984      NULL     8, 16
#> 6           ▁a▁funny, -6.1601505279541, -6.1601505279541      NULL    16, 24
#> 7      ▁joke▁about, -4.98311758041382, -4.98311758041382      NULL    24, 35
#> 8     ▁outer▁space, -8.84491539001465, -8.84491539001465      NULL    35, 47
#> 9  <|newline|>, -0.0278935506939888, -0.0278935506939888      NULL    47, 48
#> 10     <|newline|>, -2.34469556808472, -2.34469556808472      NULL    48, 49
#> 11      ▁Assistant, -5.89183330535889, -5.89183330535889      NULL    49, 58
#> 12           :, -0.0157771650701761, -0.0157771650701761      NULL    58, 59
#> 
#> 
#> $completions
#>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               data
#> 1  Mars is red\n\nWhy can't trees grow on Mars?\nBecause it's only fit for is, ▁Mars, -7.97078895568848, -7.97078895568848, ▁is, -1.78697526454926, -1.78697526454926, ▁red, -0.91144585609436, -0.91144585609436, <|newline|>, -0.178860738873482, -0.178860738873482, <|newline|>, -0.128698348999023, -0.128698348999023, ▁Why, -2.22174859046936, -2.22174859046936, ▁can't, -1.63458228111267, -1.63458228111267, ▁trees, -4.07934379577637, -4.07934379577637, ▁grow, -0.0818066596984863, -0.0818066596984863, ▁on▁Mars, -0.264529824256897, -0.264529824256897, ?, -0.0579109378159046, -0.0579109378159046, <|newline|>, -0.163965195417404, -0.163965195417404, ▁Because, -0.254661977291107, -0.254661977291107, ▁it's▁only, -5.48099517822266, -5.48099517822266, ▁fit▁for, -1.99698293209076, -1.99698293209076, ▁is, -3.21487283706665, -3.21487283706665, 0, 5, 5, 8, 8, 12, 12, 13, 13, 14, 14, 17, 17, 23, 23, 29, 29, 34, 34, 42, 42, 43, 43, 44, 44, 51, 51, 61, 61, 69, 69, 72
#>   finishReason
#> 1   length, 16

^{Created on 2024-02-20 with reprex v2.1.0}

DyfanJones commented 4 months ago

Closing as paws.common 0.7.1 is now on the cran

alex23lemm commented 4 months ago

Hi @DyfanJones , thanks for the fix and apologies for the delayed response! I was on a lengthy business trip with limited hands-on-keyboard time. You were correct with your remark regarding the return object, which should and now is a json in raw bytes. Your fixes solved the issues described above.

After the latest release of paws.common-v0.7.1, I only found one minor difference in paws compared to boto3 for invoke_model.

The default contentType value is application/json (see boto3 documentation here), as shown below, re-using the Python example from above.

response = bedrock.invoke_model(body=body, modelId="anthropic.claude-v2:1")
print(response.get("contentType"))`

application/json

However, the R equivalent returns the following:

response <- bedrock$invoke_model(body = body, modelId = "anthropic.claude-v2:1")
response$contentType

logical(0)

The behavior stays the same even when setting the contentType explicitly:

response <- bedrock$invoke_model(body = body, contentType = "application/json", modelId = "anthropic.claude-v2:1")
response$contentType

logical(0)

DyfanJones commented 4 months ago

Interesting, I will raise another ticket so we don't lose that bug

alex23lemm commented 4 months ago

Thanks!

paws-r / paws

Two MIME type issues in the Amazon BedrockRuntime client #749