Paper QA Settings for AWS Bedrock

[istranic]

Hi, do you have any guidance or examples for the Settings parameter in order to use paper QA with entirely AWS Bedrock models. I'm able to access by Bedrock's LLM and embedding models using boto3 and litellm, but I can't figure out how to make it work in Paper QA Python API, and my errors are very generic and non-specific.

[dosubot[bot]]

The Paper QA repository does not provide specific examples or documentation for integrating AWS Bedrock models directly. However, you can configure custom language models using the Settings class. Here's a general approach you can take to set up AWS Bedrock models with Paper QA:

1. Configure Your Model: Use the Settings class to define your model configuration. You can specify the model name and parameters required to connect to AWS Bedrock using boto3 and litellm.

2. Example Configuration: Here's a hypothetical example of how you might set up the Settings for a custom model:

   from paperqa import Settings, ask
   
   bedrock_llm_config = dict(
      model_list=[
          dict(
              model_name="aws_bedrock_model",
              litellm_params=dict(
                  model="your-bedrock-model",
                  api_base="your-api-endpoint",
                  api_key="your-api-key",
                  temperature=0.1,
                  frequency_penalty=1.5,
                  max_tokens=512,
              ),
          )
      ]
   )
   
   answer = ask(
      "What manufacturing challenges are unique to bispecific antibodies?",
      settings=Settings(
          llm="aws_bedrock_model",
          llm_config=bedrock_llm_config,
          summary_llm="aws_bedrock_model",
          summary_llm_config=bedrock_llm_config,
      ),
   )

3. Adjust Parameters: Ensure that the parameters in litellm_params match the requirements for connecting to your AWS Bedrock models.

4. Error Handling: If you encounter generic errors, double-check your API endpoint, keys, and any other connection parameters. Ensure that your AWS credentials are correctly configured in your environment.

This approach is based on the general method for configuring custom models in Paper QA, as described in the repository's README ^[1]. Adjust the configuration to fit the specifics of your AWS Bedrock setup.

To continue talking to Dosu, mention @dosu.

[jamesbraza]

Hi @istranic can you post your stack trace(s)?

Also, check these litellm docs on how to use Bedrock, we are just passing through to them: https://docs.litellm.ai/docs/providers/bedrock

[istranic]

Hi @jamesbraza I made progress and solved some of the issues. I can now manually add and query docs with all models hosted locally.

litellm_embedding_model = LiteLLMEmbeddingModel(
    name="openai/sentence-transformers/all-mpnet-base-v2",
    config=embedding_config,

## Add Docs
  await docs.aadd(doc, 
                  settings=settings,
                  citation=doc, 
                  docname=doc,
                  embedding_model=litellm_embedding_model)

## Query
answer = await docs.aquery(
    "What is the Nonlinearity Error and why does it matter?",
    settings=settings,
    embedding_model=litellm_embedding_model
)

However, when I add docs using the method above, it's not doing any of the advanced parsing, correct? In order to benefit from advanced parsing, I need to use something like:

answer = await agent_query(
    QueryRequest(
        query="What manufacturing challenges are unique to bispecific antibodies?",
        settings=settings,
        embedding=litellm_embedding_model
    )

However, this doesn't work because embedding, embedding_model, etc. are not recognized arguments. If I try to specify the embedding model in the settings, it doesn't work either, and I get errors related to missing Open AI keys during the embedding step. What's the correct approach for specifying the litellm_embedding_model object in the agent_query API?

[jamesbraza]

it's not doing any of the advanced parsing

So I can understand, what do you mean by "advanced parsing"?

---

However, this doesn't work because embedding, embedding_model, etc. are not recognized arguments

Yeah when using agent_query you need to specify a QueryRequest which contains settings: Settings, kind of like we do here: https://github.com/Future-House/paper-qa#ask-manually

And for setting the embedding_model in Settings, start by reading here: https://github.com/Future-House/paper-qa#changing-embedding-model

So you won't be directly instantiating a LiteLLMEmbeddingModel, it will be instantiated for you by the inner workings of paper-qa given the Settings. Does that make sense?

[istranic]

I've seen the example for changing embedding models, and instantiating it via Settings make sense in principle. However, in the example provided, it doesn't highlight how to specify the embedding configuration for a custom model. I've tried various permutations of embedding, embeddings, embedding_model, embedding_config, but none seemed to work. If I have an OpenAI-compatible-model-API with an API base and API key, how do I pass that information into Settings, without using the LiteLLMEmbeddingModel object?

To rephrase my first question, is there any logical difference between adding documents manually and running docs.query, vs using the agent_query API above? In the readme section on Adding Manually, it says "If you prefer fine grained control, and you wish to add objects to the docs object yourself (rather than using the search tool)". The statement "rather than using the search tool" made me think that adding manually in technically inferior?

[istranic]

An issue in this line is that the LiteLLMEmbeddingModel class only passes config["kwargs"] to litellm API, and it's ignoring other parameters in the config dict. Maybe this is the root-cause of the issues.

In my working code that specifies the embedding settings using the LiteLLMEmbeddingModel object instead of Settings, I hack by dumping the config arguments into the kwargs key, but I doubt this is how the API is intended, and I'm not sure that upstream code is going that either, without which you'll get errors.

embedding_config = dict(
        kwargs = dict(
            api_base="abc",
            api_key="abc",
            num_retries=3,
            timeout=120
        )
)

litellm_embedding_model = LiteLLMEmbeddingModel(
    name="openai/sentence-transformers/all-mpnet-base-v2",
    config=embedding_config,

)