lastmile-ai / aiconfig

AIConfig is a config-based framework to build generative AI applications.
https://aiconfig.lastmileai.dev
MIT License
901 stars 70 forks source link

[python] Fix PaLM Text Parser Completion Params #736

Open rholinshead opened 6 months ago

rholinshead commented 6 months ago

Currently, PaLMTextParser calls refine_chat_completion_params to deserialize model settings from the Prompt. refine_completion_params is defined but unused. Note that refine_completion_params's implementation is identical to refine_chat_completion_params and is likely incorrect since it defines the following as supported keys:

    supported_keys = {
        "candidate_count",
        "examples",
        "model",
        "temperature",
        "top_k",
        "top_p",
        "context",
    }

Despite palm's generate_text method supporting the following:

def generate_text(
    *,
    model: model_types.AnyModelNameOptions = DEFAULT_TEXT_MODEL,
    prompt: str,
    temperature: float | None = None,
    candidate_count: int | None = None,
    max_output_tokens: int | None = None,
    top_p: float | None = None,
    top_k: float | None = None,
    safety_settings: safety_types.SafetySettingOptions | None = None,
    stop_sequences: str | Iterable[str] | None = None,
    client: glm.TextServiceClient | None = None,
) -> text_types.Completion:
rossdanlm commented 6 months ago

The code for the completion params are defined in here: https://github.com/google/generative-ai-python/blob/7698c69d7e694b1da9a516db6b843553cec1bc58/google/generativeai/text.py#L132-L144

In general, these how how model parsers work overall:

  1. general config.run() is called, selecting a specific model parser: https://github.com/lastmile-ai/aiconfig/blob/4055228242640e3ee4bbd73817f927c493bf05b1/python/src/aiconfig/Config.py#L263-L279
  2. The specific implementation of a model provider's run method is invoked, in this case it's the PaLM parser: https://github.com/lastmile-ai/aiconfig/blob/4055228242640e3ee4bbd73817f927c493bf05b1/python/src/aiconfig/default_parsers/palm.py#L134
  3. Each model parser will call into the specific model's API (this is the step when we build model parsers, we need to look at public API docs to understand how to interact with their code). We send in a prompt and the API "completes" a text-based response back to us. This is why the arguments that we pass into a model's API is called "completion params". Ex: Here we call palm.generate_text() as the API call, passing in completion_data: https://github.com/lastmile-ai/aiconfig/blob/4055228242640e3ee4bbd73817f927c493bf05b1/python/src/aiconfig/default_parsers/palm.py#L136

This task specifically involves changing the supported completion params to now also match the ones that the code supports, not just what is listed on the API docs, which is out of sync with the code: https://cloud.google.com/vertex-ai/docs/generative-ai/start/quickstarts/api-quickstart#request_body

You can test this by running the cookbook we have for PaLM and making sure it still runs as expected.

Part 2 is you should also update the typescript library for this in https://github.com/lastmile-ai/aiconfig/blob/4055228242640e3ee4bbd73817f927c493bf05b1/typescript/lib/parsers/palm.ts#L207C17-L207C43

I can give you more details on how to test this later, but for now I think it's good skill to learn how to figure out how to do it on your own, so I would try doing this:

  1. Look through BLAME history of code and see other people's past PRs for these files and try to see how they tested their code: https://github.com/lastmile-ai/aiconfig/blame/4055228242640e3ee4bbd73817f927c493bf05b1/python/src/aiconfig/default_parsers/palm.py#L171
rossdanlm commented 6 months ago

Going to assign this to @Victor-Su-Ortiz