Amplify AI RFC

[dbanksdesign]

The purpose of this RFC is to get early feedback on this new full-stack AI functionality in Amplify. This functionality is currently in developer preview while we get feedback and iterate on it. There will likely be changes and potentially some breaking changes before going general availability (GA). We will keep this issue updated as we add functionality/address feedback so stay tuned!

Overview

We are introducing a new AI category to Amplify Gen 2 with backend definitions, client libraries, and UI components. With Amplify AI you can define full-stack AI functionality in their Amplify Gen 2 applications with tight integration for auth and data, allowing for easy authorization logic and using existing Amplify data as context for AI requests (like asking a chatbot about data in your applications) and generating data records with AI. Also, with Amplify Gen 2’s infrastructure-from-code approach built on top of CDK, developers can have cloud sandboxes that are deployed during local development and fully deployed with Amplify CD when a git branch is pushed.

Amplify AI is built around the idea of “routes”. An AI route is like an API endpoint for interacting with backend AI functionality. AI routes are configured in an Amplify backend where you can define the authorization rules, what type of route (generation or conversation), AI model and inference configuration like temperature, what are the inputs and outputs, and what data it has access to. There are 2 types of AI routes we are focusing on for dev preview:

1. Conversation: A conversation route is an asynchronous, multi-turn API. A conversation is made of sessions and messages, which Amplify will define. Conversations and messages are automatically stored in DynamoDB Examples of this are any chat-based AI experience.
2. Generation: A single synchronous request-response API. A generation route is just an AppSync Query. Examples of this are: generating alt text for an image, generating structured data from unstructured input, summarization, etc.

Setup

First, make sure your AWS account has access to the model you wish to use. You can do that by going in to the Bedrock console and requesting access.

Also, make sure your account is set up for Amplify: https://docs.amplify.aws/react/start/account-setup/

Quickstart

You can use this repo as a template: https://github.com/dbanksdesign/amplify-ai-starter to get started. Create a new Github repository from it and clone it locally and follow the steps in the README

Installation

Adding to an existing app

If you have an existing Amplify Gen2 application, install the latest Amplify dependencies:

npm i --save-dev @aws-amplify/backend @aws-amplify/backend-cli @aws-sdk/client-bedrock-runtime
npm i aws-amplify

You will need to directly depend on "@aws-sdk/client-bedrock-runtime" as a devDependency right now. We plan on addressing this issue as soon as possible.

Creating a new app

You can follow the quickstart guide here: https://docs.amplify.aws/react/start/quickstart/ or create a new app using the CLI:

npm create amplify@latest

If you are using React, we have a new React AI package you can install that will have React hooks and components for using AI functionality.

npm i @aws-amplify/ui-react-ai

Usage

Building a chatbot

1. Backend definition

To get started adding AI functionality to your Amplify application, open your amplify/data/resource.ts and add a new entry in your schema using a.conversation which will add a conversation route to your data backend.

const schema = a.schema({
+  // This will add a new conversation route to your Amplify data backend
+  chat: a.conversation({
+    aiModel: a.ai.model('Claude 3 Haiku'),
+    systemPrompt: `You are a helpful assistant`,
+  }),

2. Frontend

Then in your React code you can import and use the AIConversation component

import { generateClient } from "aws-amplify/data";
import { createAIHooks, AIConversation } from '@aws-amplify/ui-react-ai';
import type { Schema } from "../amplify/data/resource";

const client = generateClient<Schema>({ authMode: 'userPool' });
const { useAIConversation } = createAIHooks(client);

export default function Page() {
  const [
    {
      data: { messages },
    },
    sendMessage,
  ] = useAIConversation('chat');
  // 'chat' here should be the key in your schema

  return (
        <AIConversation
            messages={messages}
            handleSendMessage={sendMessage}
        />
    )
}

If you are not using React in your frontend, check out the section for using the Amplify Javascript libraries.

You should now be able to see a chatbot component!

4. Adding tools

Tools are functions/APIs that LLMs can choose to invoke to get information/data about the world. This allows them to be able to answer questions that were not in their training data (like the current weather or information, application-specific data, etc.). The default way you can define “tools” for the assistant to use is with data models and custom queries in your data schema.

In your amplify/data/resource.ts file, lets add a custom query our chatbot can use:

+export const getWeather = defineFunction({
+  name: 'getWeather',
+  entry: './getWeather.ts',
+});

const schema = a.schema({
+  getWeather: a.query()
+    .arguments({ city: a.string() })
+    .returns(a.string())
+    .handler(a.handler.function(getWeather))
+    .authorization((allow) => allow.authenticated()),

  chat: a.conversation({
    aiModel: a.ai.model('Claude 3 Haiku'),
    systemPrompt: `You are a helpful assistant`,
+    tools: [
+      {
+        query: a.ref('getWeather'),
+        description: 'Provides the current weather for a given city.'
+      },
+    ]
  }),

And then we will need to create the getWeather function:

export const handler = async () => {
  return {
    value: 42,
    unit: 'C'
  };
}

Now if we ask our chatbot what the weather is like in San Jose, it can respond! Although this functionality is mocked, you can use a library like https://agentic.so/ to handle calling APIs or performing math equations.

5 Adding context

The client can send back “context” to the assistant so that it can provide better answers to the user. This context can really be any unstructured data that might be helpful. Some examples include:

• Information about the user like name, location, etc
• What page they are currently on and what information it has on it
• The current state of the client-side application

This context gets passed into the system prompt of the request to the LLM.

import { generateClient } from "aws-amplify/data";
import { createAIHooks, AIConversation } from '@aws-amplify/ui-react-ai';
import type { Schema } from "../amplify/data/resource";

const client = generateClient<Schema>({ authMode: 'userPool' });
const { useAIConversation } = createAIHooks(client);

export default function Page() {
  const [
    {
      data: { messages },
    },
    sendMessage,
  ] = useAIConversation('chat');
  // 'chat' here should be the key in your schema

+  const handleSendMessage = (content) => {
+    sendMessage({
+      content, 
+      aiContext: {
+        name: 'Danny'
+      },
+    })
+  }

  return (
        <AIConversation
            messages={messages}
-            handleSendMessage={sendMessage}
+            handleSendMessage={handleSendMessage}
        />
    )
}

6. Adding custom UI responses

Wouldn’t it be cool if our chatbot could respond with custom UI components? On the AIConversation component you can add a responseComponents prop that tells the LLM it can respond with those React components and they will render in the chat history.

<AIConversation
   //... 
    responseComponents={{
        Weather: {
            description: "Used to display the weather",
            component: (props) => {
                return (
                    <Flex
                        direction="row"
                        color="font.tertiary"
                        alignItems="center"
                    >
                        <Text fontWeight="light" fontSize="large">
                            {props.value}
                        </Text>
                    </Flex>
                );
            },
            props: {
                value: { type: "string" },
            },
        },
    }}

Recipe generator

1 Backend

Add a new generation route in your data schema in amplify/data/resource.ts file:

    generateRecipe: a.generation({
        aiModel: a.ai.model('Claude 3 Haiku'),
        systemPrompt: 'You are a helpful assistant that generates recipes.',
    })
    .arguments({
        description: a.string(),
    })
    .returns(
        a.customType({
            name: a.string(),
            ingredients: a.string().array(),
            instructions: a.string(),
        })
    )
    .authorization((allow) => allow.authenticated())

2. Frontend

import * as React from "react";
import {
  Button,
  Flex,
  Heading,
  Loader,
  Text,
  TextAreaField,
  View,
} from "@aws-amplify/ui-react";
import { createAIHooks } from "@aws-amplify/ui-react-ai";
import { generateClient } from "aws-amplify/api";
import type { Schema } from "../amplify/data/resource";

const client = generateClient<Schema>({ authMode: "userPool" });
const { useAIGeneration, useAIConversation } = createAIHooks(client);

export default function Page() {
  const [description, setDescription] = React.useState("");
  const [{ data, isLoading, hasError }, generateRecipe] =
    useAIGeneration("generateRecipe");

  const handleClick = async () => {
    generateRecipe({ description: "" });
  };

  return (
    <Flex direction="column">
      <Flex direction="row">
        <TextAreaField
          autoResize
          value={description}
          onChange={(e) => setDescription(e.target.value)}
          label="Description"
        />
        <Button onClick={handleClick}>Generate recipe</Button>
      </Flex>
      {isLoading ? (
        <Loader variation="linear" />
      ) : (
        <>
          <Heading level={2}>{data?.name}</Heading>
          <View as="ul">
            {data?.ingredients?.map((ingredient) => (
              <Text as="li" key={ingredient}>
                {ingredient}
              </Text>
            ))}
          </View>
          <Text>{data?.instructions}</Text>
        </>
      )}
    </Flex>
  );
}

---

Detailed usage

Backend definition

const schema = a.schema({
  chat: a.conversation({
    aiModel: a.ai.model('Claude 3 Haiku'),
    systemPrompt: `You are a helpful assistant`,
    // all inference configuration is optional,
    // if it is not provided it will use the Bedrock defaults
    inferenceConfiguration: {
      maxTokens: 1000,
      temperature: 1,
      topP: 0.5,
    }
  }),

  generateRecipe: a.generation({
    aiModel: a.ai.model('Claude 3 Haiku'),
    systemPrompt: 'You are a helpful assistant that generates recipes.',
  })
  .arguments({
      description: a.string(),
  })
  .returns(
      a.customType({
          name: a.string(),
          ingredients: a.string().array(),
          instructions: a.string(),
      })
  )
  .authorization((allow) => allow.authenticated()),

})

Javascript/TypeScript client

To interact with the AI backed in your frontend application, you can use the Amplify Data client. There are 2 new namespaces on the client: conversations and generations. These are properly typed based on your backend definition so if you have conversation and generation routes defined, they will show up in these namespaces.

import type { Schema } from "../amplify/data/resource";
import { generateClient } from "aws-amplify/data";

const client = generateClient<Schema>({ authMode: 'userPool' );

// create a conversation
const { data: conversation, errors } = await client.conversations.chat.create();

// listen for assistant responses
conversation.onMessage((message) => { 
  console.log('Assistant message received:', message);
});

// send a message
const { data: message, errors } = await conversation.sendMessage({
  content: [
    { text: '' },
  ],
});

// send a message with context
const { data: message, errors } = await conversation.sendMessage({
  content: [
    {
      text: 'Hello'
    }
  ],
  aiContext: {
    username: 'djb'
  }
})

// list conversations
const { data, errors } = await client.conversations.chat.list();

// resume a conversation
const { data: conversation, errors } = await client.conversations.chat.get('[id]');

// run a generation
const { data, errors } = await client.generations.generateRecipe({
    description: ''
});

---

Feedback

Amplify AI functionality is currently in developer preview and we would like feedback to fix any bugs, improve the APIs, and add missing functionality before going to general availability (GA). Please comment on this ticket with any suggestions you have for Amplify AI. We will be keeping this issue up to date as we make changes.

Thank you to all the people who have worked tirelessly on this project including, but not limited to: @atierian @cshfang @thaddmt @sobolk

[jamespanyp64]

1) What you built - we built a networking app and incorporated amplify AI to support AI chatBox. 2) What worked well - in general, we were able to receive replies from the AI bot which make sense most of the time. 3) What needs improvement - I'm not sure if it's because I miss anything or it's simply not supported yet, but I'd like to do some tuning with the AI to support for example "give me a summary of this user on what they do as a career and what their interests are, together with an evaluation for how much we match with each other". This would require the AI to be able to understand for example my own user profile and the other user's profile and do comparison, which isn't something supported yet I think?

[johnpc]

1. What you built: I've added a motivational AI chatbot to my open source iOS/web calorie tracking app https://jpc.fit.
2. What worked well: The most pleasant surprise was the chatbot tools feature, which allows the bot to use information about the specific foods and amounts that user eats to inform the conversation.
3. What needs improvement: I had some trouble configuring client-side tools. For example, I want to have a feature where the user tells the bot what they ate, and the bot will automatically report the food/calories for them by creating the dynamo records. I think a client side tool would be best for this, but ran into a snag when I was trying to get it working (example code)

I also tried AI generation in a separate app to generate jokes. However I can't figure out settings that makes the jokes funny! As it is, I can't seem to find a humorous ai configuration.

I hope image generation is also on the roadmap!

[ctodd]

1. What you built: Implementing the chatbot example into an existing NextJS 14/Gen 2 app.

2. What worked well:

2a. Looks like a great way to add a VERY simple chat function to an app.

2b. Glad to see the message history is stored in one row per interaction for linear observability.

3. What needs improvement:

3a. The initial example didn't work, I received a compile error:

./node_modules/@aws-amplify/ui-react-core/dist/esm/components/FormCore/FormProvider.mjs Attempted import error: 'useForm' is not exported from 'react-hook-form' (imported as 'useForm').

Since Amplify uses NextJS by default, "use client"; is required and was not provided in the example. Clarity around the examples would be helpful as there are too many options and nuances in the front end ecosystem.

3b. I was already using Tool is my data model. It seems like any reserved words should be prefixed with a Namespace to prevent conflict.

3c. For a chat use case, streaming should be supported out of the box as well as animations to show something is happening in the background. This is already implemented in other chatbots in the aws-examples repos. Perhaps this is already in progress...

3d. The existing routes are great for very simple use cases, but feel a little too opinionated. The underlying data model for the message history is a great start, but should allow custom data fields. The route interface should provide access to the API response metadata such as usage, metrics, and stopReason, and should also store that information in the message history. This allows for cost accounting, and exception handling if a response stopped for any other reason that "end_turn". Having access to the chat session ID would be useful if we want to relate that to other data stored elsewhere.

3e. Will we have access to GuardRails and Knowledgebases? Custom headers such as Anthropic's context caching (currently in Beta, not available in Bedrock).

3f. The model identifiers are generic compared to the versioned modelds used in the API. This may reduce options, not to mention cause some confusion with yet-another-way-to-refer-to-a-model.

3g. More documentation. The Gen 2 ecosystem is already very difficult to navigate, and the documentation feels very sparse while introducing new concepts without enough definitive examples. I know this is super early for this AI feature, but the more transparent you can be with what is supported under the hood, the more meaningful our experimentation can be without digging into the underlying source code.

3h. I feel like for any significant AI use case, the current approach is too limited. Aligning more with the full capabilities of the underlying API and setting reasonable defaults seems like the best approach. Being able to make use of the message history and ability to do evals would prevent plug-and-pray use of AI.

3i. Wishlist:

• Backend management example with UI to review chat sessions, message history, errors, costs.

• Controls to prevent abuse such as excessive usage (turns, tokens, etc), or repeated inappropriate requests?

• Ability to parse different output formats, such as code blocks?

• The artifacts design pattern in Claude.ai seems like it would be incredibly useful as a virtual whiteboard in conversations.

[mkdev10]

I used the Chat feature. It's excellent that it can be implemented quickly and easily, as expected from Gen2🚀

Desired features:

• AI assistant feedback options
  • AI feedback content evaluation and recording (e.g., like button)
  • Buttons for next conversation suggestions
• Feature for easy integration with Bedrock knowledge
• AI streaming responses
• Log streaming function for AI's thought process results

Looking forward to GA.​​​​

[mmuller88]

So I recently experimented with SST ion with create a next app and did some ai sdk (https://sdk.vercel.ai/) . Pretty impressive. For me, it looks like this new amplify AI functionality looks for the same/similar experience, right?

Yeah, I definitely would need the AI streaming response as well.

[alex-breen]

From what I've seen so far, AmplfyAI is an awesome extension of Amplify. Great for developers who want to accelerate the release of new features.

In my Amplify app, I generate RAG-style context locally, in-app, before I send queries (with context) to Bedrock or openAI. It would be great if Amplify AI could provide that functionality as a callable service.

This is what I am currently doing: I run a js transformer in-app to extract embeddings (using a Huggingface LLM) and store them (with Amplify DataStore). This processes whenever data changes in my app so that the vectors are always updated. When I need to query a GPT provider (Bedrock, openAI), I generate RAG context by calculating against the previously stored embeddings (e.g., cosine similarity). (DataStore is great for this design, as it means I can process rapidly, cheaply, and offline.)

If AmplifyAI could do the above as a callable service, that would be great. For example, one could enable AmplifyAI to automatically generate embeddings and indexes based on data stored within Amplify (DynamoDB). Then, when RAG-style queries are needed, one could use AmplifyAI to run calculations against the app's data set to return the RAG context that is included in the GPT query. Or...one could use the indexes for other in-app functionality that uses vectorized embeddings directly.

[amuresia]

Hi team! I have a question about tools: Is there an out of the box way to use data from our models or should functions that query the models be implemented prior?

Example schema

``` User: a .model({ email: a.string().required(), givenName: a.string().required(), familyName: a.string().required(), }) chat: a.conversation({ aiModel: a.ai.model('Claude 3 Haiku'), systemPrompt: 'You are a helpful assistant', tools: [ { query: a.ref('User'), description: 'Application users', }, ], }), ... ```

is throwing

Failed to instantiate data construct
Caused By: Tool "User" defined in @conversation directive has no matching Query field definition

[dbanksdesign]

@amuresia we are working on having model tools work like how you have in your code, but for now what you can do is change the ref to "list[model name]s" so query: a.ref('listUsers') and that should work!

[amuresia]

I came across a limitation in trying to show the conversation history by using the TypeScript client. Calling .listMessages() on a conversation only returns the content property aka the message sent by the user and not the assistantContent aka the message generated by the AI.

Screenshot

We can see that the content property is an array, supposably because the first item should be the user message and the second the generated message? The generated message is visible in DynamoDB under the assistantContent property.

Screenshot

[amuresia]

Hi team, could we please get an update on how to continue testing as for me things have stopped working last week after upgrading to the latest aws-amplify libraries.

chat: a
      .conversation({
        aiModel: a.ai.model('Claude 3 Haiku'),
        systemPrompt:
          'You are a helpful assistant.',
      })
      .authorization((allow) => allow.owner())

I noticed that the .sendMessage() function throws the following errors:

message: "Validation error of type UnknownType: Unknown type ContentBlockInput"
message: "Validation error of type UnknownType: Unknown type ToolConfigurationInput"

I reviewed https://github.com/aws-amplify/docs/pull/8094/files to check my implementation but I can't see anything out of the ordinary that could explain the above.

I also cloned https://github.com/aws-samples/amplify-ai-examples and ran but the Assistant never replies, the spinner doesn't stop turning. I see no errors in the console logs.

Screenshot