giusedroid
commented
4 months ago Thank you so much for taking the time to test this out.
I'll start replying to the most immediate question. Please stay tuned for the others!
Getting a session token
I usually use AWS Amplify and Cognito Identity Pools. I am pasting some code below to illustrate what I mean with a sample react app, then I'll add some links to get started.
import { useState, useEffect } from 'react'
// (1) import components from Amplify UI
import {
Authenticator,
Flex,
Button,
Input,
Divider,
useTheme
}
from '@aws-amplify/ui-react';
// (2) import the StorageManager component
import { StorageManager } from '@aws-amplify/ui-react-storage';
// (3) get basic styles from Amplify UI
import '@aws-amplify/ui-react/styles.css';
import { Amplify, Auth } from 'aws-amplify';
import awsExports from './aws-exports';
Amplify.configure(awsExports);
import { fetchEventSource } from "@microsoft/fetch-event-source";
import { SignatureV4 } from "@smithy/signature-v4";
import { Sha256 } from "@aws-crypto/sha256-js";
import '@aws-amplify/ui-react/styles.css';
function Header({ signOut, user }) {
const { tokens } = useTheme();
return (
<Flex
backgroundColor={tokens.colors.purple[60]}
direction="row"
justifyContent="space-between"
>
<h3>Hello {user.attributes.email}!</h3>
<Button
colorTheme="overlay"
onClick={signOut}>Sign out</Button>
</Flex>
);
}
function GenAI() {
const { tokens } = useTheme();
const [creds, setCreds] = useState({})
const [searchQuery, setSearchQuery] = useState();
const [chat, setChat] = useState([]);
useEffect(() => {
Auth.currentCredentials().then(setCreds)
})
const streamData = async () => {
setChat([]);
const sigv4 = new SignatureV4({
service: "lambda",
region: creds.identityId.split(":")[0],
credentials: creds,
sha256: Sha256
});
// INSERT YOUR LAMBDA URL HERE
const apiUrl = new URL("<LAMBDA STREAMING URL HERE>");
const query = document.getElementById("searchQuery").value;
setSearchQuery(query);
let body = JSON.stringify({
query: query,
// Can use any Bedrock available models
model: "anthropic.claude-instant-v1",
// This is required for the @microsoft/fetch-event-source library to understand the streaming response
streamingFormat: "fetch-event-source"
});
let signed = await sigv4.sign({
body,
method: "POST",
hostname: apiUrl.hostname,
path: apiUrl.pathname.toString(),
protocol: apiUrl.protocol,
headers: {
"Content-Type": "application/json",
host: apiUrl.hostname
}
});
await fetchEventSource(apiUrl.origin, {
method: signed.method,
headers: signed.headers,
body,
onopen(res) {
if (res.ok && res.status === 200) {
console.log("Connection made ", res);
} else if (
res.status >= 400 &&
res.status < 500 &&
res.status !== 429
) {
console.log("Client-side error ", res);
}
},
onmessage(event) {
setChat((data) => [...data, event.data]);
// Important to set the data this way, otherwise old data may be overwritten if the stream is too fast
},
onclose() {
console.log("Connection closed by the server");
},
onerror(err) {
console.log("There was an error from server", err);
}
});
}
return (
<>
<div>
<Input
id="searchQuery"
placeholder="ask a question"
fontWeight={tokens.fontWeights.bold}
fontSize={tokens.fontSizes.xl}
padding="xl"
textColor={tokens.colors.purple[60]}
border={`1px solid ${tokens.colors.purple[60]}`}
/>
</div>
<div>
<Button variant="primary" onClick={() => streamData()}>Submit Question</Button>
</div>
<div className="qa_container">
<div>
<b>Question:</b> {searchQuery}
</div>
<div>
<b>Response:</b> {chat.join("")}
</div>
</div>
</>
)
}
function MainComponent({ signOut, user }) {
return (
<Flex direction="column">
<Header user={user} signOut={signOut} backgroundcolor="purple.80"/>
<Divider
label="RAG Source - LanceDB backed by S3"
size="large"
orientation="horizontal" />
<Flex
direction="column"
justifyContent="space-around"
alignItems="center"
>
<StorageManager
acceptedFileTypes={[
// you can list file extensions:
'.pdf',
// or MIME types:
]}
accessLevel="private"
maxFileCount={5}
// Size is in bytes
maxFileSize={10000000}
/>
</Flex>
<Divider
label="LLM Prompt - Amazon Bedrock"
size="large"
orientation="horizontal" />
<Flex
direction="column"
justifyContent="space-around"
alignItems="center"
>
<GenAI />
</Flex>
</Flex>
)
}
export default function App() {
return (
<Authenticator>
{({signOut, user}) => <MainComponent user={user} signOut={signOut} />}
</Authenticator>
);
}Make sure you have configured authentication and pulled your amplify configs.
For this sample to work, you must have an aws-exports.json file in the src/ folder.
You'll get this by creating an Amplify application and pulling it into this repo.
Here's how to get started with Amplify and it's authentication system:
https://docs.amplify.aws/react/start/getting-started/ https://docs.amplify.aws/react/build-a-backend/auth/set-up-auth/
my package.json
{
"name": "rag-app",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"lint": "eslint . --ext js,jsx --report-unused-disable-directives --max-warnings 0",
"preview": "vite preview"
},
"dependencies": {
"@aws-amplify/ui-react": "^5.3.1",
"@aws-amplify/ui-react-storage": "^2.3.1",
"@aws-crypto/sha256-js": "5.2.0",
"@microsoft/fetch-event-source": "2.0.1",
"@smithy/signature-v4": "2.0.12",
"aws-amplify": "^5.3.11",
"loader-utils": "^3.2.1",
"react": "18.2.0",
"react-dom": "18.2.0"
},
"devDependencies": {
"@types/react": "^18.2.15",
"@types/react-dom": "^18.2.7",
"@vitejs/plugin-react": "^4.0.3",
"eslint": "^8.45.0",
"eslint-plugin-react": "^7.32.2",
"eslint-plugin-react-hooks": "^4.6.0",
"eslint-plugin-react-refresh": "^0.4.3",
"vite": "^4.4.5"
}
}
Vite config --> ./vite.config.js
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
// https://vitejs.dev/config/
export default defineConfig({
plugins: [react()],
server:{port: 8080},
define:{global:{}}
})
Analect
@shafkevi @giusedroid ... thanks for these two sister projects to explore a serverless rag approach leveraging bedrock with lancedb. I watched your session on the AWS Innovate event and decided to explore them further. There is a good mix of approaches across both repos that present different ways to combine these technologies using both local and remote (lambda-based) components. I'll cover it off in some questions below, but bringing the best parts of both projects into one master one would be great.
Data Ingest
Rather than the 'bedrock-docs' example, I decided to use the EU AI Act, a circa 250 page PDF to test how this approach works.
I think this screenshot below as related to the ingestion from Lambda kicked-off by a file being added to S3 in the sister project. It seems it was taking in chunk sizes bigger than the 1000 specified and perhaps it wasn't capturing generating embeddings for the full chunk. Did you experience anything similar?
Getting a Session Token
I'm more used to just supplying a key and secret-key as credentials. This is how I went about getting a session-token
aws sts get-session-token --duration-seconds 43200 --profile default. Is there a better way? Eventhough this should last me 12 hours, I was sometimes getting a token-expired message what seemed like well before that window closed. Not sure why, but it required me to generate a new token each time.Token expired response:
Inconsistent Responses
The response I was getting from Claude Instant I felt were not that impressive, so I was trying to understand the best approach to peer under the hood a bit more in terms of what was happening. It seems to me that not that much context was being returned from lancedb to feed the prompt to Claude.
However, there were also instances where I was re-asking the same questions (as per below) and the responses were inconsistent - some short, some longer ... not necessarily in any given order. Did either of you experience any of these problems when experimenting?
Worst response:
Poor response:
Best response:
Questions
Thanks for your guidance. Colum