Keys are now all in server.js and index.js pulls the HeyGen Keys it needs from server

We need to create a onetime token and only use that on the server. There is an API to do this.

However starting up a session in the Streaming API is handled differently to here. Here the setup is

low level calls. In NextJS version they use Streaming API but its a bit opaque as to its use.

Special Agent with VectorDB knowledge base. Using OpenAI Assistant. Agent

Generate Assistant in OpenAI Playground and Also Drag and Drop files to create VectorDB (10,000 files allowed)

Assistant has Vector DB attached. (There are test documents in directory "testdata" that can answer a question "What is CrewAI". Add them to your agent in the OpenAI Playground )

See Video https://drive.google.com/file/d/1i3JI3gijyGKwAzP011ANOzgGRj90fj2g/view?usp=sharing

New Threads can be generated for Assistant and they will access VectorDB

Test by asking LLM "What is your source". It will respond - "Your files"

This all works with streaming Avatar

Streaming Avatar Demo

NEW: We have an SDK now! The SDK is an NPM package that you can easily add to your website and use for Streaming Avatar functionality. Please check it out at https://www.npmjs.com/package/@heygen/streaming-avatar

We also have an example React app demonstrating the SDK's functionality. That is located at: https://github.com/HeyGen-Official/StreamingAvatarTSDemo

Introduction

This HeyGen Streaming Avatar demo is a starting point from which developers can adapt and build streaming sessions into their own websites and experiences.

Below, we have outlined a few questions that frequently pop up when users first try out this demo. Below that FAQ, you will find directions for installing and running this demo.

Getting Started FAQ

How do I get an API Key?

Either an an API Key or Trial Token from HeyGen is required to run this Streaming API demo. API Keys are reserved for Enterprise customers, whereas both Creator and Teams plan users can activate and use a Trial token. You can retrieve either the API Key or Trial Token by logging in to HeyGen and navigating to this page in your settings: https://app.heygen.com/settings?nav=API

Which Avatars can I use with this project?

By default, there are several Public Avatars that can be used in Streaming. (AKA Streaming Avatars.) You can find the Avatar IDs for these Public Avatars by navigating to app.heygen.com/streaming-avatar and clicking 'Select Avatar'.

In order to use a private Avatar created under your own account in Streaming, it must be upgraded to be a Streaming Avatar. Only 1. Finetune Instant Avatars and 2. Studio Avatars are able to be upgraded to Streaming Avatars. This upgrade is a one-time fee and can be purchased by navigating to app.heygen.com/streaming-avatar and clicking 'Select Avatar'.

Photo Avatars are not compatible with Streaming and cannot be used.

Which voices can I use?

Most of HeyGen's AI Voices can be used with the Streaming API. To find the Voice IDs that you can use, please use the List Voices v2 endpoint from HeyGen: https://docs.heygen.com/reference/list-voices-v2

Why am I encountering issues with testing?

Most likely, you are hitting your concurrent session limit. While testing the Streaming API, your account is limited to 3 concurrent sessions. Please endeavor to close unused Streaming sessions with the Close Session endpoint when they are no longer being used; they will automatically close after some minutes.

You can check how many active sessions you have open with the List Sessions endpoint: https://docs.heygen.com/reference/list-sessions

Installing the Demo

1. Clone the repository.

   git clone https://github.com/HeyGen-Official/StreamingAvatar.git

2. Open the index.js file and replace 'YourApiKey' with your API key:

   "apiKey": "YourApiKey";

3. (optional) Open the server.js file and set your OpenAI API key to use talk mode:

   const openai = new OpenAI({
    apiKey: "<your openai api key>",
   });

4. open a terminal in the folder and then install the express and run the server.js:

   npm install express
   node server.js

5. you will see App is listening on port 3000!.

Using the Demo

1. Open the web browser and enter the http://localhost:3000to start the demo.
2. Click the "New" button to create a new session. The status updates will be displayed on the screen.
3. After the session is created successfully, click the "Start" button to start streaming the avatar.
4. To send a task to the avatar, type the text in the provided input field and click the "Repeat Text" button.
5. In order to use Talk mode, set your OpenAI key in server.js before starting the server and click "Talk" button
6. Once done, click the "Close Connection" button to close the session.

Remember, this is a demo and should be modified according to your needs and preferences. Happy coding!

Troubleshooting

In case you face any issues while running the demo or have any questions, feel free to raise an issue in this repository or contact our support team.

Please note, if you encounter a "Server Error", it could be due to the server being offline. In such cases, please contact the service provider.