search

alexanderatallah / window.ai

Use your own AI models on the web
https://windowai.io
MIT License
782 stars 68 forks source link

readme

Window: use your own AI models on the web

Window AI is a browser extension that lets you configure AI models in one place and use them on the web.

More about why this was made here.

Below, you'll find out how to install, how to find apps, how to make apps, and how to connect custom models.

📺 Demo

https://user-images.githubusercontent.com/1011391/230610706-96755450-4a3b-4530-b19f-5ae405a31516.mp4

ℹ️ Contents

⭐️ Main features

⚙️ How it works

  1. You configure your keys and models just once in the extension (see demo above).

  2. Apps can request permission to send prompts to your chosen model via the injected window.ai library (see the simple docs).

  3. You maintain visibility on what's being asked and when.

It works with these models:

📥 Installation

Download the Chrome extension here: https://chrome.google.com/webstore/detail/window-ai/cbhbgmdpcoelfdoihppookkijpmgahag

Browser support

✅ Chrome ✅ Brave ✏️ Microsoft Edge ✏️ Firefox ✏️ Safari: https://github.com/alexanderatallah/window.ai/issues/20

Beta builds

You can join the #beta-builds channel on Discord to get early access to features being tested and developed by the community.

👀 Find apps

Better ways of doing this are coming soon, but today, you can use the Discord #app-showcase channel to discover new window.ai-compatible apps, or you can browse user-submitted ones on aggregators:

📄 Docs

This section shows why and how to get started, followed by a reference of window.ai methods.

Why should I build with this?

Infrastructure burden: No more model API costs, timeouts, rate limiting. Reduced server billing time.

Easily go multi-model. Integrate once, and then let Window handle model upgrades and support for other providers.

Privacy: Now you can build privacy-conscious apps that just talk to the user's choice of model, and you have less liability for the model's output.

Getting started

To leverage user-managed models in your app, simply call await window.ai.generateText with your prompt and options.

Example:

const [ response ] : Output[] = await window.ai.generateText(
    { messages: [{role: "user", content: "Who are you?"}] }: Input
  )

console.log(response.message.content) // "I am an AI language model"

All public types, including error messages, are available with comments in the window.ai library. Jump down to export interface WindowAI to see the type of the root object.

Input, for example, allows you to use both simple strings and ChatML.

Example of streaming GPT-4 results to the console:

const [{ message }] = await window.ai.generateText(
  {
    messages: [{ role: "user", content: "Who are you?" }]
  },
  {
    temperature: 0.7,
    maxTokens: 800,
    model: ModelID.GPT_4,
    // Handle partial results if they can be streamed in
    onStreamResult: (res) => console.log(res.message.content)
  }
)
console.log("Full ChatML response: ", message)

Note that generateText will return an array, Output[], that only has multiple elements if numOutputs > 1.

This does not guarantee that the length of the return result will equal numOutputs. If the model doesn't support multiple choices, then only one choice will be present in the array.

The onStreamResult handler is similar. You should rely on the promise resolution and only use this handler to improve UX, since not all models and config options support it.

Examples

Functions

The Window API is simple. Just a few functions:

Generate Text: generate text from a specified model or the user-preferred model.

window.ai.generateText(
    input: Input,
    options: CompletionOptions = {}
  ): Promise<Output[]>

Input is either a { prompt : string } or { messages: ChatMessage[]}. Examples: see getting started above.

Current model: get the user's currently preferred model. Will be undefined if their chosen model provider doesn't have a model lookup, or the model is unknown.

window.ai.getCurrentModel(): Promise<ModelID | undefined>

Listen to events: to listen to events emitted by the extension, such as whenever the preferred model changes, here's what you do:

window.ai.addEventListener((event: EventType, data: unknown) => {
  // You can check `event` to see if it's the EventType you care about, e.g. "model_changed"
  console.log("EVENT received", event, data)
})

(BETA) Generate 3D Objects: Uses Shap-e.

window.ai.BETA_generate3DObject(
    input: PromptInput,
    options?: ThreeDOptions
): Promise<MediaOutput[]>

The BETA_generate3DObject function allows you to generate 3D objects with a defined model and options. The input should be a PromptInput { prompt : string }. The options parameter is optional and accepts ThreeDOptions customize the media generation request.

Here's an example request:

const [ result ] = await window.ai.BETA_generate3DObject(
  { "prompt": "a glazed donut" }, 
  { "numInferenceSteps": 32,});

// base64 representation of your 3D object, in ply format
const uri = result.uri;

All public types, including error messages, are documented in the window.ai library. Highlights below:

CompletionOptions

This options dictionary allows you to specify options for the completion request.

export interface CompletionOptions {
  // If specified, partial updates will be streamed to this handler as they become available,
  // and only the first partial update will be returned by the Promise.
  // This only works if 1) the chosen model supports streaming and
  // 2) `numOutputs` below is not > 1. Otherwise, it will be ignored, and the
  // whole result will be in the promise's resolution
  onStreamResult?: (result: Output | null, error: string | null) => unknown

  // What sampling temperature to use, between 0 and 2. Higher values like 0.8 will
  // make the output more random, while lower values like 0.2 will make it more focused and deterministic.
  // Different models have different defaults.
  temperature?: number

  // How many completion choices to generate. Defaults to 1.
  numOutputs?: number

  // The maximum number of tokens to generate in the chat completion. Defaults to infinity, but the
  // total length of input tokens and generated tokens is limited by the model's context length.
  maxTokens?: number

  // Sequences where the API will stop generating further tokens.
  stopSequences?: string[]

  // Identifier of the model to use. Defaults to the user's current model, but can be overridden here.
  // Arbitrary strings are allowed, and will be passed to the Local model as `model`.
  // NOTE: this standard is evolving - recommend not using this if you're making an immutable app.
  model?: ModelID | string
}

ThreeDOptions

This options dictionary allows you to specify options for generating a three dimensional object.

export interface ThreeDOptions{
  // The number of inference steps to run. Defaults to 32, with specific default values for each model.
  numInferenceSteps?: number
  // How many generations to create. Defaults to 1.
  numOutputs?: number
  // Identifier of the model to use. Defaults to openai/shap-e for now.
  model?: ModelID | string
}

Model ID Standard

ModelID is an enum of the available models, which are available as a TypeScript enum inside window.ai. See the library's README.

Error codes

Errors emitted by the extension API:

export enum ErrorCode {
  // Incorrect API key / auth
  NotAuthenticated = "NOT_AUTHENTICATED",

  // User denied permission to the app
  PermissionDenied = "PERMISSION_DENIED",

  // Happens when a permission request popup times out
  RequestNotFound = "REQUEST_NOT_FOUND",

  // When a request is badly formed
  InvalidRequest = "INVALID_REQUEST",

  // When an AI model refuses to fulfill a request. The returned error is
  // prefixed by this value and includes the status code that the model API returned
  ModelRejectedRequest = "MODEL_REJECTED_REQUEST"
}

Community tools

Hope to eventually make an awesome-window.ai repo, but in the meantime:

🧠 Local model setup

You can configure any local model to work with Window-compatible apps by writing a simple HTTP server.

To quickly set up a local LLM server for experimentation, you can download local.ai, which includes a GUI to download models and configure the streaming server:

Server API Spec

Types

POST /completions

Generate text to complete a prompt or list of messages. This endpoint accepts a request body containing the following parameters:

Note: apps like windowai.io will ask to stream, so your local server might not work with them until you support streaming.

Return value:

This endpoint should return an object that looks like:

{
  choices: Array<{ text: string }>
}

POST /model

Get the model that will be used for the given prompt and completion options This endpoint accepts a request body containing the same parameters as the /completions endpoint above.

Return value:

This endpoint should return an object that looks like:

{
  id: string
}

Where id is a string identifying the model, such as a known ModelID.

More WIP thinking here.

Demo comparing Alpaca with GPT-4

Demo context

https://user-images.githubusercontent.com/1011391/230620781-57b8ffdb-4081-488c-b059-0daca5806b5a.mp4

🤝 Contributing

This is a turborepo monorepo containing:

  1. A Plasmo extension project.
  2. A web app serving windowai.io.
  3. Upcoming packages to help developers (see Discord for more info).

To run the extension and the web app in parallel:

pnpm dev

To build them both:

pnpm build

After building, open your browser and load the appropriate development build by loading an unpacked extension. For example, if you are developing for the Chrome browser, using manifest v3, use: build/chrome-mv3-dev.