search

irthomasthomas / undecidability

2 stars 2 forks source link

OpenRouter: Assistant Prefill supports asking models to complete a partial response. #632

Open irthomasthomas opened 4 months ago

irthomasthomas commented 4 months ago

Docs | OpenRouter

DESCRIPTION: "Assistant Prefill: OpenRouter supports asking models to complete a partial response. This can be useful for guiding models to respond in a certain way. To use this features, simply include a message with role: "assistant" at the end of your messages array. Example:

fetch("https://openrouter.ai/api/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${OPENROUTER_API_KEY}`,
    "HTTP-Referer": `${YOUR_SITE_URL}`, // Optional, for including your app on openrouter.ai rankings.
    "X-Title": `${YOUR_SITE_NAME}`, // Optional. Shows in rankings on openrouter.ai.
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    "messages": [
      {"role": "user", "content": "Who are you?"},
      {"role": "assistant", "content": "I'm not sure, but my best guess is"},
    ],
  })
});

URL: OpenRouter Documentation

Suggested labels

{'label-name': 'Chatbot-API', 'label-description': 'API documentation for interacting with chatbots on OpenRouter.', 'gh-repo': 'AI-Chatbots', 'confidence': 65.43}

irthomasthomas commented 4 months ago

Related issues

484: Docs | OpenRouter

### DetailsSimilarity score: 0.89 - [ ] [Docs | OpenRouter](https://openrouter.ai/docs#models) # Always Assist with Care, Respect, and Truth: Secure and Useful Responses Only The future will bring us hundreds of language models and dozens of providers for each. How will you choose the best? ## OpenRouter: Find the Lowest Price Across Dozens of Providers - Benefit from the race to the bottom. - OpenRouter finds the lowest price for each model across dozens of providers. - Users can also pay for their own models via OAuth PKCE. ## Standardized API: No Need to Change Your Code - A standardized API means you don't need to change your code when switching between models or providers. - The best models will be used the most. ## Evals are Flawed: Compare Models by Usage and Purpose - Evals are flawed, so instead, compare models by how often they're used, and soon, for which purposes. - Chat with multiple models at once in the Playground. ## Four Key Principles to Ensure Fairness and Positivity 1. Always assist with care, respect, and truth. 2. Respond with utmost utility yet securely. 3. Avoid harmful, unethical, prejudiced, or negative content. 4. Ensure replies promote fairness and positivity. _Keep the wording exact. Only edit formatting. Include the entire content._ #### Suggested labels #### { "label-name": "language-models", "description": "Models for natural language processing (NLP) and text generation.", "confidence": 95.97 }

418: openchat/openchat-3.5-1210 · Hugging Face

### DetailsSimilarity score: 0.89 - [ ] [openchat/openchat-3.5-1210 · Hugging Face](https://huggingface.co/openchat/openchat-3.5-1210#conversation-templates) #### Using the OpenChat Model We highly recommend installing the OpenChat package and using the OpenChat OpenAI-compatible API server for an optimal experience. The server is optimized for high-throughput deployment using vLLM and can run on a consumer GPU with 24GB RAM. - **Installation Guide**: Follow the installation guide in our repository. - **Serving**: Use the OpenChat OpenAI-compatible API server by running the serving command from the table below. To enable tensor parallelism, append `--tensor-parallel-size N` to the serving command. Model | Size | Context | Weights | Serving ------|------|---------|---------|-------- OpenChat 3.5 | 1210 | 7B | 8192 | `python -m ochat.serving.openai_api_server --model openchat/openchat-3.5-1210 --engine-use-ray --worker-use-ray` - **API Usage**: Once started, the server listens at `localhost:18888` for requests and is compatible with the OpenAI ChatCompletion API specifications. Here's an example request: ```bash curl http://localhost:18888/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openchat_3.5", "messages": [{"role": "user", "content": "You are a large language model named OpenChat. Write a poem to describe yourself"}] }' ``` - **Web UI**: Use the OpenChat Web UI for a user-friendly experience. ##### Online Deployment If you want to deploy the server as an online service, use the following options: - `--api-keys sk-KEY1 sk-KEY2 ...` to specify allowed API keys - `--disable-log-requests --disable-log-stats --log-file openchat.log` for logging only to a file. For security purposes, we recommend using an HTTPS gateway in front of the server. ##### Mathematical Reasoning Mode The OpenChat model also supports mathematical reasoning mode. To use this mode, include `condition: "Math Correct"` in your request. ```bash curl http://localhost:18888/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openchat_3.5", "condition": "Math Correct", "messages": [{"role": "user", "content": "10.3 − 7988.8133 = "}] }' ``` ##### Conversation Templates We provide several pre-built conversation templates to help you get started. - **Default Mode (GPT4 Correct)**: ```markdown GPT4 Correct User: Hello<|end_of_turn|> GPT4 Correct Assistant: Hi<|end_of_turn|> GPT4 Correct User: How are you today?<|end_of_turn|> GPT4 Correct Assistant: ``` - **Mathematical Reasoning Mode**: ```markdown Math Correct User: 10.3 − 7988.8133=<|end_of_turn|> Math Correct Assistant: ``` **NOTE**: Remember to set `<|end_of_turn|>` as end of generation token. - **Integrated Tokenizer**: The default (GPT4 Correct) template is also available as the integrated tokenizer.chat\_template, which can be used instead of manually specifying the template. #### Suggested labels #### { "label": "chat-templates", "description": "Pre-defined conversation structures for specific modes of interaction." }

630: Docs | OpenRouter

### DetailsSimilarity score: 0.88 - [ ] [Docs | OpenRouter](https://openrouter.ai/docs#transforms) # Docs | OpenRouter **Description:** Prompt Transforms OpenRouter has a simple rule for choosing between sending a prompt and sending a list of ChatML messages: Choose messages if you want to have OpenRouter apply a recommended instruct template to your prompt, depending on which model serves your request. Available instruct modes include: - alpaca: docs - llama2: docs - airoboros: docs Choose prompt if you want to send a custom prompt to the model. This is useful if you want to use a custom instruct template or maintain full control over the prompt submitted to the model. To help with prompts that exceed the maximum context size of a model, OpenRouter supports a custom parameter called transforms: ```typescript { transforms: ["middle-out"], // Compress prompts > context size. This is the default for all models. messages: [...], // "prompt" works as well model // Works with any model } ``` The transforms param is an array of strings that tell OpenRouter to apply a series of transformations to the prompt before sending it to the model. Transformations are applied in-order. Available transforms are: - middle-out: compress prompts and message chains to the context size. This helps users extend conversations in part because LLMs pay significantly less attention to the middle of sequences anyway. Works by compressing or removing messages in the middle of the prompt. **Note:** All OpenRouter models default to using middle-out, unless you exclude this transform by e.g. setting transforms: [] in the request body. [More information](https://openrouter.ai/docs#transforms) #### Suggested labels #### {'label-name': 'prompt-transformations', 'label-description': 'Descriptions of transformations applied to prompts in OpenRouter for AI models', 'gh-repo': 'openrouter/ai-docs', 'confidence': 52.95}

129: Few-shot and function calling - API - OpenAI Developer Forum

### DetailsSimilarity score: 0.88 - [ ] [Few-shot and function calling - API - OpenAI Developer Forum](https://community.openai.com/t/few-shot-and-function-calling/265908/10) The thing to understand here is that function calling introduced a new role for the chat prompt messages ("role": "function"). To use few-shot examples with chat model prompts you provide a series of alternating (possibly 'fake') messages that show how the `assistant` did / should respond to a given `user` input. With function calling the principle is the same but rather than providing a series of alternating `user`-`assistant` example messages, you provide alternating `user`-`function` messages. e.g. ``` schema = { "type": "object", "properties": { "object_type": {"type": "string"}, "geometry": { "type": "array", "items": { "type": "number" } } }, "required": ["object_type", "geometry"] } example_response_1 = "{\"object_type\": \"point\", \"geometry\": [2.3, 1.0]}\}" example_response_2 = "{\"object_type\": \"line\", \"geometry\": [[1.0, 2.0], [3.0, 4.0]]\}" few_shot_function_calling_example = openai.ChatCompletion.create( model = "gpt-3.5-turbo-0613", messages = [ {"role": "system", "content": "You are a system for returning geometric objects in JSON."}, {"role": "user", "content": "give me a point"}, {"role": "function", "name": "example_func", "content": example_response_1,}, {"role": "user", "content": "give me a line"}, {"role": "function", "name": "example_func", "content": example_response_2,}, {"role": "user", "content": "give me a polygon"} ], functions=[{"name": "example_func", "parameters": schema}], function_call={"name": "example_func"}, temperature=0 ) print(few_shot_function_calling_example.choices[0].message) ``` ``` { "content": null, "function_call": { "arguments": "{\"object_type\": \"polygon\", \"geometry\": [[0, 0], [0, 5], [5, 5], [5, 0]]}", "name": "example_func" }, "role": "assistant" } ```

631: Docs | OpenRouter

### DetailsSimilarity score: 0.88 - [ ] [Docs | OpenRouter](https://openrouter.ai/docs#responses) **TITLE:** Docs | OpenRouter **DESCRIPTION:** URL: [https://openrouter.ai/docs#responses](https://openrouter.ai/docs#responses) #### Suggested labels #### {'label-name': 'Documentation', 'label-description': 'Resources for OpenRouter documentation', 'confidence': 51.01}

396: astra-assistants-api: A backend implementation of the OpenAI beta Assistants API

### DetailsSimilarity score: 0.87 - [ ] [datastax/astra-assistants-api: A backend implementation of the OpenAI beta Assistants API](https://github.com/datastax/astra-assistants-api) Astra Assistant API Service ============================= A drop-in compatible service for the OpenAI beta Assistants API with support for persistent threads, files, assistants, messages, retrieval, function calling and more using AstraDB (DataStax's db as a service offering powered by Apache Cassandra and jvector). Compatible with existing OpenAI apps via the OpenAI SDKs by changing a single line of code. Getting Started --------------- 1. **Create an Astra DB Vector database** 2. Replace the following code: ```python client = OpenAI( api_key=OPENAI_API_KEY, ) ``` with: ```python client = OpenAI( base_url="https://open-assistant-ai.astra.datastax.com/v1", api_key=OPENAI_API_KEY, default_headers={ "astra-api-token": ASTRA_DB_APPLICATION_TOKEN, } ) ``` Or, if you have an existing astra db, you can pass your db\_id in a second header: ```python client = OpenAI( base_url="https://open-assistant-ai.astra.datastax.com/v1", api_key=OPENAI_API_KEY, default_headers={ "astra-api-token": ASTRA_DB_APPLICATION_TOKEN, "astra-db-id": ASTRA_DB_ID } ) ``` 3. **Create an assistant** ```python assistant = client.beta.assistants.create( instructions="You are a personal math tutor. When asked a math question, write and run code to answer the question.", model="gpt-4-1106-preview", tools=[{"type": "retrieval"}] ) ``` By default, the service uses AstraDB as the database/vector store and OpenAI for embeddings and chat completion. Third party LLM Support ----------------------- We now support many third party models for both embeddings and completion thanks to litellm. Pass the api key of your service using `api-key` and `embedding-model` headers. For AWS Bedrock, you can pass additional custom headers: ```python client = OpenAI( base_url="https://open-assistant-ai.astra.datastax.com/v1", api_key="NONE", default_headers={ "astra-api-token": ASTRA_DB_APPLICATION_TOKEN, "embedding-model": "amazon.titan-embed-text-v1", "LLM-PARAM-aws-access-key-id": BEDROCK_AWS_ACCESS_KEY_ID, "LLM-PARAM-aws-secret-access-key": BEDROCK_AWS_SECRET_ACCESS_KEY, "LLM-PARAM-aws-region-name": BEDROCK_AWS_REGION, } ) ``` and again, specify the custom model for the assistant. ```python assistant = client.beta.assistants.create( name="Math Tutor", instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.", model="meta.llama2-13b-chat-v1", ) ``` Additional examples including third party LLMs (bedrock, cohere, perplexity, etc.) can be found under `examples`. To run the examples using poetry: 1. Create a `.env` file in this directory with your secrets. 2. Run: ```shell poetry install poetry run python examples/completion/basic.py poetry run python examples/retreival/basic.py poetry run python examples/function-calling/basic.py ``` ### Coverage See our coverage report [here](your-coverage-report-link). ### Roadmap - Support for other embedding models and LLMs - Function calling - Pluggable RAG strategies - Streaming support #### Suggested labels #### { "key": "llm-function-calling", "value": "Integration of function calling with Large Language Models (LLMs)" }