[!TIP] A managed version of BricksLLM is also available! It is production ready, and comes with a dashboard to make interacting with BricksLLM easier. Try us out for free today!
BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM:
The easiest way to get started with BricksLLM is through BricksLLM-Docker.
git clone https://github.com/bricks-cloud/BricksLLM-Docker
cd BricksLLM-Docker
docker compose up
You can run this in detach mode use the -d flag: docker compose up -d
curl -X PUT http://localhost:8001/api/provider-settings \
-H "Content-Type: application/json" \
-d '{
"provider":"openai",
"setting": {
"apikey": "YOUR_OPENAI_KEY"
}
}'
Copy the id
from the response.
Use id
from the previous step as settingId
to create a key with a rate limit of 2 req/min and a spend limit of 25 cents.
curl -X PUT http://localhost:8001/api/key-management/keys \
-H "Content-Type: application/json" \
-d '{
"name": "My Secret Key",
"key": "my-secret-key",
"tags": ["mykey"],
"settingIds": ["ID_FROM_STEP_FOUR"],
"rateLimitOverTime": 2,
"rateLimitUnit": "m",
"costLimitInUsd": 0.25
}'
Then, just redirect your requests to us and use OpenAI as you would normally. For example:
curl -X POST http://localhost:8002/api/providers/openai/v1/chat/completions \
-H "Authorization: Bearer my-secret-key" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "system",
"content": "hi"
}
]
}'
Or if you're using an SDK, you could change its baseURL
to point to us. For example:
// OpenAI Node SDK v4
import OpenAI from 'openai';
const openai = new OpenAI({
apiKey: "some-secret-key", // key created earlier
baseURL: "http://localhost:8002/api/providers/openai/v1", // redirect to us
});
For updating to the latest version
docker pull luyuanxin1995/bricksllm:latest
For updating to a particular version
docker pull luyuanxin1995/bricksllm:1.4.0
Name type description default POSTGRESQL_HOSTS
required Hosts for Postgresql DB. Separated by , localhost
POSTGRESQL_DB_NAME
optional Name for Postgresql DB. POSTGRESQL_USERNAME
required Postgresql DB username POSTGRESQL_PASSWORD
required Postgresql DB password POSTGRESQL_SSL_MODE
optional Postgresql SSL mode disable
POSTGRESQL_PORT
optional The port that Postgresql DB runs on 5432
POSTGRESQL_READ_TIME_OUT
optional Timeout for Postgresql read operations 2m
POSTGRESQL_WRITE_TIME_OUT
optional Timeout for Postgresql write operations 5s
REDIS_HOSTS
required Host for Redis. Separated by , localhost
REDIS_PASSWORD
optional Redis Password REDIS_PORT
optional The port that Redis DB runs on 6379
REDIS_READ_TIME_OUT
optional Timeout for Redis read operations 1s
REDIS_WRITE_TIME_OUT
optional Timeout for Redis write operations 500ms
IN_MEMORY_DB_UPDATE_INTERVAL
optional The interval BricksLLM API gateway polls Postgresql DB for latest key configurations 1s
STATS_PROVIDER
optional "datadog" or Host:Port(127.0.0.1:8125) for statsd. PROXY_TIMEOUT
optional Timeout for proxy HTTP requests. 600s
NUMBER_OF_EVENT_MESSAGE_CONSUMERS
optional Number of event message consumers that help handle counting tokens and inserting event into db. 3
AWS_SECRET_ACCESS_KEY
optional It is for PII detection feature. 5s
AWS_ACCESS_KEY_ID
optional It is for using PII detection feature. 5s
AMAZON_REGION
optional Region for AWS. us-west-2
AMAZON_REQUEST_TIMEOUT
optional Timeout for amazon requests. 5s
AMAZON_CONNECTION_TIMEOUT
optional Timeout for amazon connection. 10s
ADMIN_PASS
optional Simple password for the admin server.