Kuadrant / architecture

Architecture Documents
0 stars 10 forks source link

sa-east-1 (Sao Paulo) -- > elasticache eu-central-1 (Frankfurt) ~200ms 3 limits one 1 with low time periods #88

Closed eguzki closed 5 months ago

eguzki commented 5 months ago

Test setup

Elasticache <- EC2 Redis Proxy (socat)  < ---------------- INET --------------- EC2 Limitador - Driver

Redis proxy

1 EC2 instance

amazon linux 2023 AMI

type: c7i.large 2vCPU 4Gib Memory

sudo yum update -y
sudo yum install -y socat
socat TCP-LISTEN:9876,fork,reuseaddr TCP:${ELASTICACHE_PRIMARY_ENDPOINT}

Elasticache

Limtador and Driver

1 EC2 instance

amazon linux 2023 AMI

type: c7i xlarge 4vCPU 8Gib Memory

sudo yum update -y
sudo yum install -y docker
sudo service docker start
sudo usermod -a -G docker ec2-user
# (logout and login again)
docker ps

Create docker image with limitador and limitador-driver

cat <<'EOF' >Dockerfile
ARG LIMITADOR_IMAGE

FROM $LIMITADOR_IMAGE as limitador

# Stage 1: Build limitador-driver
FROM rust:1.76-bullseye as builder

ARG LIMITADOR_DRIVER_REPO=https://github.com/alexsnaps/limitador-driver.git
ARG LIMITADOR_DRIVER_BRANCH=main

RUN apt update && apt upgrade -y \
    && apt install -y protobuf-compiler git

RUN git clone --depth 1 $LIMITADOR_DRIVER_REPO -b $LIMITADOR_DRIVER_BRANCH /opt/limitador-driver

WORKDIR /opt/limitador-driver

RUN cargo build --release

# Stage 2: Setup the environment to run limitador-driver and limitador
FROM rust:1.76-bullseye

# Copy the Rust binary from the builder stage
COPY --from=builder /opt/limitador-driver/target/release/limitador-driver /usr/local/bin

# Copy the limitador service into the container
COPY --from=limitador /home/limitador/bin/limitador-server /usr/local/bin

# Run the script
CMD "/bin/bash"
EOF

Build image based on some limitador image, for example quay.io/kuadrant/limitador:write-behind-lock

docker build --build-arg="LIMITADOR_IMAGE=quay.io/kuadrant/limitador:write-behind-lock" -t write-behind-lock .

Limits file

cat <<EOF >limits.yaml
---
- namespace: test_namespace
  max_value: 10000000
  seconds: 3600
  conditions:
  variables:
- namespace: test_namespace
  max_value: 10000000
  seconds: 1800
  conditions:
  variables:
- namespace: test_namespace
  max_value: 100000
  seconds: 5
  conditions:
  variables:
EOF

Start container

docker run --name limitador-driver --rm -ti -v $PWD/limits.yaml:/tmp/limits.yaml write-behind-lock

Start limitador

/usr/local/bin/limitador-server -vv /tmp/limits.yaml redis_cached --ratio 1 redis://$REDIS_PROXY:9876
Limitador Server v1.4.0-dev (83fe4e51) [] release
2024-03-22T16:13:35.469516Z  INFO limitador_server: Version: v1.4.0-dev (83fe4e51) [] release    
2024-03-22T16:13:35.469537Z  INFO limitador_server: Using config: Configuration { limits_file: "/tmp/limits.yaml", storage: Redis(RedisStorageConfiguration { url: "redis://ec2-54-93-63-180.eu-central-1.compute.amazonaws.com:9876", cache: Some(RedisStorageCacheConfiguration { flushing_period: 1000, max_ttl: 5000, ttl_ratio: 1, max_counters: 10000 }) }), rls_host: "0.0.0.0", rls_port: 8081, http_host: "0.0.0.0", http_port: 8080, limit_name_in_labels: false, tracing_endpoint: "", log_level: Some(LevelFilter::INFO), rate_limit_headers: None, grpc_reflection_service: false }    
2024-03-22T16:13:35.672703Z  INFO limitador_server: limits file path: /tmp/limits.yaml    
2024-03-22T16:13:35.673178Z  INFO limitador_server: Envoy RLS server starting on 0.0.0.0:8081    
2024-03-22T16:13:35.673190Z  INFO limitador_server: HTTP server starting on 0.0.0.0:8080    
2024-03-22T16:13:35.673396Z  INFO actix_server::builder: starting 4 workers
2024-03-22T16:13:35.673404Z  INFO actix_server::server: Actix runtime found; starting in Actix runtime

Open a new shell the ec2 instance

docker exec -ti limitador-driver /bin/bash

Run the driver to start the test

/usr/local/bin/limitador-driver rpc://127.0.0.1:8081 1m

Results

root@3c37cb816fd1:/# /usr/local/bin/limitador-driver rpc://127.0.0.1:8081 1m
Hitting Limitador at rpc://127.0.0.1:8081 for a total duration of 1 minutes
Current: hits: 9725, mean: 0.102ms, p50: 0.078ms, p90: 0.091ms, p99: 0.117ms, p999: 0.369ms, max: 201.727ms
Current: hits: 12392, mean: 0.080ms, p50: 0.077ms, p90: 0.089ms, p99: 0.112ms, p999: 0.277ms, max: 0.405ms
Current: hits: 12288, mean: 0.081ms, p50: 0.077ms, p90: 0.090ms, p99: 0.118ms, p999: 0.333ms, max: 0.555ms
Current: hits: 12304, mean: 0.081ms, p50: 0.077ms, p90: 0.090ms, p99: 0.113ms, p999: 0.293ms, max: 0.419ms
Current: hits: 12256, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.115ms, p999: 0.307ms, max: 0.575ms
Current: hits: 1708, mean: 0.585ms, p50: 0.077ms, p90: 0.092ms, p99: 0.353ms, p999: 323.583ms, max: 329.727ms
Current: hits: 12144, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.118ms, p999: 0.297ms, max: 0.403ms
Current: hits: 12195, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.117ms, p999: 0.291ms, max: 0.415ms
Current: hits: 12263, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.114ms, p999: 0.313ms, max: 0.453ms
Current: hits: 12127, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.117ms, p999: 0.297ms, max: 0.531ms
Current: hits: 1746, mean: 0.573ms, p50: 0.077ms, p90: 0.091ms, p99: 0.227ms, p999: 325.631ms, max: 329.727ms
Current: hits: 7309, mean: 0.180ms, p50: 0.077ms, p90: 0.090ms, p99: 0.118ms, p999: 0.305ms, max: 403.455ms
Current: hits: 12357, mean: 0.080ms, p50: 0.077ms, p90: 0.090ms, p99: 0.113ms, p999: 0.277ms, max: 0.647ms
Current: hits: 12199, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.116ms, p999: 0.299ms, max: 0.475ms
Current: hits: 8146, mean: 0.122ms, p50: 0.079ms, p90: 0.092ms, p99: 0.121ms, p999: 0.313ms, max: 323.583ms
Current: hits: 9742, mean: 0.102ms, p50: 0.078ms, p90: 0.091ms, p99: 0.117ms, p999: 0.315ms, max: 201.727ms
Current: hits: 7292, mean: 0.136ms, p50: 0.078ms, p90: 0.091ms, p99: 0.117ms, p999: 0.339ms, max: 401.407ms
Current: hits: 2432, mean: 0.411ms, p50: 0.078ms, p90: 0.092ms, p99: 0.168ms, p999: 0.555ms, max: 403.455ms
Current: hits: 12160, mean: 0.082ms, p50: 0.078ms, p90: 0.091ms, p99: 0.119ms, p999: 0.299ms, max: 0.435ms
Current: hits: 8411, mean: 0.142ms, p50: 0.077ms, p90: 0.089ms, p99: 0.113ms, p999: 0.277ms, max: 323.583ms
Current: hits: 12216, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.115ms, p999: 0.287ms, max: 0.419ms
Current: hits: 12256, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.114ms, p999: 0.277ms, max: 0.355ms
Current: hits: 2362, mean: 0.423ms, p50: 0.078ms, p90: 0.093ms, p99: 0.139ms, p999: 0.583ms, max: 403.455ms
Current: hits: 12191, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.114ms, p999: 0.303ms, max: 0.473ms
Current: hits: 5710, mean: 0.174ms, p50: 0.078ms, p90: 0.092ms, p99: 0.133ms, p999: 0.443ms, max: 323.583ms
Current: hits: 12025, mean: 0.082ms, p50: 0.079ms, p90: 0.092ms, p99: 0.119ms, p999: 0.311ms, max: 0.603ms
Current: hits: 12234, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.118ms, p999: 0.275ms, max: 0.329ms
Current: hits: 9756, mean: 0.102ms, p50: 0.077ms, p90: 0.091ms, p99: 0.117ms, p999: 0.329ms, max: 202.751ms
Current: hits: 7256, mean: 0.137ms, p50: 0.078ms, p90: 0.092ms, p99: 0.123ms, p999: 0.357ms, max: 399.359ms
Current: hits: 5695, mean: 0.175ms, p50: 0.079ms, p90: 0.093ms, p99: 0.124ms, p999: 0.401ms, max: 323.583ms
Current: hits: 12129, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.117ms, p999: 0.303ms, max: 0.711ms
Current: hits: 12245, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.116ms, p999: 0.287ms, max: 0.403ms
Current: hits: 12370, mean: 0.080ms, p50: 0.077ms, p90: 0.090ms, p99: 0.112ms, p999: 0.255ms, max: 0.315ms
Current: hits: 7314, mean: 0.136ms, p50: 0.078ms, p90: 0.091ms, p99: 0.125ms, p999: 0.467ms, max: 397.311ms
Current: hits: 2385, mean: 0.586ms, p50: 0.078ms, p90: 0.092ms, p99: 0.142ms, p999: 397.311ms, max: 403.455ms
Current: hits: 12251, mean: 0.081ms, p50: 0.077ms, p90: 0.090ms, p99: 0.116ms, p999: 0.327ms, max: 0.469ms
Current: hits: 12207, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.119ms, p999: 0.293ms, max: 0.509ms
Current: hits: 12196, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.118ms, p999: 0.315ms, max: 0.551ms
Current: hits: 12258, mean: 0.113ms, p50: 0.077ms, p90: 0.090ms, p99: 0.116ms, p999: 0.279ms, max: 403.455ms
Current: hits: 4768, mean: 0.209ms, p50: 0.078ms, p90: 0.093ms, p99: 0.126ms, p999: 0.333ms, max: 403.455ms
Current: hits: 12160, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.117ms, p999: 0.301ms, max: 0.487ms
Current: hits: 12203, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.118ms, p999: 0.279ms, max: 0.403ms
Current: hits: 12197, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.117ms, p999: 0.289ms, max: 0.367ms
Current: hits: 12032, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.123ms, p999: 0.367ms, max: 0.571ms
Current: hits: 4823, mean: 0.207ms, p50: 0.077ms, p90: 0.090ms, p99: 0.127ms, p999: 0.427ms, max: 403.455ms
Current: hits: 12262, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.114ms, p999: 0.289ms, max: 0.401ms
Current: hits: 12102, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.120ms, p999: 0.289ms, max: 0.353ms
Current: hits: 12113, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.124ms, p999: 0.363ms, max: 0.687ms
Current: hits: 12096, mean: 0.082ms, p50: 0.078ms, p90: 0.091ms, p99: 0.127ms, p999: 0.385ms, max: 0.699ms
Current: hits: 4, mean: 251.565ms, p50: 201.727ms, p90: 403.455ms, p99: 403.455ms, p999: 403.455ms, max: 403.455ms
Current: hits: 12120, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.124ms, p999: 0.343ms, max: 0.627ms
Current: hits: 12180, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.121ms, p999: 0.291ms, max: 0.375ms
Current: hits: 12238, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.119ms, p999: 0.289ms, max: 0.559ms
Current: hits: 12239, mean: 0.097ms, p50: 0.077ms, p90: 0.090ms, p99: 0.114ms, p999: 0.315ms, max: 201.727ms
Current: hits: 6, mean: 201.016ms, p50: 0.389ms, p90: 403.455ms, p99: 403.455ms, p999: 403.455ms, max: 403.455ms
Current: hits: 12246, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.114ms, p999: 0.297ms, max: 0.409ms
Current: hits: 12046, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.124ms, p999: 0.323ms, max: 0.547ms
Current: hits: 12041, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.122ms, p999: 0.309ms, max: 0.507ms

 ============== Done! =================

Ok: hits: 573377, mean: 0.104ms, p50: 0.078ms, p90: 0.091ms, p99: 0.118ms, p999: 0.315ms, max: 403.455ms
OverLimit: hits: 0, mean: 0.000ms, p50: 0.000ms, p90: 0.000ms, p99: 0.000ms, p999: 0.000ms, max: 0.000ms
Unknown: hits: 0, mean: 0.000ms, p50: 0.000ms, p90: 0.000ms, p99: 0.000ms, p999: 0.000ms, max: 0.000ms
Overall: hits: 573377, mean: 0.104ms, p50: 0.078ms, p90: 0.091ms, p99: 0.118ms, p999: 0.315ms, max: 403.455ms

0.01ms |                                                                                  |  0.0th %-ile
0.02ms |                                                                                  |  0.0th %-ile
0.03ms |                                                                                  |  0.0th %-ile
0.04ms |                                                                                  |  0.0th %-ile
0.06ms | *                                                                                |  0.0th %-ile
0.07ms | *                                                                                |  0.8th %-ile
0.08ms | *********************************                                                | 41.2th %-ile
0.09ms | *********************************                                                | 81.8th %-ile
0.10ms | ************                                                                     | 96.1th %-ile
0.11ms | **                                                                               | 98.4th %-ile
0.12ms | *                                                                                | 99.1th %-ile
0.13ms | *                                                                                | 99.4th %-ile
0.14ms | *                                                                                | 99.5th %-ile
0.15ms | *                                                                                | 99.6th %-ile
0.17ms | *                                                                                | 99.6th %-ile
0.18ms | *                                                                                | 99.6th %-ile
0.19ms | *                                                                                | 99.7th %-ile
0.20ms | *                                                                                | 99.7th %-ile
0.21ms | *                                                                                | 99.7th %-ile
0.22ms | *                                                                                | 99.7th %-ile
0.23ms | *                                                                                | 99.7th %-ile
0.24ms | *                                                                                | 99.7th %-ile
0.25ms | *                                                                                | 99.7th %-ile
0.26ms | *                                                                                | 99.8th %-ile
0.28ms | *                                                                                | 99.8th %-ile
0.29ms | *                                                                                | 99.8th %-ile
0.30ms | *                                                                                | 99.9th %-ile
0.31ms | *                                                                                | 99.9th %-ile
0.32ms | *                                                                                | 99.9th %-ile