Serve Scalability Benchmark for RLlib Models

Microbenchmark for batching

Run batch_size_benchmark.py. linear_scaling_us is the hypothetical number for single_query_latency * batch_size.

Scalability Benchmark

• Use app_config.json to configure the runtime environment. Note that we use wrk for HTTP benchmark.
• Use compute_config.json to configure nodes. Make sure to change the cloud id.
• Use app.py to deploy initial app.
• Use NUM_REPLICAS=<int> python scale.py to dynamically scale the deployment.
• Use benchmark_client.py to benchmark each http proxy to see the individual and aggregated throughput.
• For deployment scenario, you will need to add a load balancer in front. See register_elb.sh for the shell command to register all instances within an anyscale session to load balancer. You can also use the benchmark_client.py, but make sure to point it to the load balancer address.
• Notice we we start one HTTP proxy per Ray node, this is so that we can horizontally scale out I/O bottleneck

Latency Benchmark

• Ray Serve deployment at app.py: ray start --head; python app.py
  • Use wrk --latency localhost:8000/predict/ to benchmark
• Flask deployment at flask_app.py: FLASK_APP=flask_app:app flask run
  • Use wrk --latency localhost:5000/predict/ to benchmark