how to compare the performance with vllm/tgit/lightllm or other llm serving framework?

[irasin]

Great works! I was wondering do you have some online or offline inference api that can be used to test the throughput/latency performance and will the performance be superior compared with vllm/tgit/lightllm, etc.?

Hope to get your answer, thanks

[irasin]

BTW, could you add some docs or example codes about how to deploy a model with minimal code in Atom. For example, here is the sample code used in vllm, refer to https://github.com/vllm-project/vllm/blob/main/examples/offline_inference.py

from vllm import LLM, SamplingParams

# Sample prompts.
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

# Create an LLM.
llm = LLM(model="facebook/opt-125m")
# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

[happierpig]

Hi @irasin ,

Thanks for your great questions! Here are some of my thoughts and please stay tuned.

1. Atom is a quantization approach, which is a piece of inference optimization techniques and is orthonormal to a "serving framework". Atom can be evaluated on different frameworks (vLLM, lightLLM, Punica, etc.) compared with different quantization mechanisms. And we arbitrarily chose Punica.
2. This codebase is a minimal version for reproducing the experiment results in our paper. The algorithm and kernels are coupled with and manually integrated into Punica's codebase. Currently, it doesn't provide an easy-to-use API to deploy Atom. We plan to do this later.

[irasin]

Thanks a lot, I'm looking forward to use your deploying API ASAP.