why is ModelRunneCpp await_responses blocked?

[GooVincent]

System Info

I wanna cancel the request in some case and the cancel_request need to pass the request id, then I call await_responses to obtain it. following is is my code.

what I am using is TensorRT-LLM version: 0.12.0.

    outputs = runner.generate(..., streaming=true..)
    for curr_outputs in outputs:
        print(f'step 1')
        responses = runner.session.await_responses()
        print(f'step 2')
        for response in responses:
            runner.session.cancel_request(response.request_id)
            print(f'step 3')
            runner.session.await_responses()
            print(f'step 4')

I try to test it in several times, sometime it blocked after print 'step 1'? I am confused why this happen. Please anyone could help me?

Who can help?

No response

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

run above code

Expected behavior

no

actual behavior

no

additional notes

no

[lfr-0531]

We recommend using ModelRunnerCpp in the same way of run.py.

I try to test it in several times, sometime it blocked after print 'step 1'? I am confused why this happen.

This seems that there is no active request after canceling the request, then no response.

[GooVincent]

We recommend using ModelRunnerCpp in the same way of run.py.

I try to test it in several times, sometime it blocked after print 'step 1'? I am confused why this happen.

This seems that there is no active request after canceling the request, then no response.

Then how to know if there is activate request? It will block the entire process, I can't stop it even by kill.

[lfr-0531]

Then how to know if there is activate request?

stats = runner.session.get_latest_iteration_stats()
for stat in stats:
    print(stat.to_json_str())

If you want to cancel a request, please don't call runner.session.await_responses() out of the runner.generate. You can get the request_ids in runner.generate(link).

for curr_outputs in throttle_generator(outputs,
                                       args.streaming_interval):
    stats = runner.session.get_latest_iteration_stats()
    for stat in stats:
        print(stat.to_json_str())
    runner.session.cancel_request(request_ids[0])

[GooVincent]

Then how to know if there is activate request?

stats = runner.session.get_latest_iteration_stats() for stat in stats: print(stat.to_json_str()) If you want to cancel a request, please don't call runner.session.await_responses() out of the runner.generate. You can get the request_ids in runner.generate(link).

for curr_outputs in throttle_generator(outputs, args.streaming_interval): stats = runner.session.get_latest_iteration_stats() for stat in stats: print(stat.to_json_str()) runner.session.cancel_request(request_ids[0])

the way to obtain the request id showing in the generate, it's also running inner runner.generate. still confused how to fetch the request id out of the runner.generate.

    request_ids = self.session.enqueue_requests(requests)

[lfr-0531]

You can hack the generate in model_runner_cpp.py to add request_ids to the output.

But actually we don't recommend users using cancel_request when using model_runner_cpp.py.

[GooVincent]

why? what's the purpose for cancel_request.

[lfr-0531]

model_runner_cpp.py is used to run some simple examples. Of course, we welcome users to make some changes to use it, including cancel_request.

[GooVincent]

then how about ModelRunner, which is py session. is it recommended to production?

[lfr-0531]

which is py session.

py session is https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/runtime/generation.py.

is it recommended to production?

It depends on your needs. Now we recommend to deploy with Triton Inference Server for production, please refer to https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html#deploy-with-triton-inference-server.

And we also provide a Python API, please refer to https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html#llm-api and https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/apps

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."