CUDA Graphs support for Tensorflow Serving

tensorflow / serving

A flexible, high-performance serving system for machine learning models

https://www.tensorflow.org/serving

Apache License 2.0

6.18k stars 2.19k forks source link

CUDA Graphs support for Tensorflow Serving #2216

Open ndeep27 opened 6 months ago

ndeep27 commented 6 months ago

Does TF Serving support CUDA graphs?

ndeep27 commented 6 months ago

@singhniraj08 Can you please help with above query? I want to see if there is a way to convert a TF graph to CUDA graph for serving optimization

singhniraj08 commented 6 months ago

@ndeep27, I don't think Tensorflow serving supports CUDA graphs. We will keep this as a feature request and and I will pass this feature to owners internally for implementation. Thank you!