Open guillermoayllon opened 1 month ago
Converting the tensor to host memory before inference fixes the issue, but throughput is significantly reduced. To the point where the throughput is larger with the non-optimized model.
I have the same issue. Are there any examples available on how to integrate intel gpu with openvino code? Appart from the device
argument in the core.compile
function? Also I saw the compress_to_FP16
argument in the ov.convert_model
function is deprecated. Is there an equivalent updated compression option?
@guillermoayllon could you share
Hello @Iffa-Intel, thank you for looking into the issue.
The model we are using is TorchVision's VGG16: https://pytorch-org.translate.goog/vision/main/models/generated/torchvision.models.vgg16.html?_x_tr_sl=en&_x_tr_tl=es&_x_tr_hl=es&_x_tr_pto=sc
One should be able to reproduce the issue by importing the model and using the default weights as indicated in the first code block above:
import torch
**from torchvision import models**
import openvino as ov
import intel_extension_for_pytorch
**model = models.vgg16(weights="VGG16_Weights.DEFAULT")**
model.eval()
ov_model = ov.convert_model(model, input=[128, 3, 224, 224])
@guillermoayllon to get that VGG16 model (as you mentioned), intel_extension_for_pytorch is required. May I know which version did you use? I noticed that only v1.10.200+gpu supports Ubuntu 20.04. If you are using other versions, perhaps this is the underlying cause for the throughput issue.
Hello @Iffa-Intel, thank you for pointing that out.
The Ubuntu version above is wrong. The version that we are using is Ubuntu 22.04.4 LTS (not 20.04). The mentioned intel_extension_for_pytorch version is correct : 2.1.30.post0
Nevertheless, the throughput issue appears when converting the tensor to host memory before inference. But that is not ideal. We would like to keep the tensor in GPU memory.
OpenVINO Version
2024.2.0
Operating System
Ubuntu 20.04 (LTS)
Device used for inference
GPU
Framework
PyTorch
Model used
VGG16
Issue description
Hiya,
My goal is to perform inference on Intel GPU with an openvino model. However, inference fails if I move the input data to GPU before performing inference.
Step-by-step reproduction
First, I convert the VGG16 model to openvino IR format in this way:
Then I compile the model:
Finally, I attempt inference in the following way:
To produce the dataloader, I use this function:
Environment:
Device Info: x2 Intel(R) Data Center GPU Max 1100
Relevant log output
Issue submission checklist