Is there a way to try speedster in docker container?

Ciao Diego,

I have tried your solution in several environments but it seems it is hard to keep correctly all version of packages.

I finally could run your notebook for yolov5 in Google Colab but I can't see any improvement using my method to measure performance, which is using the following code:

import numpy as np

dummy_input = torch.randn(1, 3, 640, 640, dtype=torch.float).to(device)

# INIT LOGGERS
starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
repetitions = 100
no_data_write_timings=np.zeros((repetitions,1))

#GPU-WARM-UP
for _ in range(10):
    _ = model(dummy_input)
# MEASURE PERFORMANCE WITHOUT DATA TRANSFERS
with torch.no_grad():
    for rep in range(repetitions):
        starter.record()
        #dummy_input_on_device = dummy_input.to(device)
        outputs = model(dummy_input)
        ender.record()
        # WAIT FOR GPU SYNC
        torch.cuda.synchronize()
        curr_time = starter.elapsed_time(ender)
        no_data_write_timings[rep] = curr_time

mean_no_data_write_syn = np.sum(no_data_write_timings) / repetitions
std_no_data_write_syn = np.std(no_data_write_timings)

print('Optimized model results WITHOUT data transfers:')
print('The Optimized model mean batch inference time is:' +  str(mean_no_data_write_syn))
print('The Optimized model std batch inference time is:' +  str(std_no_data_write_syn))

And unfortunately it has been really hard to run it locally and I still am getting several errors, for example while I try to install your library I get:

2023-01-12 14:00:14 | WARNING  | Unable to install tensor_rt on this platform. The compiler will be skipped. 
2023-01-12 14:00:14 | INFO     | Trying to install deepsparse on the platform...

When I try to run the way your measure performance:

times = []
for _ in range(100):
    st = time.time()
    results = model("zidane.jpg") #imgs[0] got the same error
    times.append((time.time() - st)*1000)
yolo_optimized_time = sum(times) / len(times)
print(f"Average prediction time: {yolo_optimized_time} ms")

Because I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[30], line 4
      2 for _ in range(100):
      3     st = time.time()
----> 4     results = model("zidane.jpg")
      5     times.append((time.time() - st)*1000)
      6 yolo_optimized_time = sum(times) / len(times)

File ~/.virtualenvs/speedster/lib/python3.8/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.virtualenvs/speedster/lib/python3.8/site-packages/torch/autograd/grad_mode.py:27, in _DecoratorContextManager.__call__.<locals>.decorate_context(*args, **kwargs)
     24 @functools.wraps(func)
     25 def decorate_context(*args, **kwargs):
     26     with self.clone():
---> 27         return func(*args, **kwargs)

File ~/.cache/torch/hub/ultralytics_yolov5_master/models/common.py:705, in AutoShape.forward(self, ims, size, augment, profile)
    702 with amp.autocast(autocast):
    703     # Inference
    704     with dt[1]:
--> 705         y = self.model(x, augment=augment)  # forward
    707     # Post-process
    708     with dt[2]:

File ~/.virtualenvs/speedster/lib/python3.8/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.cache/torch/hub/ultralytics_yolov5_master/models/common.py:515, in DetectMultiBackend.forward(self, im, augment, visualize)
    512     im = im.permute(0, 2, 3, 1)  # torch BCHW to numpy BHWC shape(1,320,192,3)
    514 if self.pt:  # PyTorch
--> 515     y = self.model(im, augment=augment, visualize=visualize) if augment or visualize else self.model(im)
    516 elif self.jit:  # TorchScript
    517     y = self.model(im)

File ~/.virtualenvs/speedster/lib/python3.8/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

Cell In[20], line 9, in OptimizedYolo.forward(self, x, *args, **kwargs)
      7 def forward(self, x, *args, **kwargs):
      8     x = list(self.core(x)) # it's a tuple
----> 9     return self.head(x)

File ~/.virtualenvs/speedster/lib/python3.8/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.cache/torch/hub/ultralytics_yolov5_master/models/yolo.py:59, in Detect.forward(self, x)
     57 z = []  # inference output
     58 for i in range(self.nl):
---> 59     x[i] = self.m[i](x[i])  # conv
     60     bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
     61     x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()

File ~/.virtualenvs/speedster/lib/python3.8/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/.virtualenvs/speedster/lib/python3.8/site-packages/torch/nn/modules/conv.py:463, in Conv2d.forward(self, input)
    462 def forward(self, input: Tensor) -> Tensor:
--> 463     return self._conv_forward(input, self.weight, self.bias)

File ~/.virtualenvs/speedster/lib/python3.8/site-packages/torch/nn/modules/conv.py:459, in Conv2d._conv_forward(self, input, weight, bias)
    455 if self.padding_mode != 'zeros':
    456     return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
    457                     weight, bias, self.stride,
    458                     _pair(0), self.dilation, self.groups)
--> 459 return F.conv2d(input, weight, bias, self.stride,
    460                 self.padding, self.dilation, self.groups)

RuntimeError: Input type (c10::Half) and bias type (float) should be the same

So here we are several things:

I think my method is not the best to measure performance, because I am using the method torch.cuda.synchronize() but in Google Colab the best model is based on TensorRT.
It has been impossible to install the TensorRT-based compiler
Locally, the best model is Torchscript-based and it seems I cannot even run your way to measure performace.

I think the best way is to try your optimization using a container, do you have some container where we can try speedster?

Thanks in advance.

I was finally able to run it locally using the docker container from ultralytics

However I just get and optimization (according to your library) of 1.35x faster:

This optimization is not faster than the normal tensorRT provided by ultralytics: python export.py --include engine --device 0

is there something I am missing?

Hi @hdnh2006,

thank you for the contribution. Happy to assist you and accelerate your model together.

We ran again some tests on Yolov5 in this colab and got the following results:

Original model results WITHOUT data transfers:
- The Original model mean batch inference time is:8.265047960281372
- The Original model std batch inference time is:1.0898744956447648
Optimized model results WITHOUT data transfers:
- The Optimized model mean batch inference time is:4.355607032775879
- The Optimized model std batch inference time is:1.2045244697740418

And regarding your first message

We noticed that as input size you used 640, 640 while we in the optimization use 384x640. We suggest that you always use in optimization data that is very similar as a size distribution to that which will later be used in inference, since optimization comes with the shapes of the input data.
We recommend to set the metric_drop_ths to about 0.05 to allow speedster to test more optimization techniques and further increase the latency speed-up.

About local testing:

It would help us a lot to know the hardware you are running them on (which CPU and GPU) and the operating system. That way we can try to replicate the issue and propose solutions
Can you share with us the snippet where you are using the optimize_model function as well?

Thanks @diegofiori for your cooperation.

This notebook is not comparing the optimized model after applying the following code:

class OptimizedYolo(torch.nn.Module):
    def __init__(self, optimized_core, head_layer):
        super().__init__()
        self.core = optimized_core
        self.head = head_layer

    def forward(self, x, *args, **kwargs):
        x = list(self.core(x)) # it's a tuple
        return self.head(x)

final_core = OptimizedYolo(model_optimized, last_layer)

model.model.model = final_core

Why is like this? Maybe here is my mistake. Because I am applying my code after these lines of code:

import numpy as np

dummy_input = torch.randn(1, 3, 384, 640, dtype=torch.float).to(device)

# INIT LOGGERS
starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
repetitions = 100
no_data_write_timings=np.zeros((repetitions,1))

#GPU-WARM-UP
for _ in range(10):
    _ = model(dummy_input)
# MEASURE PERFORMANCE WITHOUT DATA TRANSFERS
with torch.no_grad():
    for rep in range(repetitions):
        starter.record()
        #dummy_input_on_device = dummy_input.to(device)
        outputs = model(dummy_input)
        ender.record()
        # WAIT FOR GPU SYNC
        torch.cuda.synchronize()
        curr_time = starter.elapsed_time(ender)
        no_data_write_timings[rep] = curr_time

mean_no_data_write_syn = np.sum(no_data_write_timings) / repetitions
std_no_data_write_syn = np.std(no_data_write_timings)

print('Optimized model results WITHOUT data transfers:')
print('The Optimized model mean batch inference time is:' +  str(mean_no_data_write_syn))
print('The Optimized model std batch inference time is:' +  str(std_no_data_write_syn))

Then, what is the final model? I mean, I want to replace this line of code: https://github.com/ultralytics/yolov5/blob/cdd804d39ff84b413bde36a84006f51769b6043b/detect.py#L98

For your optimized model, what should I put to replace it?

Tried locally with RTX2060 and these results are gotten:

TensorRT optimization by Ultralytics:

Optimized model by Ultralytics results WITHOUT data transfers: The Optimized model by Ultralytics mean batch inference time is:1.3865331208705902 The Optimized model by Ultralytics std batch inference time is:0.011859980193208422

Nebullvm optimization:

Optimized model results WITHOUT data transfers: The Optimized model mean batch inference time is:3.081862072944641 The Optimized model std batch inference time is:0.14880756351818578

😞😞😞😞😞

I tried again with your modified Google Colab notebook and it is true that is 2x faster than the original PyTorch version but it is slower than the TensorRT provided by Ultralytics:

PyTorch model yolov5s

Original model results WITHOUT data transfers: The Original model mean batch inference time is:7.7102195215225215 The Original model std batch inference time is:0.967732421795685

Nebullvm optimization:

Optimized model results WITHOUT data transfers: The Optimized model mean batch inference time is:3.6612223982810974 The Optimized model std batch inference time is:0.11819371827359895

TensorRT by Ultralytics

Optimized model by Ultralytics results WITHOUT data transfers: The Optimized model by Ultralytics mean batch inference time is:1.5688025617599488 The Optimized model by Ultralytics std batch inference time is:0.03936980002624129

Check the notebook I modified here: https://colab.research.google.com/drive/1Nde0tCx28g3BTe2nxfLhTCMgIreqcvtw?usp=sharing

Hello @hdnh2006 ,

On the different performance respect to the Ultralytics implementation I think this can be due to the input your are giving to the optimize_model function. In fact, when the metric_drop_ths parameter is not given, speedster by default keeps the model in full precision 32 bits. Speedster supports both fp16 and int8 precisions but you have to activate them passing the metric_drop_ths parameter to the optimize_model function.

With fp16 precision in speedster I am getting 1.187 ms of inference time. I'm waiting for the int8 result.

Ok, so what are the values I should put to get fp16 precision?

I can see in the documentation that metric_drop_ths is a float number: https://github.com/nebuly-ai/nebullvm/blob/8aacdd7593746fd3cb71e6575847f028c9f6193d/apps/accelerate/speedster/speedster/api/functions.py#L86

model_optimized = optimize_model(
    model=core_wrapper,
    input_data=input_data,
    optimization_time="unconstrained",
    metric_drop_ths=0.1
)

I tried on TeslaV100 32GB and these are the results obtained:

Nebullvm optimization:

model_optimized = optimize_model(
    model=core_wrapper,
    input_data=input_data,
    optimization_time="unconstrained",
    metric_drop_ths=0.1
)

2023-01-12 19:11:22 | INFO     | Running Speedster on GPU
2023-01-12 19:11:23 | WARNING  | Missing Frameworks: tensorflow.
 Please install them to include them in the optimization pipeline.
2023-01-12 19:11:25 | INFO     | Benchmark performance of original model
2023-01-12 19:11:26 | INFO     | Original model latency: 0.005212109088897705 sec/iter
2023-01-12 19:11:27 | INFO     | Optimizing with PytorchBackendCompiler and q_type: None.
2023-01-12 19:11:29 | INFO     | Optimized model latency: 0.0037038326263427734 sec/iter
2023-01-12 19:11:29 | INFO     | Optimizing with PytorchBackendCompiler and q_type: QuantizationType.HALF.
2023-01-12 19:11:29 | WARNING  | Unable to trace model with torch.fx
2023-01-12 19:11:31 | INFO     | Optimized model latency: 0.0037364959716796875 sec/iter
2023-01-12 19:11:31 | INFO     | Optimizing with ONNXCompiler and q_type: None.
2023-01-12 19:11:33 | INFO     | Optimized model latency: 0.006529092788696289 sec/iter
2023-01-12 19:11:33 | INFO     | Optimizing with ONNXCompiler and q_type: QuantizationType.DYNAMIC.
2023-01-12 19:11:40 | WARNING  | The optimized model will be discarded due to poor results obtained with the given metric.
2023-01-12 19:11:40 | INFO     | Optimizing with ONNXCompiler and q_type: QuantizationType.HALF.
2023-01-12 19:11:43 | INFO     | Optimized model latency: 0.005489349365234375 sec/iter
2023-01-12 19:11:43 | INFO     | Optimizing with ONNXCompiler and q_type: QuantizationType.STATIC.
2023-01-12 19:11:58 | WARNING  | The optimized model will be discarded due to poor results obtained with the given metric.
2023-01-12 19:11:58 | INFO     | Optimizing with ONNXTensorRTCompiler and q_type: None.
2023-01-12 19:12:22 | INFO     | Optimized model latency: 0.00531458854675293 sec/iter
2023-01-12 19:12:22 | INFO     | Optimizing with ONNXTensorRTCompiler and q_type: QuantizationType.HALF.
2023-01-12 19:13:43 | WARNING  | The optimized model will be discarded due to poor results obtained with the given metric.
2023-01-12 19:13:43 | INFO     | Optimizing with ONNXTensorRTCompiler and q_type: QuantizationType.STATIC.
2023-01-12 19:16:53 | WARNING  | The optimized model will be discarded due to poor results obtained with the given metric.

[ Speedster results on GPU]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ Metric                                    ┃ Original Model   ┃ Optimized Model   ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━┫
┃ backend                                   ┃ PYTORCH          ┃ TorchScript       ┃
┃ latency                                   ┃ 0.0052 sec/batch ┃ 0.0037 sec/batch  ┃
┃ throughput                                ┃ 191.86 data/sec  ┃ 269.99 data/sec   ┃
┃ model size                                ┃ 35.18 MB         ┃ 28.38 MB          ┃
┃ metric drop (compute_relative_difference) ┃                  ┃ 0                 ┃
┃ speedup                                   ┃                  ┃ 1.41x             ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━┛

Optimized model results WITHOUT data transfers: The Optimized model mean batch inference time is:3.505920317173004 The Optimized model std batch inference time is:0.1691654167164366

TensorRT optimization by Ultralytics with half precision

Optimized model by Ultralytics results WITHOUT data transfers: The Optimized model by Ultralytics mean batch inference time is:1.442447043657303 The Optimized model by Ultralytics std batch inference time is:0.05623926929897324

Definitely, there's something I am doing wrong but I am following all the steps you provide.

Anyway, as you can see in this notebook, the same happens on Tesla T4

nebuly-ai / optimate