search

johnolafenwa / DeepStack

The World's Leading Cross Platform AI Engine for Edge Devices
Apache License 2.0
673 stars 106 forks source link

The size of tensor a (32) must match the size of tensor b (40) at non-singleton dimension 2 #151

Open babbahotep opened 2 years ago

babbahotep commented 2 years ago

Using Ubuntu Deepstack docker 2022.01.1 on esxi with grid Nvidia GPU. RTX A5000 After sometime Deepstack seems to crash with:

File "/app/intelligencelayer/shared/detection.py", line 81, in run_task det = detector.predict(img_path, threshold) File "/app/intelligencelayer/shared/./process.py", line 63, in predict pred = self.model(img, augment=False)[0] File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "/app/intelligencelayer/shared/./models/yolo.py", line 149, in forward return self.forward_once(x, profile) # single-scale inference, train File "/app/intelligencelayer/shared/./models/yolo.py", line 176, in forward_once x = m(x) # run File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, *kwargs) File "/app/intelligencelayer/shared/./models/yolo.py", line 75, in forward y[..., 0:2] 2.0 - 0.5 + self.grid[i].to(x[i].device) RuntimeError: The size of tensor a (32) must match the size of tensor b (40) at non-singleton dimension 2

Anyway increase logging level to get timestamps? Anyone have an idea what’s causing this crash? Any options to catch this and restart deepstack docker instance automatically?

MisterEkole commented 2 years ago

Hi @babbahotep, this error is as a result of shape mismatch. I'd suggest you make sure the shape of both input tensors match, so in place addition can be executed hitch free.

babbahotep commented 2 years ago

Input is blue iris i have no control over the input. this only happens on esxi docker with vgpu. DS on windows bare metal works fine.