search

Deci-AI / data-gradients

Computer Vision dataset analysis
Apache License 2.0
293 stars 33 forks source link

I am using data-gradients for object-detection data analysis. but end up with the following error. Is this caused by different number of objects in each image? I am using Yolo-format label file. #172

Open krishiv26 opened 1 year ago

krishiv26 commented 1 year ago

💡 Your Question

File "analyse_dataset.py", line 27, in analyzer.run() File "/usr/local/lib/python3.8/dist-packages/data_gradients/managers/abstract_manager.py", line 226, in run self.execute() File "/usr/local/lib/python3.8/dist-packages/data_gradients/managers/abstract_manager.py", line 114, in execute for i, (train_batch, val_batch) in enumerate(datasets_tqdm): File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1195, in iter for obj in iterable: File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 635, in next data = self._next_data() File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 679, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch return self.collate_fn(data) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate return collate(batch, collate_fn_map=default_collate_fn_map) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/collate.py", line 143, in collate return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] # Backwards compatibility. File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/collate.py", line 143, in return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] # Backwards compatibility. File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/collate.py", line 120, in collate return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/collate.py", line 172, in collate_numpy_array_fn return collate([torch.as_tensor(b) for b in batch], collate_fn_map=collate_fn_map) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/collate.py", line 120, in collate return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map) File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/collate.py", line 163, in collate_tensor_fn return torch.stack(batch, 0, out=out) RuntimeError: stack expects each tensor to be equal size, but got [1, 5] at entry 0 and [4, 5] at entry 2

Versions

[pip3] numpy==1.22.2 [pip3] pytorch-quantization==2.1.2 [pip3] torch==1.14.0a0+44dac51 [pip3] torch-tensorrt==1.4.0.dev0 [pip3] torchtext==0.13.0a0+fae8e8c [pip3] torchvision==0.15.0a0 [pip3] triton==2.0.0

Louis-Dupont commented 1 year ago

Hi @krishiv26 could you please share the snippet of code you are using?

DarshnaZE commented 1 year ago

Hi @Louis-Dupont I am facing the same issue. and i am following the https://github.com/Deci-AI/data-gradients/blob/e468899d2368296ad7515439eca9524fbe38718e/examples/example_detection_super_gradients.py

When working with batch_size=1 , it works perfectly fine, faces the issue while increasing the batch_size

Louis-Dupont commented 1 year ago

@DarshnaZE you run the script as is and got the error ? Also which version of DG are you using ? I just tried with the master branch and it all worked without adding batch_size.

DarshnaZE commented 1 year ago

@Louis-Dupont Thanks. i tried with removing the batch_size param. it works same as when set batch_size=1. It takes around 3 hrs for around 200k images, as it runs on cpu (if i am not wrong). so increasing the batch_size will help decreasing the analysis time.