search

ultralytics / yolov5

YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite
https://docs.ultralytics.com
GNU Affero General Public License v3.0
48.37k stars 15.87k forks source link

Ram usage #12838

Closed vishalb6401 closed 1 month ago

vishalb6401 commented 3 months ago

Search before asking

YOLOv5 Component

Training

Bug

wandb: WARNING ⚠️ wandb is deprecated and will be removed in a future release. See supported integrations at https://github.com/ultralytics/yolov5#integrations. 2024-03-22 08:15:25.291390: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-03-22 08:15:25.291487: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-03-22 08:15:25.431006: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered wandb: (1) Create a W&B account wandb: (2) Use an existing W&B account wandb: (3) Don't visualize my results wandb: Enter your choice: (30 second timeout) wandb: W&B disabled due to login timeout. train: weights=/kaggle/input/yolov5/yolov5m.pt, cfg=, data=/kaggle/input/yamltest123/set_1.yaml, hyp=data/hyps/hyp.scratch-low.yaml, epochs=50, batch_size=16, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, noplots=False, evolve=None, evolve_population=data/hyps, resume_evolve=None, bucket=, cache=None, image_weights=False, device=0, multi_scale=False, single_cls=False, optimizer=SGD, sync_bn=False, workers=8, project=yolo_vindr_12k_test, name=yolo_vindr_12k_1, exist_ok=False, quad=False, cos_lr=False, label_smoothing=0.0, patience=100, freeze=[0], save_period=-1, seed=0, local_rank=-1, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest, ndjson_console=False, ndjson_file=False github: up to date with https://github.com/ultralytics/yolov5 ✅ YOLOv5 🚀 v7.0-294-gdb125a20 Python-3.10.13 torch-2.1.2 CUDA:0 (Tesla P100-PCIE-16GB, 16276MiB)

hyperparameters: lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0 Comet: run 'pip install comet_ml' to automatically track and visualize YOLOv5 🚀 runs in Comet TensorBoard: Start with 'tensorboard --logdir yolo_vindr_12k_test', view at http://localhost:6006/ Downloading https://ultralytics.com/assets/Arial.ttf to /root/.config/Ultralytics/Arial.ttf... 100%|████████████████████████████████████████| 755k/755k [00:00<00:00, 23.0MB/s] Overriding model.yaml nc=80 with nc=2

             from  n    params  module                                  arguments

0 -1 1 5280 models.common.Conv [3, 48, 6, 2, 2]
1 -1 1 41664 models.common.Conv [48, 96, 3, 2]
2 -1 2 65280 models.common.C3 [96, 96, 2]
3 -1 1 166272 models.common.Conv [96, 192, 3, 2]
4 -1 4 444672 models.common.C3 [192, 192, 4]
5 -1 1 664320 models.common.Conv [192, 384, 3, 2]
6 -1 6 2512896 models.common.C3 [384, 384, 6]
7 -1 1 2655744 models.common.Conv [384, 768, 3, 2]
8 -1 2 4134912 models.common.C3 [768, 768, 2]
9 -1 1 1476864 models.common.SPPF [768, 768, 5]
10 -1 1 295680 models.common.Conv [768, 384, 1, 1]
11 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
12 [-1, 6] 1 0 models.common.Concat [1]
13 -1 2 1182720 models.common.C3 [768, 384, 2, False]
14 -1 1 74112 models.common.Conv [384, 192, 1, 1]
15 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
16 [-1, 4] 1 0 models.common.Concat [1]
17 -1 2 296448 models.common.C3 [384, 192, 2, False]
18 -1 1 332160 models.common.Conv [192, 192, 3, 2]
19 [-1, 14] 1 0 models.common.Concat [1]
20 -1 2 1035264 models.common.C3 [384, 384, 2, False]
21 -1 1 1327872 models.common.Conv [384, 384, 3, 2]
22 [-1, 10] 1 0 models.common.Concat [1]
23 -1 2 4134912 models.common.C3 [768, 768, 2, False]
24 [17, 20, 23] 1 28287 models.yolo.Detect [2, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [192, 384, 768]] Model summary: 291 layers, 20875359 parameters, 20875359 gradients, 48.2 GFLOPs

Transferred 475/481 items from /kaggle/input/yolov5/yolov5m.pt AMP: checks passed ✅ optimizer: SGD(lr=0.01) with parameter groups 79 weight(decay=0.0), 82 weight(decay=0.0005), 82 bias albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) train: Scanning /kaggle/input/test123/set_1/labels/train... 1600 images, 0 backg train: WARNING ⚠️ Cache directory /kaggle/input/test123/set_1/labels is not writeable: [Errno 30] Read-only file system: '/kaggle/input/test123/set_1/labels/train.cache.npy' val: Scanning /kaggle/input/test123/set_1/labels/val... 200 images, 0 background val: WARNING ⚠️ Cache directory /kaggle/input/test123/set_1/labels is not writeable: [Errno 30] Read-only file system: '/kaggle/input/test123/set_1/labels/val.cache.npy'

AutoAnchor: 3.35 anchors/target, 0.999 Best Possible Recall (BPR). Current anchors are a good fit to dataset ✅ Plotting labels to yolo_vindr_12k_test/yolo_vindr_12k_1/labels.jpg... /opt/conda/lib/python3.10/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead. with pd.option_context('mode.use_inf_as_na', True): /opt/conda/lib/python3.10/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead. with pd.option_context('mode.use_inf_as_na', True): /opt/conda/lib/python3.10/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead. with pd.option_context('mode.use_inf_as_na', True): /opt/conda/lib/python3.10/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead. with pd.option_context('mode.use_inf_as_na', True): /opt/conda/lib/python3.10/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead. with pd.option_context('mode.use_inf_as_na', True): /opt/conda/lib/python3.10/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead. with pd.option_context('mode.use_inf_as_na', True):

These are the warnings generated when i try to run a model and due to this my cpu usage gets over the limit provided in kaggle

Environment

Package Version Editable project location

absl-py 1.4.0 accelerate 0.28.0 access 1.1.9 affine 2.4.0 aiobotocore 2.12.1 aiofiles 22.1.0 aiohttp 3.9.1 aiohttp-cors 0.7.0 aioitertools 0.11.0 aiorwlock 1.3.0 aiosignal 1.3.1 aiosqlite 0.19.0 albumentations 1.4.0 alembic 1.13.1 altair 5.2.0 annotated-types 0.6.0 annoy 1.17.3 anyio 4.2.0 apache-beam 2.46.0 aplus 0.11.0 appdirs 1.4.4 archspec 0.2.2 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 array-record 0.5.0 arrow 1.3.0 arviz 0.17.1 astroid 3.0.3 astropy 6.0.0 astropy-iers-data 0.2024.3.18.0.29.47 asttokens 2.4.1 astunparse 1.6.3 async-lru 2.0.4 async-timeout 4.0.3 attrs 23.2.0 audioread 3.0.1 autopep8 2.0.4 Babel 2.14.0 backoff 2.2.1 bayesian-optimization 1.4.3 beatrix_jupyterlab 2023.128.151533 beautifulsoup4 4.12.2 bidict 0.23.1 blake3 0.2.1 bleach 6.1.0 blessed 1.20.0 blinker 1.7.0 blis 0.7.10 blosc2 2.5.1 bokeh 3.3.4 boltons 23.1.1 Boruta 0.3 boto3 1.26.100 botocore 1.34.51 bq_helper 0.4.1 /root/src/BigQuery_Helper bqplot 0.12.43 branca 0.7.1 brewer2mpl 1.4.1 Brotli 1.0.9 brotlipy 0.7.0 cached-property 1.5.2 cachetools 4.2.4 Cartopy 0.22.0 catalogue 2.0.10 catalyst 22.4 catboost 1.2.3 category-encoders 2.6.3 certifi 2024.2.2 cesium 0.12.1 cffi 1.16.0 charset-normalizer 3.3.2 chex 0.1.85 cleverhans 4.0.0 click 8.1.7 click-plugins 1.1.1 cligj 0.7.2 cloud-tpu-client 0.10 cloud-tpu-profiler 2.4.0 cloudpathlib 0.16.0 cloudpickle 2.2.1 cmdstanpy 1.2.1 cmudict 1.0.21 colorama 0.4.6 colorcet 3.1.0 colorful 0.5.6 colorlog 6.8.2 colorlover 0.3.0 comm 0.2.1 conda 23.7.4 conda-libmamba-solver 23.7.0 conda-package-handling 2.2.0 conda_package_streaming 0.9.0 confection 0.1.4 contextily 1.5.2 contourpy 1.2.0 convertdate 2.4.0 crcmod 1.7 cryptography 41.0.7 cuda-python 12.4.0 cudf 23.8.0 cufflinks 0.17.3 cuml 23.8.0 cupy 13.0.0 CVXcanon 0.1.2 cycler 0.12.1 cymem 2.0.8 Cython 3.0.8 cytoolz 0.12.3 daal 2024.1.0 daal4py 2024.1.0 dacite 1.8.1 dask 2024.3.1 dask-cuda 23.8.0 dask-cudf 23.8.0 dask-expr 1.0.4 dataclasses-json 0.6.4 dataproc_jupyter_plugin 0.1.66 datasets 2.1.0 datashader 0.16.0 datatile 1.0.3 db-dtypes 1.2.0 deap 1.4.1 debugpy 1.8.0 decorator 5.1.1 deepdiff 6.7.1 defusedxml 0.7.1 Deprecated 1.2.14 deprecation 2.1.0 descartes 1.1.0 dill 0.3.8 dipy 1.9.0 distlib 0.3.8 distributed 2023.7.1 distro 1.9.0 dm-tree 0.1.8 docker 7.0.0 docker-pycreds 0.4.0 docopt 0.6.2 docstring-parser 0.15 docstring-to-markdown 0.15 docutils 0.20.1 earthengine-api 0.1.394 easydict 1.13 easyocr 1.7.1 ecos 2.0.13 eli5 0.13.0 emoji 2.10.1 en-core-web-lg 3.7.1 en-core-web-sm 3.7.1 entrypoints 0.4 ephem 4.1.5 esda 2.5.1 essentia 2.1b6.dev1110 et-xmlfile 1.1.0 etils 1.6.0 exceptiongroup 1.2.0 executing 2.0.1 explainable-ai-sdk 1.3.3 Farama-Notifications 0.0.4 fastai 2.7.14 fastapi 0.108.0 fastavro 1.9.3 fastcore 1.5.29 fastdownload 0.0.7 fasteners 0.19 fastjsonschema 2.19.1 fastprogress 1.0.3 fastrlock 0.8.2 fasttext 0.9.2 feather-format 0.4.1 featuretools 1.30.0 filelock 3.13.1 fiona 1.9.6 fitter 1.7.0 flake8 7.0.0 flashtext 2.7 Flask 3.0.2 flatbuffers 23.5.26 flax 0.8.2 folium 0.16.0 fonttools 4.47.0 fqdn 1.5.1 frozendict 2.4.0 frozenlist 1.4.1 fsspec 2024.3.0 funcy 2.0 fury 0.10.0 future 1.0.0 fuzzywuzzy 0.18.0 gast 0.5.4 gatspy 0.3 gcsfs 2023.12.2.post1 gensim 4.3.2 geographiclib 2.0 Geohash 1.0 geojson 3.1.0 geopandas 0.14.3 geoplot 0.5.1 geopy 2.4.1 geoviews 1.11.1 ggplot 0.11.5 giddy 2.3.5 gitdb 4.0.11 GitPython 3.1.41 google-ai-generativelanguage 0.4.0 google-api-core 2.11.1 google-api-python-client 2.122.0 google-apitools 0.5.31 google-auth 2.26.1 google-auth-httplib2 0.1.1 google-auth-oauthlib 1.2.0 google-cloud-aiplatform 0.6.0a1 google-cloud-artifact-registry 1.10.0 google-cloud-automl 1.0.1 google-cloud-bigquery 2.34.4 google-cloud-bigtable 1.7.3 google-cloud-core 2.4.1 google-cloud-datastore 2.19.0 google-cloud-dlp 3.14.0 google-cloud-jupyter-config 0.0.5 google-cloud-language 2.13.3 google-cloud-monitoring 2.18.0 google-cloud-pubsub 2.19.0 google-cloud-pubsublite 1.9.0 google-cloud-recommendations-ai 0.7.1 google-cloud-resource-manager 1.11.0 google-cloud-spanner 3.40.1 google-cloud-storage 1.44.0 google-cloud-translate 3.12.1 google-cloud-videointelligence 2.13.3 google-cloud-vision 2.8.0 google-crc32c 1.5.0 google-generativeai 0.4.1 google-pasta 0.2.0 google-resumable-media 2.7.0 googleapis-common-protos 1.62.0 gplearn 0.4.2 gpustat 1.0.0 gpxpy 1.6.2 graphviz 0.20.2 greenlet 3.0.3 grpc-google-iam-v1 0.12.7 grpcio 1.51.1 grpcio-status 1.48.1 gviz-api 1.10.0 gym 0.26.2 gym-notices 0.0.8 gymnasium 0.29.0 h11 0.14.0 h2o 3.46.0.1 h5netcdf 1.3.0 h5py 3.10.0 haversine 2.8.1 hdfs 2.7.3 hep-ml 0.7.2 hijri-converter 2.3.1 hmmlearn 0.3.2 holidays 0.24 holoviews 1.18.3 hpsklearn 0.1.0 html5lib 1.1 htmlmin 0.1.12 httpcore 1.0.4 httplib2 0.21.0 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.21.4 hunspell 0.5.5 husl 4.0.3 hydra-slayer 0.5.0 hyperopt 0.2.7 hypertools 0.8.0 idna 3.6 igraph 0.11.4 imagecodecs 2024.1.1 ImageHash 4.3.1 imageio 2.33.1 imbalanced-learn 0.12.0 imgaug 0.4.0 importlib-metadata 6.11.0 importlib-resources 6.1.1 inequality 1.0.1 iniconfig 2.0.0 ipydatawidgets 4.3.5 ipykernel 6.28.0 ipyleaflet 0.18.2 ipympl 0.7.0 ipython 8.20.0 ipython-genutils 0.2.0 ipython-sql 0.5.0 ipyvolume 0.6.3 ipyvue 1.10.2 ipyvuetify 1.9.2 ipywebrtc 0.6.0 ipywidgets 7.7.1 isoduration 20.11.0 isort 5.13.2 isoweek 1.3.3 itsdangerous 2.1.2 Janome 0.5.0 jaraco.classes 3.3.0 jax 0.4.23 jax-jumpy 1.0.0 jaxlib 0.4.23.dev20240116 jedi 0.19.1 jeepney 0.8.0 jieba 0.42.1 Jinja2 3.1.2 jmespath 1.0.1 joblib 1.3.2 json5 0.9.14 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.20.0 jsonschema-specifications 2023.12.1 jupyter_client 7.4.9 jupyter-console 6.6.3 jupyter_core 5.7.1 jupyter-events 0.9.0 jupyter-http-over-ws 0.0.8 jupyter-lsp 1.5.1 jupyter_server 2.13.0 jupyter_server_fileid 0.9.1 jupyter-server-mathjax 0.2.6 jupyter_server_proxy 4.1.0 jupyter_server_terminals 0.5.1 jupyter_server_ydoc 0.8.0 jupyter-ydoc 0.2.5 jupyterlab 4.1.5 jupyterlab_git 0.44.0 jupyterlab-lsp 5.1.0 jupyterlab_pygments 0.3.0 jupyterlab_server 2.25.2 jupyterlab-widgets 3.0.9 jupytext 1.16.0 kaggle 1.6.6 kaggle-environments 1.14.3 kagglehub 0.2.0 keras 3.0.5 keras-cv 0.8.2 keras-nlp 0.8.2 keras-tuner 1.4.6 kernels-mixer 0.0.7 keyring 24.3.0 keyrings.google-artifactregistry-auth 1.1.2 kfp 2.5.0 kfp-pipeline-spec 0.2.2 kfp-server-api 2.0.5 kiwisolver 1.4.5 kmapper 2.0.1 kmodes 0.12.2 korean-lunar-calendar 0.3.1 kornia 0.7.2 kornia_rs 0.1.2 kt-legacy 1.0.5 kubernetes 26.1.0 langcodes 3.3.0 langid 1.1.6 lazy_loader 0.3 learntools 0.3.4 leven 1.0.4 Levenshtein 0.25.0 libclang 16.0.6 libmambapy 1.5.0 libpysal 4.9.2 librosa 0.10.1 lightgbm 4.2.0 lightning-utilities 0.10.1 lime 0.2.0.1 line-profiler 4.1.2 linkify-it-py 2.0.3 llvmlite 0.41.1 lml 0.1.0 locket 1.0.0 loguru 0.7.2 LunarCalendar 0.0.9 lxml 5.1.0 lz4 4.3.3 Mako 1.3.2 mamba 1.5.0 mapclassify 2.6.1 Markdown 3.5.2 markdown-it-py 3.0.0 markovify 0.9.4 MarkupSafe 2.1.3 marshmallow 3.21.1 matplotlib 3.7.5 matplotlib-inline 0.1.6 matplotlib-venn 0.11.10 mccabe 0.7.0 mdit-py-plugins 0.4.0 mdurl 0.1.2 memory-profiler 0.61.0 menuinst 2.0.1 mercantile 1.2.1 mgwr 2.2.1 missingno 0.5.2 mistune 0.8.4 mizani 0.11.0 ml-dtypes 0.2.0 mlcrate 0.2.0 mlens 0.2.3 mlxtend 0.23.1 mmh3 4.1.0 mne 1.6.1 mnist 0.2.2 mock 5.1.0 momepy 0.7.0 more-itertools 10.2.0 mpld3 0.5.10 mpmath 1.3.0 msgpack 1.0.7 msgpack-numpy 0.4.8 multidict 6.0.4 multimethod 1.10 multipledispatch 1.0.0 multiprocess 0.70.16 munkres 1.1.4 murmurhash 1.0.10 mypy-extensions 1.0.0 namex 0.0.7 nb_conda 2.2.1 nb-conda-kernels 2.3.1 nbclassic 1.0.0 nbclient 0.5.13 nbconvert 6.4.5 nbdime 3.2.0 nbformat 5.9.2 ndindex 1.8 nest-asyncio 1.5.8 networkx 3.2.1 nibabel 5.2.1 nilearn 0.10.3 ninja 1.11.1.1 nltk 3.2.4 nose 1.3.7 notebook 6.5.4 notebook_executor 0.2 notebook_shim 0.2.3 numba 0.58.1 numexpr 2.9.0 numpy 1.26.4 nvidia-ml-py 11.495.46 nvtx 0.2.10 oauth2client 4.1.3 oauthlib 3.2.2 objsize 0.6.1 odfpy 1.4.1 olefile 0.47 onnx 1.15.0 opencensus 0.11.4 opencensus-context 0.1.3 opencv-contrib-python 4.9.0.80 opencv-python 4.9.0.80 opencv-python-headless 4.9.0.80 openpyxl 3.1.2 openslide-python 1.3.1 opentelemetry-api 1.22.0 opentelemetry-exporter-otlp 1.22.0 opentelemetry-exporter-otlp-proto-common 1.22.0 opentelemetry-exporter-otlp-proto-grpc 1.22.0 opentelemetry-exporter-otlp-proto-http 1.22.0 opentelemetry-proto 1.22.0 opentelemetry-sdk 1.22.0 opentelemetry-semantic-conventions 0.43b0 opt-einsum 3.3.0 optax 0.2.1 optuna 3.6.0 orbax-checkpoint 0.5.6 ordered-set 4.1.0 orjson 3.9.10 ortools 9.4.1874 osmnx 1.9.1 overrides 7.4.0 packaging 21.3 pandas 2.1.4 pandas-datareader 0.10.0 pandas-profiling 3.6.6 pandas-summary 0.2.0 pandasql 0.7.3 pandocfilters 1.5.0 panel 1.3.8 papermill 2.5.0 param 2.0.2 parso 0.8.3 partd 1.4.1 path 16.10.0 path.py 12.5.0 pathos 0.3.2 pathy 0.10.3 patsy 0.5.6 pdf2image 1.17.0 pettingzoo 1.24.0 pexpect 4.8.0 phik 0.12.4 pickleshare 0.7.5 Pillow 9.5.0 pip 24.0 pkgutil_resolve_name 1.3.10 platformdirs 4.2.0 plotly 5.18.0 plotly-express 0.4.1 plotnine 0.13.2 pluggy 1.4.0 pointpats 2.4.0 polars 0.20.15 polyglot 16.7.4 pooch 1.8.1 pox 0.3.4 ppca 0.0.4 ppft 1.7.6.8 preprocessing 0.1.13 preshed 3.0.9 prettytable 3.9.0 progressbar2 4.4.2 prometheus-client 0.19.0 promise 2.3 prompt-toolkit 3.0.42 pronouncing 0.2.0 prophet 1.1.1 proto-plus 1.23.0 protobuf 3.20.3 psutil 5.9.3 ptyprocess 0.7.0 pudb 2024.1 PuLP 2.8.0 pure-eval 0.2.2 py-cpuinfo 9.0.0 py-spy 0.3.14 py4j 0.10.9.7 pyaml 23.12.0 PyArabic 0.6.15 pyarrow 11.0.0 pyasn1 0.5.1 pyasn1-modules 0.3.0 PyAstronomy 0.21.0 pybind11 2.11.1 pyclipper 1.3.0.post5 pycodestyle 2.11.1 pycosat 0.6.6 pycparser 2.21 pycryptodome 3.20.0 pyct 0.5.0 pycuda 2024.1 pydantic 2.5.3 pydantic_core 2.14.6 pydegensac 0.1.2 pydicom 2.4.4 pydocstyle 6.3.0 pydot 1.4.2 pydub 0.25.1 pyemd 1.0.0 pyerfa 2.0.1.1 pyexcel-io 0.6.6 pyexcel-ods 0.6.0 pyflakes 3.2.0 pygltflib 1.16.2 Pygments 2.17.2 PyJWT 2.8.0 pykalman 0.9.5 pyLDAvis 3.4.1 pylibraft 23.8.0 pylint 3.0.4 pymc3 3.11.4 PyMeeus 0.5.12 pymongo 3.13.0 Pympler 1.0.1 pynndescent 0.5.11 pynvml 11.4.1 pynvrtc 9.2 pyocr 0.8.5 pyOpenSSL 23.3.0 pyparsing 3.1.1 pypdf 4.1.0 pyproj 3.6.1 pysal 24.1 pyshp 2.3.1 PySocks 1.7.1 pytesseract 0.3.10 pytest 8.1.1 python-bidi 0.4.2 python-dateutil 2.9.0.post0 python-dotenv 1.0.0 python-json-logger 2.0.7 python-Levenshtein 0.25.0 python-louvain 0.16 python-lsp-jsonrpc 1.1.2 python-lsp-server 1.10.1 python-slugify 8.0.4 python-utils 3.8.2 pythreejs 2.4.2 pytoolconfig 1.3.1 pytools 2023.1.1 pytorch-ignite 0.4.13 pytorch-lightning 2.2.1 pytz 2023.3.post1 pyu2f 0.1.5 PyUpSet 0.1.1.post7 pyviz_comms 3.0.1 PyWavelets 1.5.0 PyYAML 6.0.1 pyzmq 24.0.1 qgrid 1.3.1 qtconsole 5.5.1 QtPy 2.4.1 quantecon 0.7.2 quantities 0.15.0 qudida 0.0.4 raft-dask 23.8.0 rapidfuzz 3.6.2 rasterio 1.3.9 rasterstats 0.19.0 ray 2.9.0 ray-cpp 2.9.0 referencing 0.32.1 regex 2023.12.25 requests 2.31.0 requests-oauthlib 1.3.1 requests-toolbelt 0.10.1 responses 0.18.0 retrying 1.3.3 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rgf-python 3.12.0 rich 13.7.0 rich-click 1.7.4 rmm 23.8.0 rope 1.12.0 rpds-py 0.16.2 rsa 4.9 Rtree 1.2.0 ruamel.yaml 0.17.40 ruamel.yaml.clib 0.2.7 ruamel-yaml-conda 0.15.100 s2sphere 0.2.5 s3fs 2024.3.0 s3transfer 0.6.2 safetensors 0.4.2 scattertext 0.1.19 scikit-image 0.22.0 scikit-learn 1.2.2 scikit-learn-intelex 2024.1.0 scikit-multilearn 0.2.0 scikit-optimize 0.10.1 scikit-plot 0.3.7 scikit-surprise 1.1.3 scipy 1.11.4 seaborn 0.12.2 SecretStorage 3.3.3 segment_anything 1.0 segregation 2.5 semver 3.0.2 Send2Trash 1.8.2 sentencepiece 0.2.0 sentry-sdk 1.42.0 setproctitle 1.3.3 setuptools 69.2.0 setuptools-git 1.2 setuptools-scm 8.0.4 shap 0.44.1 Shapely 1.8.5.post1 shellingham 1.5.4 Shimmy 1.3.0 simpervisor 1.0.0 SimpleITK 2.3.1 simplejson 3.19.2 six 1.16.0 sklearn-pandas 2.2.0 slicer 0.0.7 smart-open 6.4.0 smmap 5.0.1 sniffio 1.3.0 snowballstemmer 2.2.0 snuggs 1.4.7 sortedcontainers 2.4.0 soundfile 0.12.1 soupsieve 2.5 soxr 0.3.7 spacy 3.7.2 spacy-legacy 3.0.12 spacy-loggers 1.0.5 spaghetti 1.7.5.post1 spectral 0.23.1 spglm 1.1.0 sphinx-rtd-theme 0.2.4 spint 1.0.7 splot 1.1.5.post1 spopt 0.6.0 spreg 1.4.2 spvcm 0.3.0 SQLAlchemy 2.0.25 sqlparse 0.4.4 squarify 0.4.3 srsly 2.4.8 stable-baselines3 2.1.0 stack-data 0.6.2 stanio 0.3.0 starlette 0.32.0.post1 statsmodels 0.14.1 stemming 1.0.1 stop-words 2018.7.23 stopit 1.1.2 stumpy 1.12.0 sympy 1.12 tables 3.9.2 tabulate 0.9.0 tangled-up-in-unicode 0.2.0 tbb 2021.11.0 tblib 3.0.0 tenacity 8.2.3 tensorboard 2.15.1 tensorboard-data-server 0.7.2 tensorboard-plugin-profile 2.15.0 tensorboardX 2.6.2.2 tensorflow 2.15.0 tensorflow-cloud 0.1.16 tensorflow-datasets 4.9.4 tensorflow-decision-forests 1.8.1 tensorflow-estimator 2.15.0 tensorflow-hub 0.16.1 tensorflow-io 0.35.0 tensorflow-io-gcs-filesystem 0.35.0 tensorflow-metadata 0.14.0 tensorflow-probability 0.23.0 tensorflow-serving-api 2.14.1 tensorflow-text 2.15.0 tensorflow-transform 0.14.0 tensorpack 0.11 tensorstore 0.1.56 termcolor 2.4.0 terminado 0.18.0 testpath 0.6.0 text-unidecode 1.3 textblob 0.18.0.post0 texttable 1.7.0 tf_keras 2.15.1 tfp-nightly 0.24.0.dev0 Theano 1.0.5 Theano-PyMC 1.1.2 thinc 8.2.2 thop 0.1.1.post2209072238 threadpoolctl 3.2.0 tifffile 2023.12.9 timm 0.9.16 tinycss2 1.2.1 tobler 0.11.2 tokenizers 0.15.2 toml 0.10.2 tomli 2.0.1 tomlkit 0.12.4 toolz 0.12.1 torch 2.1.2 torchaudio 2.1.2 torchdata 0.7.1 torchinfo 1.8.0 torchmetrics 1.3.2 torchtext 0.16.2 torchvision 0.16.2 tornado 6.3.3 TPOT 0.12.1 tqdm 4.66.1 traceml 1.0.8 traitlets 5.9.0 traittypes 0.2.1 transformers 4.38.2 treelite 3.2.0 treelite-runtime 3.2.0 trueskill 0.4.5 truststore 0.8.0 trx-python 0.2.9 tsfresh 0.20.2 typeguard 4.1.5 typer 0.9.0 types-python-dateutil 2.8.19.20240106 typing_extensions 4.9.0 typing-inspect 0.9.0 typing-utils 0.1.0 tzdata 2023.4 uc-micro-py 1.0.3 ucx-py 0.33.0 ujson 5.9.0 ultralytics 8.1.30 umap-learn 0.5.5 unicodedata2 15.1.0 Unidecode 1.3.8 update-checker 0.18.0 uri-template 1.3.0 uritemplate 3.0.1 urllib3 1.26.18 urwid 2.6.9 urwid_readline 0.14 uvicorn 0.25.0 uvloop 0.19.0 vaex 4.17.0 vaex-astro 0.9.3 vaex-core 4.17.1 vaex-hdf5 0.14.1 vaex-jupyter 0.8.2 vaex-ml 0.18.3 vaex-server 0.9.0 vaex-viz 0.5.4 vec_noise 1.1.4 vecstack 0.4.0 virtualenv 20.21.0 visions 0.7.5 vowpalwabbit 9.9.0 vtk 9.3.0 Wand 0.6.13 wandb 0.16.4 wasabi 1.1.2 watchfiles 0.21.0 wavio 0.0.8 wcwidth 0.2.13 weasel 0.3.4 webcolors 1.13 webencodings 0.5.1 websocket-client 1.7.0 websockets 12.0 Werkzeug 3.0.1 wfdb 4.1.2 whatthepatch 1.0.5 wheel 0.43.0 widgetsnbextension 3.6.6 witwidget 1.8.1 woodwork 0.29.0 wordcloud 1.9.3 wordsegment 1.3.1 wrapt 1.14.1 xarray 2024.2.0 xarray-einstats 0.7.0 xgboost 2.0.3 xvfbwrapper 0.2.9 xxhash 3.4.1 xyzservices 2023.10.1 y-py 0.6.2 yapf 0.40.2 yarl 1.9.3 ydata-profiling 4.6.4 yellowbrick 1.5 ypy-websocket 0.8.4 zict 3.0.0 zipp 3.17.0 zstandard 0.22.0

Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?

github-actions[bot] commented 3 months ago

👋 Hello @vishalb6401, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Status

YOLOv5 CI

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics
vishalb6401 commented 3 months ago

https://www.kaggle.com/code/vishalbbbb/notebookaffd953d12

u can refer my notebook as well

glenn-jocher commented 3 months ago

@vishalb6401 hi there! 👋

Thanks for reaching out and for sharing your notebook. It seems like there might not have been a specific question asked or an issue described related to YOLOv5. Could you please clarify or provide more details on the issue you're facing or the question you have? That way, I can offer you more targeted help or advice.

Looking forward to hearing from you!

vishalb6401 commented 3 months ago

https://github.com/ultralytics/yolov5/issues/12839

can u come to this by mistake i have opened a new acc

glenn-jocher commented 3 months ago

@vishalb6401 hi there 👋,

No worries about the mix-up! Could you please summarize the issue or question you're facing here? That way, we can assist you directly within this thread.

Thanks! Looking forward to helping out.

vishalb6401 commented 3 months ago

https://www.kaggle.com/code/vishalb19mia1012/vindr-yolo

please refer this notebook

While executing the 12k and 10k there are issues where my ram usage goes to the peak and my notebook crashes and the list of warnings are very new as well

but if i do the same for 4k-test-640 the code is running

idk what im dong wrong here im been stuck for 2 weeks please help

and here the number basically means the number of images(breast cancer images) ill share the code how the images are generated as well im doing some preprocessing

from PIL import Image import matplotlib.pyplot as plt import albumentations as A import numpy as np import os import zipfile import pydicom import cv2 import matplotlib.patches as patches from math import isnan

zip_file_path = 'D:\Python files\yolo_breast_cancer\vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0.zip'

def load_and_transform_dicom_image(zip_file_path, study_id, image_id, output_image_dir, output_label_dir, original_bounding_boxes, image_number): try: with zipfile.ZipFile(zip_file_path, 'r') as zip_ref: with zip_ref.open(f'vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/images/{study_id}/{image_id}.dicom') as file: ds = pydicom.dcmread(file)

pixel_data = ds.pixel_array
min_pixel_value = pixel_data.min()
max_pixel_value = pixel_data.max()
pixel_data = ((pixel_data - min_pixel_value) / (max_pixel_value - min_pixel_value) * 255).astype(int)
pixel_data = cv2.convertScaleAbs(pixel_data)
pixel_data = cv2.cvtColor(pixel_data, cv2.COLOR_BGR2GRAY) if len(pixel_data.shape) == 3 else pixel_data
clahe = cv2.createCLAHE(clipLimit=30.0, tileGridSize=(8, 8))
pixel_data = clahe.apply(pixel_data)

for i in range(len(original_bounding_boxes)):
    bbox_list = list(original_bounding_boxes[i])
    bbox_list[2] = np.abs(bbox_list[2] - 0.5 / pixel_data.shape[0])
    bbox_list[3] = np.abs(bbox_list[3] - 0.5 / pixel_data.shape[1])
    original_bounding_boxes[i] = tuple(bbox_list)

# Check the value of 'upsampling'
if row['Oversampling'] == 1:
    transform = A.Compose([
        A.RandomRotate90(p=0.5),
        A.Rotate(limit=(-60, 60), interpolation=cv2.INTER_LINEAR, border_mode=cv2.BORDER_CONSTANT, p=0.5),
        A.Transpose(p=0.5),
        A.HorizontalFlip(p=0.5),
        A.VerticalFlip(p=0.5),
        A.Resize(height=640, width=640, always_apply=True),
    ], bbox_params={'format': 'pascal_voc'})

    transformed = transform(image=pixel_data, bboxes=np.array(original_bounding_boxes, dtype=np.float32))
    transformed_bboxes = transformed["bboxes"]
else:
    transform = A.Compose([
        A.Resize(height=640, width=640, always_apply=True),
    ], bbox_params={'format': 'pascal_voc'})

    transformed = transform(image=pixel_data, bboxes=np.array(original_bounding_boxes, dtype=np.float32))
    transformed_bboxes = transformed["bboxes"]

study_id = row['study_id']
image_id = row['image_id']
classification = row['classification']
oversampling = row['Oversampling']

for bbox in transformed_bboxes:
    yolo_format = convert_to_yolo_format(640, 640, bbox)
    print(f"YOLO Coordinates: {yolo_format}")

image_file_name = f'{oversampling}_{classification}_{image_number}.png'
bbox_file_name = f'{oversampling}_{classification}_{image_number}.txt'

image_file_path = os.path.join(output_image_dir, image_file_name)
bbox_file_path = os.path.join(output_label_dir, bbox_file_name)

cv2.imwrite(image_file_path, transformed["image"])

with open(bbox_file_path, 'w') as bbox_file:
    for bbox in transformed_bboxes:
        yolo_format = convert_to_yolo_format(640, 640, bbox)
        bbox_file.write(yolo_format + "\n")

return {
    "transformed_image": transformed["image"],
    "transformed_bboxes": transformed_bboxes
}

except ValueError as ve: print(f"Error processing Study ID: {study_id}, Image ID: {image_id}") print(f"Error message: {ve}") return { "study_id": study_id, "image_id": image_id } def display_image_dimensions(transformed_image): height, width = transformed_image.shape[:2] print(f"Transformed Image Dimensions: Height={height}, Width={width}")

def convert_to_yolo_format(image_width, image_height, bbox): x_min, y_min, x_max, y_max, class_label = bbox center_x = (x_min + x_max) / 2 center_y = (y_min + y_max) / 2 width = x_max - x_min height = y_max - y_min center_x /= image_width center_y /= image_height width /= image_width height /= image_height return f"{class_label} {center_x} {center_y} {width} {height}"

error_images = [] image_number_counter = 1

for _, row in dataset_new.iterrows(): study_id = row['study_id'] image_id = row['image_id'] if row['classification'] == 'Malignant': class_label = 1 else: class_label = 0

x_min = row['xmin'] y_min = row['ymin'] x_max = row['xmax'] y_max = row['ymax'] image_width = row['width'] image_height = row['height']

if isnan(x_min) or isnan(x_max) or isnan(y_min) or isnan(y_max): x_min, y_min, x_max, y_max = 0, 0, image_width, image_height

original_bounding_boxes = [(x_min, y_min, x_max, y_max, class_label)]

output_image_directory = 'D:\Python files\yolo_breast_cancer\dataset_new_images' output_label_directory = 'D:\Python files\yolo_breast_cancer\dataset_new_labels' os.makedirs(output_image_directory, exist_ok=True) os.makedirs(output_label_directory, exist_ok=True)

result = load_and_transform_dicom_image(zip_file_path, study_id, image_id, output_image_directory, output_label_directory, original_bounding_boxes, image_number_counter) if result is not None and "transformed_image" in result and "transformed_bboxes" in result: error_images.append(result)

Display the transformed image dimensions

display_image_dimensions(result["transformed_image"])

image_number_counter += 1 print("Study ID and Image ID pairs for images with ValueError:", error_images)

vishalb6401 commented 3 months ago

image image

vishalb6401 commented 3 months ago

image image now u can see that this is running perfectly this is 4k images

glenn-jocher commented 3 months ago

@vishalb6401 it looks like the issue you're experiencing with RAM consumption and your notebook crashing when processing larger datasets (12k and 10k images) versus a smaller one (4k images) could be due to several factors, including the size of the images, preprocessing steps, and overall memory management in your code.

When working with larger datasets:

  1. Optimize Image Loading: Ensure images are loaded in a memory-efficient manner. Consider reading images in batches if you're not already doing so.
  2. Reduce Image Size: If high-resolution images are not essential for your task, resizing images to a lower resolution can significantly reduce memory usage.
  3. Memory Profiling: Use memory profiling tools to identify where the most memory is being used in your script. This can help you pinpoint areas for optimization.

Given your preprocessing steps involve transformations which could be memory-intensive (like resizing and augmentation), ensure these transformations are necessary and optimized for your task.

Since your code is running fine for 4k images, also consider if parallel processing or distributing the load across multiple notebooks/sessions is feasible for your larger datasets, breaking down the problem into smaller, more manageable chunks.

Lastly, checking for memory leaks in your loop or function that handles image processing might be beneficial. Python's garbage collector should automatically manage memory, but in some cases, manual intervention may be needed to free up unused objects.

Unfortunately, without specific error messages or deeper insights into the exact failure point, these are general suggestions. Optimizing for large-scale data processing is an iterative process, focusing on efficient data handling and memory management.

vishalb6401 commented 3 months ago

oh yea i forgot to add this under 12k and 10k ive basically split the dataset into 6 and 5 sets each of 2k images respectively so its only using less ram already but if i directly run for 4k images it is running

glenn-jocher commented 3 months ago

@vishalb6401 hi there! 😊

Thanks for adding more context. If splitting the large datasets into smaller chunks still leads to high RAM usage, it may suggest the processing per chunk (even with 2k images) is quite intensive. Since the direct run with 4k images works fine, it might be a good idea to ensure that between processing each chunk, resources are being efficiently released and any in-memory data structures are minimized or cleared.

For effective memory management, consider explicitly releasing variables that are no longer needed using del variable_name and invoking garbage collection manually with import gc; gc.collect() after processing each chunk. Another strategy might involve reviewing any data augmentation or image transformation steps to ensure they are memory-efficient.

Here's a quick snippet to illustrate manual resource management between chunks:

import gc

for chunk in dataset_chunks:
    process_chunk(chunk)  # Your processing function
    del chunk  # Explicitly delete the chunk to free up RAM
    gc.collect()  # Manually invoke garbage collection

Optimizing chunk processing by carefully managing memory can mitigate the high RAM usage issue. Keep an eye on the resource usage throughout the process to pinpoint any specific step that may be responsible for spikes in RAM consumption.

Hope this helps you move forward! Let us know how it goes.

vishalb6401 commented 3 months ago

where should i add this code? when im running my yolo or when im preprocessing my data?

glenn-jocher commented 3 months ago

@vishalb6401 hi there! 👋

Great question! If you're referring to memory management or any data processing code, it's best to incorporate that during your preprocessing stage. This helps ensure your data is in the optimal format and size before it goes into your YOLO model for training or inference, which can significantly affect performance and efficiency.

In short, manage and preprocess your data first to get it ready, then run your YOLO model with the prepared data. This approach keeps everything running smoothly! 😊

Happy coding!

vishalb6401 commented 3 months ago

Even then the preprocessing is only going to be done efficiently but not the running of a yolo model

glenn-jocher commented 2 months ago

@vishalb6401 Hey there! 😊 Absolutely, preprocessing optimizes data handling, but for running YOLO models efficiently, consider:

  1. Using a smaller model variant (if not already) like yolov5s can help reduce memory and computational requirements.
  2. Reduce batch size during training or inference to lessen RAM usage.
  3. Enable mixed precision training with --precision half if supported by your hardware to improve speed and reduce memory footprint.

By combining optimized data preprocessing with these running strategies, you should see an improvement in efficiency. Happy coding! 🚀

github-actions[bot] commented 1 month ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐