search

ultralytics / ultralytics

Ultralytics YOLO11 🚀
https://docs.ultralytics.com
GNU Affero General Public License v3.0
31.22k stars 6k forks source link

Error occurs after each epoch - AttributeError: 'NoneType' object has no attribute '_free_weak_ref' #702

Closed BraunGe closed 1 year ago

BraunGe commented 1 year ago

Search before asking

YOLOv8 Component

Training

Bug

Hi

I'm getting an error everytime I try to train the model. The error occurs before the first epoch starts and repeats for each epoch

AttributeError: 'NoneType' object has no attribute '_free_weak_ref'

The training continues, but I'm not sure what is the impact of this issue. Plus I would like to see the metrics of the epochs on a list, which with these errors is not possible as the erros occupy the whole command line. I know this error occured in yolov5 thanks to a pytorch version that was not compatible. I would like to know what should be done regarding this on yolov8.

Thanks

Environment

-YOLO: Ultralytics YOLOv8.0.21 Python-3.9.12 torch-1.11.0+cu113 CUDA: 0 (MX350, 2GB） -OS: Windows 10 -Python:3.9.12

Minimal Reproducible Example

yolo cfg=ultralytics\yolo\cfg\detect_train2.yaml

detect_train2.yaml： tralytics YOLO 🚀, GPL-3.0 license Default training settings and hyperparameters for medium-augmentation COCO training

task: "detect" # inference task, i.e. detect, segment, classify mode: "train" # YOLO mode, i.e. train, val, predict, export

Train settings ------------------------------------------------------------------------------------------------------- model: yolov8n.pt # path to model file, i.e. yolov8n.pt, yolov8n.yaml data: data/mydata2.yaml # path to data file, i.e. i.e. coco128.yaml epochs: 3 # number of epochs to train for patience: 50 # epochs to wait for no observable improvement for early stopping of training batch: 2 # number of images per batch (-1 for AutoBatch) imgsz: 640 # size of input images as integer or w,h save: True # save train checkpoints and predict results cache: False # True/ram, disk or False. Use cache for data loading device: null # device to run on, i.e. cuda device=0 or device=0,1,2,3 or device=cpu workers: 8 # number of worker threads for data loading (per RANK if DDP) project: null # project name name: null # experiment name exist_ok: False # whether to overwrite existing experiment pretrained: False # whether to use a pretrained model optimizer: 'SGD' # optimizer to use, choices=['SGD', 'Adam', 'AdamW', 'RMSProp'] verbose: False # whether to print verbose output seed: 0 # random seed for reproducibility deterministic: True # whether to enable deterministic mode single_cls: False # train multi-class data as single-class image_weights: False # use weighted image selection for training rect: False # support rectangular training cos_lr: False # use cosine learning rate scheduler close_mosaic: 10 # disable mosaic augmentation for final 10 epochs resume: False # resume training from last checkpoint Segmentation overlap_mask: True # masks should overlap during training (segment train only) mask_ratio: 4 # mask downsample ratio (segment train only) Classification dropout: 0.0 # use dropout regularization (classify train only)

Val/Test settings ---------------------------------------------------------------------------------------------------- val: True # validate/test during training save_json: False # save results to JSON file save_hybrid: False # save hybrid version of labels (labels + additional predictions) conf: 0.001 # object confidence threshold for detection (default 0.25 predict, 0.001 val) iou: 0.7 # intersection over union (IoU) threshold for NMS max_det: 300 # maximum number of detections per image half: False # use half precision (FP16) dnn: False # use OpenCV DNN for ONNX inference plots: True # save plots during train/val

Prediction settings -------------------------------------------------------------------------------------------------- source: null # source directory for images or videos show: False # show results if possible save_txt: False # save results as .txt file save_conf: False # save results with confidence scores save_crop: False # save cropped images with results hide_labels: False # hide labels hide_conf: False # hide confidence scores vid_stride: 1 # video frame-rate stride line_thickness: 3 # bounding box thickness (pixels) visualize: False # visualize model features augment: False # apply image augmentation to prediction sources agnostic_nms: False # class-agnostic NMS classes: null # filter results by class, i.e. class=0, or class=[0,2,3] retina_masks: False # use high-resolution segmentation masks box: True # Show boxes in segmentation predictions

Export settings ------------------------------------------------------------------------------------------------------ format: torchscript # format to export to keras: False # use Keras optimize: False # TorchScript: optimize for mobile int8: False # CoreML/TF INT8 quantization dynamic: False # ONNX/TF/TensorRT: dynamic axes simplify: False # ONNX: simplify model opset: 17 # ONNX: opset version workspace: 4 # TensorRT: workspace size (GB) nms: False # CoreML: add NMS

Hyperparameters ------------------------------------------------------------------------------------------------------ lr0: 0.01 # initial learning rate (i.e. SGD=1E-2, Adam=1E-3) lrf: 0.01 # final learning rate (lr0 * lrf) momentum: 0.937 # SGD momentum/Adam beta1 weight_decay: 0.0005 # optimizer weight decay 5e-4 warmup_epochs: 3.0 # warmup epochs (fractions ok) warmup_momentum: 0.8 # warmup initial momentum warmup_bias_lr: 0.1 # warmup initial bias lr box: 7.5 # box loss gain cls: 0.5 # cls loss gain (scale with pixels) dfl: 1.5 # dfl loss gain fl_gamma: 0.0 # focal loss gamma (efficientDet default gamma=1.5) label_smoothing: 0.0 # label smoothing (fraction) nbs: 64 # nominal batch size hsv_h: 0.015 # image HSV-Hue augmentation (fraction) hsv_s: 0.7 # image HSV-Saturation augmentation (fraction) hsv_v: 0.4 # image HSV-Value augmentation (fraction) degrees: 0.0 # image rotation (+/- deg) translate: 0.1 # image translation (+/- fraction) scale: 0.5 # image scale (+/- gain) shear: 0.0 # image shear (+/- deg) perspective: 0.0 # image perspective (+/- fraction), range 0-0.001 flipud: 0.0 # image flip up-down (probability) fliplr: 0.0 # image flip left-right (probability) mosaic: 1.0 # image mosaic (probability) mixup: 0.0 # image mixup (probability) copy_paste: 0.0 # segment copy-paste (probability)

Custom config.yaml --------------------------------------------------------------------------------------------------- cfg: null # for overriding defaults.yaml

Debug, do not modify ------------------------------------------------------------------------------------------------- v5loader: True # use legacy YOLOv5 dataloader

Additional

Packages in environment

absl-py 1.3.0 pypi_0 pypi addict 2.4.0 pypi_0 pypi albumentations 1.3.0 pypi_0 pypi antlr4-python3-runtime 4.9.3 pypi_0 pypi brotlipy 0.7.0 py39h2bbff1b_1003 ca-certificates 2022.3.29 haa95532_1 cachetools 5.2.0 pypi_0 pypi certifi 2021.10.8 py39haa95532_2 cffi 1.15.0 py39h2bbff1b_1 charset-normalizer 2.0.4 pyhd3eb1b0_0 click 8.1.3 pypi_0 pypi colorama 0.4.4 pyhd3eb1b0_0 coloredlogs 15.0.1 pypi_0 pypi commonmark 0.9.1 pypi_0 pypi conda 4.12.0 py39haa95532_0 conda-content-trust 0.1.1 pyhd3eb1b0_0 conda-package-handling 1.8.1 py39h8cc25b3_0 console_shortcut 0.1.1 4 contourpy 1.0.6 pypi_0 pypi cryptography 36.0.0 py39h21b164f_0 cvu-python 0.0.1a1 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.32 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi docker-pycreds 0.4.0 pypi_0 pypi ensemble-boxes 1.0.9 pypi_0 pypi fast-ctc-decode 0.3.2 pypi_0 pypi filelock 3.8.2 pypi_0 pypi flatbuffers 22.11.23 pypi_0 pypi fonttools 4.38.0 pypi_0 pypi gitdb 4.0.9 pypi_0 pypi gitpython 3.1.29 pypi_0 pypi google-auth 2.14.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi grpcio 1.50.0 pypi_0 pypi huggingface-hub 0.11.1 pypi_0 pypi humanfriendly 10.0 pypi_0 pypi hydra-core 1.3.1 pypi_0 pypi idna 3.3 pyhd3eb1b0_0 imagecodecs 2022.2.22 pypi_0 pypi imageio 2.22.3 pypi_0 pypi importlib-metadata 5.0.0 pypi_0 pypi install 1.3.5 pypi_0 pypi joblib 1.2.0 pypi_0 pypi jstyleson 0.0.2 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi llvmlite 0.39.1 pypi_0 pypi lmdb 1.3.0 pypi_0 pypi markdown 3.4.1 pypi_0 pypi markupsafe 2.1.1 pypi_0 pypi matplotlib 3.6.2 pypi_0 pypi menuinst 1.4.18 py39h59b6b97_0 mpmath 1.2.1 pypi_0 pypi networkx 2.8 pypi_0 pypi nibabel 4.0.2 pypi_0 pypi nltk 3.7 pypi_0 pypi numba 0.56.4 pypi_0 pypi numpy 1.23.1 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi omegaconf 2.3.0 pypi_0 pypi onnx 1.12.0 pypi_0 pypi onnx-simplifier 0.4.13 pypi_0 pypi onnxruntime 1.12.0 pypi_0 pypi onnxruntime-gpu 1.12.0 pypi_0 pypi opencv-python 4.6.0.66 pypi_0 pypi opencv-python-headless 4.6.0.66 pypi_0 pypi openssl 1.1.1n h2bbff1b_0 openvino 2022.2.0 pypi_0 pypi openvino-dev 2022.2.0 pypi_0 pypi openvino-telemetry 2022.1.2 pypi_0 pypi packaging 21.3 pypi_0 pypi pandas 1.1.5 pypi_0 pypi parasail 1.2.4 pypi_0 pypi pathtools 0.1.2 pypi_0 pypi pillow 9.3.0 pypi_0 pypi pip 21.2.4 py39haa95532_0 powershell_shortcut 0.0.1 3 progress 1.6 pypi_0 pypi promise 2.3 pypi_0 pypi protobuf 3.19.6 pypi_0 pypi py-cpuinfo 9.0.0 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pyclipper 1.3.0.post4 pypi_0 pypi pycocotools 2.0.2 pypi_0 pypi pycosat 0.6.3 py39h2bbff1b_0 pycparser 2.21 pyhd3eb1b0_0 pydicom 2.3.1 pypi_0 pypi pyopenssl 22.0.0 pyhd3eb1b0_0 pyparsing 3.0.9 pypi_0 pypi pyreadline3 3.4.1 pypi_0 pypi pysocks 1.7.1 py39haa95532_0 python 3.9.12 h6244533_0 python-dateutil 2.8.2 pypi_0 pypi pytz 2022.6 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pywin32 302 py39h2bbff1b_2 pyyaml 6.0 pypi_0 pypi qudida 0.0.4 pypi_0 pypi rawpy 0.17.3 pypi_0 pypi regex 2022.10.31 pypi_0 pypi requests 2.28.1 pypi_0 pypi requests-oauthlib 1.3.1 pypi_0 pypi rich 13.0.1 pypi_0 pypi rsa 4.9 pypi_0 pypi ruamel_yaml 0.15.100 py39h2bbff1b_0 scikit-image 0.19.3 pypi_0 pypi scikit-learn 0.24.2 pypi_0 pypi scipy 1.5.4 pypi_0 pypi seaborn 0.12.1 pypi_0 pypi sentencepiece 0.1.97 pypi_0 pypi sentry-sdk 1.10.1 pypi_0 pypi setproctitle 1.3.2 pypi_0 pypi setuptools 61.2.0 py39haa95532_0 shapely 1.8.5.post1 pypi_0 pypi shortuuid 1.0.9 pypi_0 pypi six 1.16.0 pyhd3eb1b0_1 smmap 5.0.0 pypi_0 pypi sqlite 3.38.2 h2bbff1b_0 sympy 1.11.1 pypi_0 pypi tensorboard 2.10.1 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi texttable 1.6.7 pypi_0 pypi thop 0.1.1-2209072238 pypi_0 pypi threadpoolctl 3.1.0 pypi_0 pypi tifffile 2022.10.10 pypi_0 pypi tokenizers 0.13.2 pypi_0 pypi torch 1.11.0+cu113 pypi_0 pypi torchaudio 0.11.0+cu113 pypi_0 pypi torchvision 0.12.0+cu113 pypi_0 pypi tqdm 4.64.1 pypi_0 pypi transformers 4.25.1 pypi_0 pypi typing-extensions 4.4.0 pypi_0 pypi tzdata 2022a hda174b7_0 ultralytics 8.0.17 pypi_0 pypi urllib3 1.26.12 pypi_0 pypi vc 14.2 h21ff451_1 vidsz 0.2.0 pypi_0 pypi vs2015_runtime 14.27.29016 h5e58377_2 wandb 0.13.5 pypi_0 pypi werkzeug 2.2.2 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0 win_inet_pton 1.1.0 py39haa95532_0 wincertstore 0.2 py39haa95532_2 yaml 0.2.5 he774522_0 zipp 3.10.0 pypi_0 pypi

Are you willing to submit a PR?

glenn-jocher commented 1 year ago

@BraunGe we don't see this in the latest torch 1.13.1 or torch 1.7 in CI. Can you upgrade torch and see if this resolves your issue?

BraunGe commented 1 year ago

@BraunGe we don't see this in the latest torch 1.13.1 or torch 1.7 in CI. Can you upgrade torch and see if this resolves your issue?

I will try that. Thank you.

BraunGe commented 1 year ago

@BraunGe we don't see this in the latest torch 1.13.1 or torch 1.7 in CI. Can you upgrade torch and see if this resolves your issue? upgrading torch resolves the issue. Thanks a lot.