search

ocean-data-factory-sweden / kso

Notebooks to upload/download marine footage, connect to a citizen science project, train machine learning models and publish marine biological observations.
GNU General Public License v3.0
5 stars 11 forks source link

Issue with Training polygon based model in Alvis #443

Open Bergylta opened 2 weeks ago

Bergylta commented 2 weeks ago

🐛 Bug

The workaround might have worked, but a new issue showed up in the train_models.ipynb notebook, unsure if this has to do with the baseline model, Alvis, SNIC, mlflow or something else entirely.

To Reproduce (REQUIRED)

Batch size: 8 Epochs: 50 mlp.registry= mlflow Model: YoloV8 Segmentation Input:

mlp.train_yolo(
    exp_name=exp_name.value,
    weights=weights.artifact_path,
    project=mlp.project_name,
    epochs=epochs.value,
    batch_size=batch_size.value,
    img_size=img_h.value,  # this requires an int
)

a = torch.tensor([5])
c = a / 0

Output:

Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8m.pt to 'yolov8m.pt'...
100%|██████████| 49.7M/49.7M [00:00<00:00, 71.7MB/s]
New https://pypi.org/project/ultralytics/8.3.27 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.0.200 🚀 Python-3.8.18 torch-2.4.1+cu121 CUDA:0 (NVIDIA A40, 45739MiB)
engine/trainer: task=detect, mode=train, model=yolov8m.pt, data=/mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/Koster_Seafloor_Obs_11_42_08.yaml, epochs=50, patience=50, batch=8, imgsz=640, save=True, save_period=-1, cache=False, device=None, workers=8, project=koster_seafloor_obs, name=KSO_Paulina_oldmodel_2, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, vid_stride=1, stream_buffer=False, line_width=None, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, boxes=True, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, tracker=botsort.yaml, save_dir=koster_seafloor_obs/KSO_Paulina_oldmodel_2
Overriding model.yaml nc=80 with nc=8

                   from  n    params  module                                       arguments                     
  0                  -1  1      1392  ultralytics.nn.modules.conv.Conv             [3, 48, 3, 2]                 
  1                  -1  1     41664  ultralytics.nn.modules.conv.Conv             [48, 96, 3, 2]                
  2                  -1  2    111360  ultralytics.nn.modules.block.C2f             [96, 96, 2, True]             
  3                  -1  1    166272  ultralytics.nn.modules.conv.Conv             [96, 192, 3, 2]               
  4                  -1  4    813312  ultralytics.nn.modules.block.C2f             [192, 192, 4, True]           
  5                  -1  1    664320  ultralytics.nn.modules.conv.Conv             [192, 384, 3, 2]              
  6                  -1  4   3248640  ultralytics.nn.modules.block.C2f             [384, 384, 4, True]           
  7                  -1  1   1991808  ultralytics.nn.modules.conv.Conv             [384, 576, 3, 2]              
  8                  -1  2   3985920  ultralytics.nn.modules.block.C2f             [576, 576, 2, True]           
  9                  -1  1    831168  ultralytics.nn.modules.block.SPPF            [576, 576, 5]                 
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 11             [-1, 6]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 12                  -1  2   1993728  ultralytics.nn.modules.block.C2f             [960, 384, 2]                 
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 14             [-1, 4]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 15                  -1  2    517632  ultralytics.nn.modules.block.C2f             [576, 192, 2]                 
 16                  -1  1    332160  ultralytics.nn.modules.conv.Conv             [192, 192, 3, 2]              
 17            [-1, 12]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 18                  -1  2   1846272  ultralytics.nn.modules.block.C2f             [576, 384, 2]                 
 19                  -1  1   1327872  ultralytics.nn.modules.conv.Conv             [384, 384, 3, 2]              
 20             [-1, 9]  1         0  ultralytics.nn.modules.conv.Concat           [1]                           
 21                  -1  2   4207104  ultralytics.nn.modules.block.C2f             [960, 576, 2]                 
 22        [15, 18, 21]  1   3780328  ultralytics.nn.modules.head.Detect           [8, [192, 384, 576]]          
Model summary: 295 layers, 25860952 parameters, 25860936 gradients, 79.1 GFLOPs

Transferred 469/475 items from pretrained weights
TensorBoard: Start with 'tensorboard --logdir koster_seafloor_obs/KSO_Paulina_oldmodel_2', view at http://localhost:6006/
Freezing layer 'model.22.dfl.conv.weight'
AMP: running Automatic Mixed Precision (AMP) checks with YOLOv8n...
/opt/conda/envs/myenv/lib/python3.8/site-packages/ultralytics/nn/tasks.py:567: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  return torch.load(file, map_location='cpu'), file  # load
YOLOv8n summary (fused): 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs
/opt/conda/envs/myenv/lib/python3.8/site-packages/ultralytics/utils/checks.py:558: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(True):
AMP: checks passed ✅
/opt/conda/envs/myenv/lib/python3.8/site-packages/ultralytics/engine/trainer.py:239: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  self.scaler = amp.GradScaler(enabled=self.amp)
train: Scanning /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/labels.cache... 129 images, 0 backgrounds, 129 corrupt: 100%|██████████| 129/129 [00:00<?, ?it/s]
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/01211c55-4dd6-4453-981a-2bc85bbbbb18.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0400de70-02e4-4f44-9dd1-f0a1c654a328.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/04016f4b-c2e2-465e-accd-8fc2db8f624a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/048bc083-6d15-4105-b83b-e95bf810d3c7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/059fbf23-5a21-4bc5-9456-9f7efc400932.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0637fb53-8ad4-4188-8124-875b3741939a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/06873e3c-1cdf-4468-a420-cd4bc958af6c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0727ebd2-72d9-4727-9a7e-c9b0434ac3a1.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0b6c1fb5-31f7-4fcb-87fb-db7b7631607a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0bdc28b1-aa4d-48ff-9ed2-2f4518c746f6.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0c2ba32b-9bda-43d4-9dce-22183a7060f4.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/0e1b964b-a1de-4ebc-8d64-be220aee85d3.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/13a9539a-f53f-4809-a623-5db747b76485.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/15181051-477d-42ee-8901-c27954e799cd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/19516876-d94b-43e2-b12f-23129cdcba5f.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/1d0acf63-30ab-4bb1-a603-23dcc48e3003.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/1ea603fe-aab4-4c0b-aea1-1b27623dd653.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/23be80d8-307a-4906-9ca2-ecb1b22f92d8.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/27d2cd3d-daf4-4b74-90cb-a73b84e13404.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/29832828-e270-4f98-9b98-d239147c903d.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/2adaaadf-11c9-4b49-a741-5612cac31889.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/2b0abc51-33f9-4ec2-8970-75576deada25.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/2e0b0a6f-7554-4aef-bbdd-0f0b20463fdc.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/2fe79d9e-dd94-4f03-9f64-df136a318303.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/30836d9e-4f12-41d7-85d4-458d38323fc5.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/31fc17b2-347e-4681-bc63-5385102905a7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/353143aa-c89d-4ec7-ab43-ae7779ed2785.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/36872e87-858d-4859-9324-277021c44cc0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/39d69402-471c-4953-9531-c9169bd7f4ea.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/3c777de5-617e-4627-bd35-212233828b17.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/3c9b97a4-0125-4c12-bb96-45dd3b6e523b.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/3d303da6-bc75-4805-9265-168862aa6b62.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/41431376-8845-444d-a0bd-b0d0d9c65199.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/4a75bf9d-2e5e-4051-a670-35644e0e7d9d.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/4c4a531d-4a53-4b6b-926f-7d7f20d4d768.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/4f250bd2-95a0-4639-a00f-5f1365c0303e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/4fdb58f6-765f-43c8-9adc-72526985b9ee.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/520f17c1-8a22-460e-88d5-e583ec19ee55.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/535e679b-c3fa-4802-926c-58c109affef2.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/54f55aed-2954-4a00-aa0e-22d1f30fb336.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/593d8971-5158-4509-8f1f-cb6647ae923c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/5d574ec6-e08c-47e7-b516-f197bf658bee.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/6036f363-2d21-42e0-b4d7-3c807bbaf491.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/6215867d-654b-491d-8798-01d0d1740077.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/6269399d-e6fe-465e-bac7-fbd39868ef9f.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/64a5680a-b0cc-4ffb-8017-569e33da441e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/6c4a4912-da9f-4c32-9728-b6c515f242d7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/70a2b982-b455-4e55-9d05-d6e0b08ee686.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/71051f70-2e82-407d-95f1-1c00804a797a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/72842bf5-92d2-4c79-8c67-1b5d5c968fe4.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/73a5f098-a742-45cf-98a7-66a62167d2d0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/74eff295-7ae5-4056-b139-ffc9738bef90.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/75edb4dc-d804-4eea-a079-20e1f5718d61.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/7c85f00c-9c10-4aad-82d5-5be915463cb0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/7e1697cd-a490-401f-b7dc-97c807ce7aee.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/800f2b27-8be5-4d0a-aa37-32bf981fa475.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/82efb338-f70a-4df7-a0bb-70050f421c31.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/85ab9efc-51c1-4da5-9520-79eafd5a98f2.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/86fbbf12-0f59-4733-97dd-64c927108319.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/87de2613-bcc5-4a3a-b954-661895eec80d.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8829c67d-ebae-4c1b-91f9-449e0862e319.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/885805c3-e559-4003-a3a8-6cb433d21496.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8955a583-4ee6-41a5-a957-fc34f843d6cf.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/89b89a28-f645-451f-93cb-7cd25f372fea.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8ad5146a-ff5b-44d5-96fd-d7903ec10cbf.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8b4d9e1f-1f00-4a95-8025-1f00596b1392.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8baf41df-67f9-439d-b4d3-cb9485eb320a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/8c35cec3-9d63-49d6-b869-5f5e3895caa7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/928449a4-5b9a-418f-aac4-89b1cc3d5e40.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/92a5bd3e-30b4-4b41-998a-e56d87c4f94c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/92ca90f3-1c20-4cfd-af82-c616f5971fea.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/9425fac5-f192-4fe4-b4df-3121b413d8a0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/98e208d5-1bf3-4858-8f6b-a8c9169a72a0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/98f7395c-2e78-4471-95e4-52fc803fe003.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/9d1c17db-4a36-43ab-ba55-e4038d8945fd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/9d657191-dc44-40b6-be76-7efad95b0ddd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/a1f2df08-61d7-46c0-afc6-f48bb99d5126.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/a65c0e61-bfbd-4289-8354-1dc6a140a572.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/a6aaef68-83fb-4609-bd73-2ec908eb9dcd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/a7788144-0d71-4f86-8c8e-7048f8be018b.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/aacab416-b803-44d0-8767-f7f78effaaec.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ab21420b-b2ae-437e-b61a-85d3b0025639.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/abd006b4-6cfc-4dd3-af76-10e5256d431e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/acbec617-7359-43fc-af3b-53c95d817929.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ad9229f8-0ae8-4400-94b5-249249f47cdb.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/b273b6f0-85dc-4587-ae6b-cee1f58b8fcd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/b51b4e6d-b38e-4892-a23d-93c3ea93861d.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/b5255532-3976-4484-8b3a-1e1657dcc80c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/b60a7359-e1ad-40fe-b609-3a2ea31a1351.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/bc3ac1dc-124d-4cfa-9e62-1cbe52bddab5.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/c422b70a-51c4-4f63-8b29-374bf4434871.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/c52d8c64-e7a1-43ae-966b-889d3b41f243.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/c6e7d8e4-a018-41f7-9eae-dbb12613099e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/c8b82793-5a0c-4fd9-9a9c-23952279f9d7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ca2399b8-946c-4a5c-b706-8f8ae11915d1.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/cb9ef33f-2e14-4105-9982-a9db9e7d6d75.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/cbf3257f-b5b5-40de-bb60-6d13f24ababd.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ccc06186-57f3-4380-bf3b-db259ded1402.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ce2926dc-a549-4fdb-ab40-9bca29a1fd8b.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ce3ad9d6-bada-404b-b2a4-df4b2005ea7f.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/cea07903-89b3-4f2b-a7b7-047588580c21.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d2d4632e-a0c2-4d5a-90ff-54d936d32e3c.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d31e2ebf-122e-4675-9fe0-d56da54a12bc.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d34eaad8-eb92-4371-a922-07455329bfaf.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d5792a2a-00ba-4068-94a4-952c41e12129.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/d7f37c06-02cd-4e58-9ccd-896c58806da0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/dbe19372-8b54-41f5-b187-ab6ef2bb727e.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/dc184311-de22-46fb-bdc1-0fc779a1cbbe.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/dcf3d6d0-a71c-4211-aa33-edc51aa7ddf6.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/dd3cb739-7e74-4191-b199-cc0d6bbe7769.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e0e11c98-f320-44df-881a-58fc64c7a01a.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e23ea91e-ea34-454a-86d0-03ca78f99cad.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e431c828-10ce-418e-a548-ae219f5716b5.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e4f92f2b-78f6-443a-bf95-c56c2f926150.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e5435fc8-e49b-47fc-a118-ebac9367b745.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e65150fc-1778-4472-878b-73a8b95ea9f7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e6f33643-0f80-4463-9061-f772d1ef76d7.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/e8547cdf-9a4f-4f5b-a288-93a633d81f36.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ea0204f2-2683-4bd4-96d8-6629ca55b663.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ea8d038b-642a-4768-b5f2-11e3a14e7524.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/eb0fb5e3-eb39-4bfd-9115-9dd1c780c8ad.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/ee1ac174-bf1a-4c73-8461-9d2eed31b040.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/f471dccd-04b3-4174-9f88-3f6be8d16d23.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/f894a137-cc72-486f-b998-8d85065a3433.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fa0e4b5a-1f07-4c4c-93ef-c75724dfbe10.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fac84af3-4e73-449b-a27e-f8b4c2893b94.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fba49ae1-bf11-4836-b28d-2e33c0408c02.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fc1939bd-8361-4bba-8aea-89fd83b03d58.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/images/fdfad66b-81fc-4b3e-9dad-9909082e523f.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates []
WARNING ⚠️ No images found in /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulina_frames/labels.cache, training may not work correctly. See https://docs.ultralytics.com/datasets/detect for dataset formatting guidance.
INFO:root:Training failed due to: not enough values to unpack (expected 3, got 0)
View run KSO_Paulina_oldmodel_1 at: https://wandb.ai/koster/koster_seafloor_obs/runs/qaowczru
View project at: https://wandb.ai/koster/koster_seafloor_obs
Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: /mimer/NOBACKUP/groups/snic2021-6-9/wandb/run-20241101_111940-qaowczru/logs

Additional context

Add any other context about the problem here.

Diewertje11 commented 2 weeks ago

Is my following understanding correct? I think that you are trying to retrain a model with Paulina's data, by taking an already existing baseline model and now retraining it. Is that correct?

From the following message: _train: WARNING ⚠️ /mimer/NOBACKUP/groups/snic2021-6-9/tmp_dir/KSO_Paulinaframes/images/01211c55-4dd6-4453-981a-2bc85bbbbb18.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates [] I understand it as that the data from Paulina does not fit the model. It says that the data from Paulina is 'non-normalized or out of bounds'. So I guess (but not sure) that that means that the values for every pixel in the image does not match the model. For example, that the model was trained on data that have pixel values from 1-255 and that Paulina's data goes from 0-1 for example. Or something like that. But I am not sure about this as I am not that familiar with the training process and the specific model. Does this ring any bells and that you know how to trouble shoot? Or do you want me to further investigate?

Bergylta commented 2 weeks ago

@Diewertje11 In this case, we are taking a baseline model (yolov8 baseline segmentation) and using it to train on paulinas data that she has annotated. I think you are correct in that the pixels might not match, there was something that happened when zooniverse did a update a while back, and that our bounding boxes were not centred above the object, but instead was a bit off, which might be related to this (also unsure). But this is a new issue that i haven't seen before i think, nor do i know how to troubleshoot it sadly. We can put this at prio 2-3