Build model using fasterrcnn_mobilenetv3_large_fpn

[WawanFirgiawan]

I want to run my training process with the command:

!python train.py --data data_configs/data_training.yaml --epochs 40 --model fasterrcnn_mobilenetv3_large_fpn --project-dir fasterrcnn_mobilenetv3_large_fpn --seed 8

and I get an error in my program as follows:

2024-06-24 15:23:20.794655: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-06-24 15:23:20.794717: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-06-24 15:23:20.796062: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-06-24 15:23:20.803158: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-06-24 15:23:21.919523: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Not using distributed mode wandb: Currently logged in as: pusatstudiaiunsulbar (pusatsudiaiusb). Use wandb login --relogin to force relogin wandb: Tracking run with wandb version 0.17.2 wandb: Run data is saved locally in /content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/wandb/run-20240624_152326-bw79izjd wandb: Run wandb offline to turn off syncing. wandb: Syncing run expert-fire-4 wandb: ⭐️ View project at https://wandb.ai/pusatsudiaiusb/fasterrcnn-pytorch-training-pipeline wandb: 🚀 View run at https://wandb.ai/pusatsudiaiusb/fasterrcnn-pytorch-training-pipeline/runs/bw79izjd device cuda Checking Labels and images... 100% 886/886 [00:00<00:00, 116878.55it/s] Checking Labels and images... 0it [00:00, ?it/s] Creating data loaders /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:558: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. warnings.warn(_create_warning_msg( Number of training samples: 886 Number of validation samples: 0

Building model from scratch...

Layer (type (var_name)) Input Shape Output Shape Param #

FasterRCNN (FasterRCNN) [4, 3, 640, 640] [0, 4] -- ├─GeneralizedRCNNTransform (transform) [4, 3, 640, 640] [4, 3, 640, 640] -- ├─BackboneWithFPN (backbone) [4, 3, 640, 640] [4, 256, 10, 10] -- │ └─IntermediateLayerGetter (body) [4, 3, 640, 640] [4, 960, 20, 20] -- │ │ └─Conv2dNormActivation (0) [4, 3, 640, 640] [4, 16, 320, 320] (432) │ │ └─InvertedResidual (1) [4, 16, 320, 320] [4, 16, 320, 320] (400) │ │ └─InvertedResidual (2) [4, 16, 320, 320] [4, 24, 160, 160] (3,136) │ │ └─InvertedResidual (3) [4, 24, 160, 160] [4, 24, 160, 160] (4,104) │ │ └─InvertedResidual (4) [4, 24, 160, 160] [4, 40, 80, 80] (9,960) │ │ └─InvertedResidual (5) [4, 40, 80, 80] [4, 40, 80, 80] (20,432) │ │ └─InvertedResidual (6) [4, 40, 80, 80] [4, 40, 80, 80] (20,432) │ │ └─InvertedResidual (7) [4, 40, 80, 80] [4, 80, 40, 40] 30,960 │ │ └─InvertedResidual (8) [4, 80, 40, 40] [4, 80, 40, 40] 33,800 │ │ └─InvertedResidual (9) [4, 80, 40, 40] [4, 80, 40, 40] 31,096 │ │ └─InvertedResidual (10) [4, 80, 40, 40] [4, 80, 40, 40] 31,096 │ │ └─InvertedResidual (11) [4, 80, 40, 40] [4, 112, 40, 40] 212,280 │ │ └─InvertedResidual (12) [4, 112, 40, 40] [4, 112, 40, 40] 383,208 │ │ └─InvertedResidual (13) [4, 112, 40, 40] [4, 160, 20, 20] 426,216 │ │ └─InvertedResidual (14) [4, 160, 20, 20] [4, 160, 20, 20] 793,200 │ │ └─InvertedResidual (15) [4, 160, 20, 20] [4, 160, 20, 20] 793,200 │ │ └─Conv2dNormActivation (16) [4, 160, 20, 20] [4, 960, 20, 20] 153,600 │ └─FeaturePyramidNetwork (fpn) [4, 160, 20, 20] [4, 256, 10, 10] -- │ │ └─ModuleList (inner_blocks) -- -- (recursive) │ │ └─ModuleList (layer_blocks) -- -- (recursive) │ │ └─ModuleList (inner_blocks) -- -- (recursive) │ │ └─ModuleList (layer_blocks) -- -- (recursive) │ │ └─LastLevelMaxPool (extra_blocks) [4, 256, 20, 20] [4, 256, 20, 20] -- ├─RegionProposalNetwork (rpn) [4, 3, 640, 640] [0, 4] -- │ └─RPNHead (head) [4, 256, 20, 20] [4, 15, 20, 20] -- │ │ └─Sequential (conv) [4, 256, 20, 20] [4, 256, 20, 20] 590,080 │ │ └─Conv2d (cls_logits) [4, 256, 20, 20] [4, 15, 20, 20] 3,855 │ │ └─Conv2d (bbox_pred) [4, 256, 20, 20] [4, 60, 20, 20] 15,420 │ │ └─Sequential (conv) [4, 256, 20, 20] [4, 256, 20, 20] (recursive) │ │ └─Conv2d (cls_logits) [4, 256, 20, 20] [4, 15, 20, 20] (recursive) │ │ └─Conv2d (bbox_pred) [4, 256, 20, 20] [4, 60, 20, 20] (recursive) │ │ └─Sequential (conv) [4, 256, 10, 10] [4, 256, 10, 10] (recursive) │ │ └─Conv2d (cls_logits) [4, 256, 10, 10] [4, 15, 10, 10] (recursive) │ │ └─Conv2d (bbox_pred) [4, 256, 10, 10] [4, 60, 10, 10] (recursive) │ └─AnchorGenerator (anchor_generator) [4, 3, 640, 640] [13500, 4] -- ├─RoIHeads (roi_heads) [4, 256, 20, 20] [0, 4] -- │ └─MultiScaleRoIAlign (box_roi_pool) [4, 256, 20, 20] [0, 256, 7, 7] -- │ └─TwoMLPHead (box_head) [0, 256, 7, 7] [0, 1024] -- │ │ └─Linear (fc6) [0, 12544] [0, 1024] 12,846,080 │ │ └─Linear (fc7) [0, 1024] [0, 1024] 1,049,600 │ └─FastRCNNPredictor (box_predictor) [0, 1024] [0, 3] -- │ │ └─Linear (cls_score) [0, 1024] [0, 3] 3,075 │ │ └─Linear (bbox_pred) [0, 1024] [0, 12] 12,300

Total params: 18,935,354 Trainable params: 18,876,458 Non-trainable params: 58,896 Total mult-adds (G): 11.49

Input size (MB): 19.66 Forward/backward pass size (MB): 1172.14 Params size (MB): 75.74 Estimated Total Size (MB): 1267.54

18,935,354 total parameters. 18,876,458 training parameters. /usr/lib/python3.10/multiprocessing/popen_fork.py:66: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock. self.pid = os.fork() Epoch: [0] [ 0/222] eta: 0:11:02 lr: 0.000006 loss: 1.8196 (1.8196) loss_classifier: 1.4352 (1.4352) loss_box_reg: 0.3557 (0.3557) loss_objectness: 0.0227 (0.0227) loss_rpn_box_reg: 0.0060 (0.0060) time: 2.9830 data: 1.9134 max mem: 704 Epoch: [0] [100/222] eta: 0:00:22 lr: 0.000458 loss: 1.2597 (1.3672) loss_classifier: 0.5182 (0.6553) loss_box_reg: 0.7019 (0.6994) loss_objectness: 0.0014 (0.0098) loss_rpn_box_reg: 0.0025 (0.0027) time: 0.1611 data: 0.0257 max mem: 811 Epoch: [0] [200/222] eta: 0:00:03 lr: 0.000910 loss: 0.8597 (1.1901) loss_classifier: 0.2865 (0.5291) loss_box_reg: 0.5280 (0.6531) loss_objectness: 0.0006 (0.0057) loss_rpn_box_reg: 0.0013 (0.0023) time: 0.1735 data: 0.0235 max mem: 811 Epoch: [0] [221/222] eta: 0:00:00 lr: 0.001000 loss: 0.8436 (1.1645) loss_classifier: 0.3099 (0.5145) loss_box_reg: 0.5193 (0.6426) loss_objectness: 0.0005 (0.0053) loss_rpn_box_reg: 0.0012 (0.0022) time: 0.1591 data: 0.0203 max mem: 811 Epoch: [0] Total time: 0:00:34 (0.1552 s / it) creating index... index created! Traceback (most recent call last): File "/content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/train.py", line 571, in main(args) File "/content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/train.py", line 423, in main stats, val_pred_image = evaluate( File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/torch_utils/engine.py", line 136, in evaluate for images, targets in metric_logger.log_every(data_loader, 100, header): File "/content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/torch_utils/utils.py", line 202, in log_every log(f"{header} Total time: {total_time_str} ({total_time / len(iterable):.4f} s / it)") ZeroDivisionError: float division by zero Traceback (most recent call last): File "/content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/train.py", line 571, in main(args) File "/content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/train.py", line 423, in main stats, val_pred_image = evaluate( File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/torch_utils/engine.py", line 136, in evaluate for images, targets in metric_logger.log_every(data_loader, 100, header): File "/content/drive/MyDrive/Program/CupangDetection/fasterrcnn-pytorch-training-pipeline/torch_utils/utils.py", line 202, in log_every log(f"{header} Total time: {total_time_str} ({total_time / len(iterable):.4f} s / it)") ZeroDivisionError: float division by zero** wandb: 🚀 View run expert-fire-4 at: https://wandb.ai/pusatsudiaiusb/fasterrcnn-pytorch-training-pipeline/runs/bw79izjd wandb: ⭐️ View project at: https://wandb.ai/pusatsudiaiusb/fasterrcnn-pytorch-training-pipeline wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240624_152326-bw79izjd/logs wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with wandb.require("core")! See https://wandb.me/wandb-core for more information.

[maexrakete]

Number of validation samples: 0

looks like you lack validation data

fyi: if you wrap your error log into triple backticks it would be way more readable.