training time does not reduces after increasing batch_size on 32 GB CPU instance

[suyogkute]

with batch_size = 1, ETA - 1 hour with batch_size = 16 ETA - 10 hour

this behaviour is same on 32 GB RAM CPU intsnace vs local 16 GB CPU system.

[ithllc]

@elbruno @mmphego @fmorenovr @prateekralhan

Any answer on this from the team would be highly appreciated. I am currently training this on A100's in Google Colab and I second the matter of discussion. I have 12 classes, including the background for image classification. The screenshot is below:

output from the train_model method is: Train 808 images Validate 350 images Applying augmentation on dataset Checkpoint Path: /content/drive/MyDrive/computer_vision_model Selecting layers to train

the code for training the model is just like what is discussed in the documentation, and I applied all the software downgrades, currently running it on tensorflow 2.8.0.

code below: from google.colab import drive drive.mount('/content/drive')

!pip3 uninstall tensorflow !pip3 install tensorflow==2.8.0 # current version 2.14.0, reinstall it after this !pip3 install tensorflow--gpu

!pip install requests numpy pillow scipy scikit-image==0.18.3 imgaug matplotlib labelme2coco==0.1.0 pixellib==0.5.2

import pixellib from pixellib.custom_train import instance_custom_training

import json import numpy as np import pandas as pd import os import tensorflow as tf print(tf.version)

train_maskrcnn = instance_custom_training() train_maskrcnn.modelConfig(network_backbone = 'resnet101', num_classes= 12, batch_size = 4) train_maskrcnn.load_pretrained_model(models_dir+'/mask_rcnn_coco.h5') train_maskrcnn.load_dataset(exports_dir) train_maskrcnn.train_model(num_epochs = 100, augmentation=True, path_trained_models = models_dir)