The model is automatically killing itself after 3 epochs

[vdsprakash]

I am getting the Total Loss of about 10.032443234 starting , and after 3 epochs it is automatically killed.

Systerm Infomation:-

Processor:- Intel XEON RAM:- 8 GB Graphics: NVIDIA GTX-1060 6GB Python 3.6

[robertlugg]

The training will end when it can no longer improve. Please include the message (4-5 lines) towards the end of your output. It will give a clue as to the problem. But bottom line is that training has failed.

[danFromTelAviv]

I also get something similar - I am using python 3.6 , tensorflow + keras ( almost latest versions of each ). I am trying to train with squeezenet with the raccon dataset (I split it so that image 1-40 is my validation set). I had to change the congif file abit for it to run :

{
    "model" : {
        "backend":              "SqueezeNet",
        "input_size":           416,
        "anchors":              [0.57273, 0.677385, 1.87446, 2.06253, 3.33843, 5.47434, 7.88282, 3.52778, 9.77052, 9.16828],
        "max_box_per_image":    10,
        "labels":               ["raccoon"]
    },

    "train": {
        "train_image_folder":   "path/to/raccoon_dataset-master/raccoon_dataset-master/images/train/",
        "train_annot_folder":   "path/to/raccoon_dataset-master/raccoon_dataset-master/annotations/train/",

        "train_times":          10,
        "pretrained_weights":   "",
        "batch_size":           16,
        "learning_rate":        1e-4,
        "nb_epochs":             50,
        "warmup_epochs":        3,
        "object_scale":         5.0 ,
        "no_object_scale":      1.0,
        "coord_scale":          1.0,
        "class_scale":          1.0,
        "saved_weights_name":   "racoon_detector.h5",

        "debug":                true
    },

    "valid": {
        "valid_image_folder":   "path/to/raccoon_dataset-master/raccoon_dataset-master/images/val/",
        "valid_annot_folder":   "path/to/raccoon_dataset-master/raccoon_dataset-master/annotations/val/",

        "valid_times":          1
    }
}

1) The recall only decreases during the first 3 (warmup) iterations. typical print out for loss at this stage:

Loss XY [2.52920581e-05] Loss WH [0.0361119919] Loss Conf [0.12189471] Loss Class [0] Total Loss [10.1580324]

I am not sure how it gets to total loss of 10 ? 2) on the first real epoch I there are warning fro almost all batches and finally a failure :

if all(x == 1 for x in tup) and isinstance(A, _nx.ndarray): SystemError: error return without exception set Exception ignored in: <generator object tile.. at 0x000001B577C54E60> Traceback (most recent call last): File "C:\Users\dan\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\lib\shape_base.py", line 1140, in if all(x == 1 for x in tup) and isinstance(A, _nx.ndarray): SystemError: error return without exception set raccoon 0.0000 mAP: 0.0000 Exception ignored in: <async_generator object _ag at 0x000001B39081CAD8> Traceback (most recent call last): File "C:\Users\dan\AppData\Local\Programs\Python\Python36\lib\types.py", line 27, in _ag

SystemError: error return without exception set raccoon 0.0000 mAP: 0.0000 Exception ignored in: <async_generator object _ag at 0x000001B39081CAD8> Traceback (most recent call last): File "C:\Users\dan\AppData\Local\Programs\Python\Python36\lib\types.py", line 27, in _ag SystemError: error return without exception set

Process finished with exit code -1

[akhil451]

I am also getting an error along the same lines. Please share if anyone has figured out a solution to this. my config file:

{
    "model" : {
        "backend":              "Tiny Yolo",
        "input_size":           416,
        "anchors":              [0.38,0.29, 0.54,0.47, 0.78,0.58, 0.97,0.85, 1.94,1.83],
        "max_box_per_image":    10,        
        "labels":               ["Plate"]
    },

    "train": {
        "train_image_folder":   "train_img/",
        "train_annot_folder":   "annotations/",     

        "train_times":          8,
        "pretrained_weights":   "",
        "batch_size":           16,
        "learning_rate":        1e-4,
        "nb_epochs":            500,
        "warmup_epochs":        3,

        "object_scale":         5.0 ,
        "no_object_scale":      1.0,
        "coord_scale":          1.0,
        "class_scale":          1.0,

        "saved_weights_name":   "weights.h5",
        "debug":                true
    },

    "valid": {
        "valid_image_folder":   "",
        "valid_annot_folder":   "",

        "valid_times":          1
    }
}

the error:

Epoch 00001: val_loss improved from inf to 10.00044, saving model to weights.h5
Traceback (most recent call last):
  File "train.py", line 101, in <module>
    _main_(args)
  File "train.py", line 97, in _main_
    debug              = config['train']['debug'])
  File "/home/sasuke/Downloads/keras-yolo2-master/frontend.py", line 401, in train
    max_queue_size   = 8)      
  File "/home/sasuke/anaconda3/envs/yolov2/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/sasuke/anaconda3/envs/yolov2/lib/python3.6/site-packages/keras/engine/training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "/home/sasuke/anaconda3/envs/yolov2/lib/python3.6/site-packages/keras/engine/training_generator.py", line 251, in fit_generator
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/home/sasuke/anaconda3/envs/yolov2/lib/python3.6/site-packages/keras/callbacks.py", line 79, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/home/sasuke/Downloads/keras-yolo2-master/frontend.py", line 36, in on_epoch_end
    clear_output(wait=True)
NameError: name 'clear_output' is not defined
terminate called without an active exception
terminate called recursively
Aborted (core dumped)

[akhil451]

Seems like the early stopping was the cause, you can either increase the value of patience(number of epochs) or remove that argument altogether.