std::bad_alloc issue

[haramoz]

Hello,

I am filing this issue after being stagnated here for couple of weeks. My issue here is that my evaluation always fails with bad_alloc issue after few evaluations, After much googling this seems to be a memory related issue. I have tried the following things

1. decrease batchsize to 2 from 64 still fails
2. use of K.clear_session() at the start of my create_model or at the end, still bad_alloc,
3. I noticed the code fails after one evaluation is done and time to evaluate the result. So I decreased the evaluation batchsize to 4 from default 32, still fails Can not really delete the model inside create_model function as this is returned and needed for TPE.

So anyone here have any idea what is the issue here and how can it be fixed?

my code as follows(using hyperas on Densenet):

def create_model(X_train,y_train,X_val,y_val,X_test,y_test):
    K.clear_session()
    epochs = {{choice([5,7,10])}}
    es_patience = 3
    lr_patience = 2
    dropout = {{uniform(0.1,0.5)}}
    depth = {{choice([28,31,34,25])}}
    nb_dense_block = {{choice([3,4])}}
    nb_filter = 16
    growth_rate = {{choice([12,18,24,30])}}
    bn = True
    reduction_ = 0.5
    bs = 32
    lr = {{choice([2E-4,1E-4,5E-4])}}
    weight_file = 'keras_dn_wt_16Oct2200.h5'
    nb_classes = 1
    img_dim = (2,96,96)
    n_channels = 2
    print("------------------------ current config for the test -------------------------")
    print("Depth: ",depth," Growth_rate: ",growth_rate," lr: ",lr," nb_filter: ",nb_filter," dropout: ",dropout)
    print("Epochs ",epochs," batch_size: ",bs," es_patience: ",es_patience," lr_patience: ",lr_patience)
    print("dense_block ",nb_dense_block," reduction_: ",reduction_," bottleneck: ",bn)
    print("------------------------   end of configs        -------------------------")

    model  = DenseNet(depth=depth, nb_dense_block=nb_dense_block,
                 growth_rate=growth_rate, nb_filter=nb_filter,
                 dropout_rate=dropout,activation='sigmoid',
                 input_shape=img_dim,include_top=True,
                 bottleneck=bn,reduction=reduction_,
                 classes=nb_classes,pooling='avg',
                 weights=None)](url)
    model.summary()
    opt = Adam(lr=lr)
    model.compile(loss=binary_crossentropy, optimizer=opt, metrics=['accuracy'])

    es = EarlyStopping(monitor='val_loss', patience=es_patience,verbose=1)
    checkpointer = ModelCheckpoint(filepath=weight_file,verbose=1, save_best_only=True)

    lr_reducer = ReduceLROnPlateau(monitor='val_loss', factor=np.sqrt(0.1), cooldown=0, patience=lr_patience, min_lr=0.5e-6,verbose=1)

    model.fit(X_train,y_train,
          batch_size=bs,
          epochs=epochs,
          callbacks=[lr_reducer,es,checkpointer],
          validation_data=(X_val,y_val),
          verbose=2)

    score, acc = model.evaluate(X_test,y_test)
    print('current test accuracy:', acc)
    pred = model.predict(X_test)
    auc_score = roc_auc_score(y_test,pred)
    print("current test auc_score ------------------> ",auc_score)