Memory leaks in model.run_graph()

[ChunHsinWang]

Hello, I'm trying to extract the feature vectors of each ROI pooling feature maps, by running the image through model.run_graph literately. However, the consumed memory will keep increasing every iteration and then to 100%, then kills the program. My code is as follow:

class_fc_layer =  model.keras_model.get_layer("pool_squeeze").output
proposals_layer =  model.keras_model.get_layer("ROI").output
probs_layer =  model.keras_model.get_layer("mrcnn_class").output
deltas_layer =  model.keras_model.get_layer("mrcnn_bbox").output
for image_id in dataset.image_ids:
    image = dataset.load_image(image_id)
    mrcnn = model.run_graph([image], [
        ("class_fc", class_fc_layer),
        ("proposals", proposals_layer),
        ("probs", probs_layer),
        ("deltas", deltas_layer)])

I think the the line outputs_np = kf(model_in) in run_graph() is causing the leaks, and I have found people reporting this similar issues with Keras https://github.com/keras-team/keras/issues/8495. Any idea on how to fix this, or other recommended approach for extracting the feature vectors?

[canerozer]

I also encountered that while I was extracting the RoI probabilities and anchor boxes. I debugged a similar program to yours, and after processing around 10 thousand images with model.run_graph(), the program stopped processing since it took up most of my RAM and some small portion of my swap memory.

[mehmetnur123]

Hi @ChunHsinWang , I am facing the same problem at the moment. After about 8000 Images, I am receiving a Resorce Exhausted Error with a GTX 1080 Ti. Have you been able to solve the problem?