Memory leak when loading and unloading multiple graphs

uziela commented 5 years ago

System information

What is the top-level directory of the model you are using: Not applicable
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04.2 LTS
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Not applicable
TensorFlow installed from (source or binary): I used tensorflow/tensorflow:1.14.0-gpu-py3 docker image (tested also on different version both with and without docker)
TensorFlow version (use command below): v1.14.0-rc1-22-gaf24dc91b5 1.14.0
Python version: Python 3.6.8
Bazel version (if compiling from source): Not applicable
GCC/Compiler version (if compiling from source): Not applicable
CUDA/cuDNN version: Cuda 10.0, cuDNN 7
GPU model and memory: GeForce GTX TITAN X (tested also on GeForce GTX 1080)
Exact command to reproduce: See "Code to reproduce the issue" section

Describe the current behavior

When I load multiple Inception V3 graphs into memory and afterwards unload all of them I get a memory leak. With 20 Inception graphs loaded and unloaded, the RAM usage goes up to around 2GB. With 100 Inception graphs, the RAM usage goes up to around 10GB. If I load and unload Inception V3 graphs one by one, there is no memory leak, RAM usage stays below 1GB, does not matter how many Inception V3 graphs I load.

I tried to load this Inception V3 graph multiple times for purposes of testing: http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz . In reality, I need to load different Inception V3 graphs into memory.

Describe the expected behavior

The expected behaviour is that I am able to load multiple Inception V3 graphs and afterwards unload all of them without any memory leak.

Code to reproduce the issue

import sys 
import os
import tensorflow as tf
import gc
import time

class InceptionV3Graph:
    def __init__(self, graph_path):
        with tf.gfile.FastGFile(graph_path, 'rb') as f:
            graph_def = tf.GraphDef()
            graph_def.ParseFromString(f.read())
            _ = tf.import_graph_def(graph_def, name='')
        self.sess = tf.Session(graph=tf.get_default_graph())

    def close(self):
        tf.reset_default_graph()
        gc.collect()
        self.sess.close()

if __name__ == "__main__":
    graph_path = "/path/to/classify_image_graph_def.pb" # you can get classify_image_graph_def.pb from http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz
    N_GRAPHS = 100 
    graphs = dict()
    for i in range(N_GRAPHS):
        print("Loading graph {}".format(i+1))
        graphs[i] = InceptionV3Graph(graph_path)
        # If you uncomment these two lines below there won't be any the memory leak
        #graphs[i].close()
        #del graphs[i]
    for i in range(N_GRAPHS):
        print("Unloading graph {}".format(i+1))
        if i in graphs:
            graphs[i].close()
            del graphs[i]
    print(graphs)
    gc.collect()
    print("All graphs unloaded")
    time.sleep(120)
    print("Quitting...")

Other info / logs I successfully reproduced the problem on two different Linux machines (Ubuntu 16.04 and 18.04) with and without docker. However, the memory leak does not seem to reproduce on Windows 10.

tensorflowbutler commented 5 years ago

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks. What is the top-level directory of the model you are using Exact command to reproduce

uziela commented 5 years ago

I have updated my post with the required information.

tensorflow / models

Memory leak when loading and unloading multiple graphs #7600