search

IBM / federated-learning-lib

A library for federated learning (a distributed machine learning process) in an enterprise environment.
Other
500 stars 136 forks source link

first party allocates all GPU for itself #135

Open omarelnakib opened 11 months ago

omarelnakib commented 11 months ago

Screenshot from 2023-11-29 14-26-08 Hi, when I try running using GPU, the first party allocates all the memory of the GPU, and when the other parties are created they throw exception out of memory as they cannot allocate anymore from the memory. I use: Tensorflow 1.15.0 keras 2.11.0 python 3.6 i tried adding this code in the keras_fl_model and they run successfully but no change happens, still allocates the whole memory

   config = tf.compat.v1.ConfigProto()
        config.gpu_options.allow_growth=True
        config.gpu_options.per_process_gpu_memory_fraction = 0.1
        self.sess = tf.compat.v1.Session(config=config)
        set_session(self.sess)