Different Metrics

[sinagh72]

What is your question?

I recently worked on training a ResNet-18 model for binary image classification using PyTorch lightning with the FedAvg strategy simulation. I had 3 clients and rounds = 10. However, I noticed that the results (AUC) varied across different runs, despite setting the torch seed, PyTorch lightning seed, and numpy seed: torch.backends.cudnn.deterministic = True torch.backends.cudnn.benchmark = False torch.manual_seed(42) pl.seed_everything(42) np.random.seed(seed=42)

Is there anything else I need to set to get the same AUC each time on my test set?

[adam-narozniak]

Hi, could you set the seed for the Python random library as below:

import random

random.seed(10)

The only part of Flower that I can think of that introduces randomness is sampling in client_manager (by default a SimpleClientManager is created). https://github.com/adap/flower/blob/60a7230ab3f60b1ba56b5232718472e44c97037e/src/py/flwr/server/client_manager.py#L170-L199 Line 198 exactly: https://github.com/adap/flower/blob/60a7230ab3f60b1ba56b5232718472e44c97037e/src/py/flwr/server/client_manager.py#L198

Let me know if that help.

[sinagh72]

It helped somehow. By adding random.seed, the AUCs I'm getting are closer after each run compared to before but not identical. Thank you

[adam-narozniak]

Are you running a Python script or Jupyter Notebook? (In Jupyter Notebook, setting seed before each new run will be necessary if you run one after the other).

[sinagh72]

I am running a Python script. Just to be clear, I call the following function before the simulation start (inside the server) and inside the client_fn

def set_seed(seed=10): 
  pl.seed_everything(seed) 
  np.random.seed(seed=seed) 
  torch.cuda.manual_seed(seed) 
  torch.cuda.manual_seed_all(seed) 
  random.seed(seed) 
  torch.backends.cudnn.deterministic = True 
  torch.backends.cudnn.benchmark = False 
  os.environ['PYTHONHASHSEED'] = str(seed)