Pretrained model evaluation

[nkotelevskii]

First of all, thank you very much for the project! I think a lot of researchers/practitioners can benefit from it! It is a huge contribution to the area!

I have a question related to the evaluation of the pretrained model. I have trained an algorithm (e.g. FedRep) with standard hyperparams on cifar10 and saved the resulting model.

Next, following the instruction, I provided a path to the --external_model_params_file parameter (just hardcoded appropriate value to default path) and everything successfully loaded.

But for me, it is not clear how to evaluate this model.

I did the following:

from src.server.fedrep import FedRepServer

model: FedRepServer = FedRepServer() # path to the weights hardcoded

model.test()
out = model.test_results

print(out)

, and the results I have are way worse compared to the ones I have in cifar10_log.html file:

When training ended: {100: {'loss': '0.6021 -> 0.0000', 'accuracy': '74.69% -> 0.00%'}}

When I loaded: {1: {'loss': '98.6518 -> 0.0000', 'accuracy': '18.83% -> 0.00%'}}

I am pretty sure I can miss something. Can you elaborate on this, please?

And also a question more specific to FedRep. Shouldn't we have here unique_model=True, as each of the clients keeps the version of the head from the previous round?

Thank you!

[KarhouTam]

Hi, @nkotelevskii. Thanks for your appreciation first. Appreciation from users like you is my motivation for maintaining this project.

For your questions:

1. The FedRep method needs finetune on the client side to perform as it should. You can check the FedRep paper for more details. Thus, you can set --finetune_epoch greater than 0 (1 for example) to allow the pretrained model to perform finetuning before evaluating on testset. Also, you should set full_model = False in FedRepClient.fientune(), after that, FedRep should work as you expect.

2. In fact, I choose to save each client model's classifier layer to the 'FedRepClient' side. Because most of the parameters of each FedRep client are the same. FedRepClient is inherited from FedPerClient and I explicitly set the classifier parameters as personal at the latter:

https://github.com/KarhouTam/FL-bench/blob/0e94245bf4ff7d0b01754d378c957ae798820c71/src/client/fedrep.py#L4

https://github.com/KarhouTam/FL-bench/blob/0e94245bf4ff7d0b01754d378c957ae798820c71/src/client/fedper.py#L4-L14

The FedRepClient would save all parameters with name in self.personal_params_dict when it has done the local training:

https://github.com/KarhouTam/FL-bench/blob/0e94245bf4ff7d0b01754d378c957ae798820c71/src/client/fedavg.py#L121-L124

https://github.com/KarhouTam/FL-bench/blob/0e94245bf4ff7d0b01754d378c957ae798820c71/src/client/fedavg.py#L166-L173

I have evaluated for you:

Run python fedrep.py -d cifar10 -ge 100 --save_model 1

Results:

Run python fedrep.py --external_model_params_file ../../out/FedRep/cifar10_100_lenet5.pt -ge 1 -fe 5 -tg 1

Results:

[nkotelevskii]

@KarhouTam thank you for your quick and verbose response! Now I see, many thanks to you! This solves my issue.

Have a good weekend!