search

Somedaywilldo / BM-NAS

BM-NAS: Bilevel Multimodal Neural Architecture Search (AAAI 2022 Oral)
18 stars 6 forks source link

Different results with fixed seed #7

Open bezirganyan opened 1 year ago

bezirganyan commented 1 year ago

Dear BM-NAS authors,

In the code the seed is fixed with lines: 

    np.random.seed(args.seed)
    cudnn.benchmark = True
    torch.manual_seed(args.seed)
    cudnn.enabled=True
    torch.cuda.manual_seed(args.seed)

However, every time I run the code, I get different results. Particularly with MM-IMDB, with the same hyper-parameters and the same seed, I got 62.43%, 62.55% and 62.36% Weighted F-1 scores. I believe, that fixing the score must make sure that we will get the exact same results on each run, can you please let me know where else I shall fix the seed? Furthermore, in the the weighted F1 score on MM-IMDB is reported as 62.92% +- 0.03, can you please explain whether this range is because of the different results after fixing the seed?

Thanks in advance, Best regards, Grigor