How to verify model

[wuboya0003]

I am wondering how could I get a more comprehensive evaluation of this model, should I make a five-fold cross validation or multiple repeated experiments, for example run 3 or 4 times CycleGAN. thanks for your answer.

[taesungp]

One challenge about evaluating CycleGAN (or image manipulation model in general) is that there is no absolute automated metric that we can rely on. There is FID that measures realism, but it has its own problems like validity of the metric, sensitivity toward the number of samples, and most importantly not measuring its alignment with the inputs. Therefore, it'd be difficult to do tricks like five-fold cross validation. Nevertheless, one reasonable thing to do would be doing multiple repeated experiments and measuring FIDs.

[wuboya0003]

Your answer helped me a lot. Now I am investigating on medical image denoising and SSIM and PSNR are usually utilized to evaluate the performance of model, if I want to evaluate the performance of my model(based on CycleGAN), does it mean that I have to do multiple repeated experiments with different train and test set.

[wuboya0003]

And if multiple repeated experiments are necessary, how many times should I do this experiment？

[junyanz]

some people did 5 runs.