Hyper parameter selection for practical applications

[Charles-333]

I noticed that the method in the paper relies heavily on hyperparameter tuning. However, since the target domain lacks labels, tuning ultimately relies on validation set performance for optimal results, which is then reported. Does this approach ensure a fair comparison, especially given the variability that some methods show over epochs?

Additionally, a recent paper [1] raised a similar concern. It evaluates by plotting performance curves and calculating stability metrics, which might be relevant. Looking forward to your response!

[1] Haitao Mao, Lun Du, Yujia Zheng, Qiang Fu, Zelin Li, Xu Chen, Shi Han, and Dongmei Zhang. 2024. Source Free Graph Unsupervised Domain Adaptation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM ’24). Association for Computing Machinery, New York, NY, USA, 520–528. https://doi.org/10.1145/3616855.3635802

[cszhangzhen]

Thanks for your interest in this work.

You're correct. The performance of models that utilize adversarial training can indeed fluctuate across different epochs.

I think it is reasonable to use the validation set or a fixed number of epochs, provided that the models have converged.

The reference you mentioned is also a valid option. Alternatively, we could consider using the average performance from the last several epochs.

Thanks!