thuml / Deep-Embedded-Validation

Code release for Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation (ICML 2019)
MIT License
61 stars 10 forks source link

get_dev_risk returns NaN and very large negative values #4

Open trivecitoni opened 3 years ago

trivecitoni commented 3 years ago

There doesn't seem to be any bound on the value returned by get_dev_risk.

For example:

N = 1000
weight = np.ones((N,1))
error = np.random.rand(N,1)
get_dev_risk(weight, error) 
# outputs nan

I can obtain very large negative values as well:

N = 1000
np.random.seed(6382)
weight = np.random.normal(loc=0.5, scale=0.000001, size=(N,1))
error = np.random.rand(N,1)
get_dev_risk(weight, error)
# outputs -8220.64

Is this the expected behavior?

youkaichao commented 3 years ago

Hi, sorry for the late reply (I forgot to set the notification of this repo =_=|| ). The code assumes weight is a random variable with expectation of 1, and the variance of weight is used for the control variable to reduce the variance. You can check the paper for details.

In your first code, weight = np.ones((N,1)) results in NaN because the variance is 0, so this line would not work.

In your second code, weight = np.random.normal(loc=0.5, scale=0.000001, size=(N,1)) does not obey the requirement that its expectation should be 1.

Actually the weight should come from the dev.py:get_weight function and it is not arbitrary. Therefore we can guarantee meaningful results.