DANN for Regression

[Farahani1]

Hi, I have changed this project in order to try domain adversarial learning in a regression problem. actually, I replaced the label predictor with a regressor. But it just makes the regression accuracy worse. Do you have any idea why it happens?

[jaehongyoon]

Well it’s not right to directly apply the loss functions of classification problem on regression problem as your not calculating the posterior of y in regression. Can you specify the loss you used?

Instead I would suggest using Monte Carlo to generate the entropy of y and use it as a loss. Try to sample y accordingly to the latent variables for the given X and generate empirical p(y) (I use KDE for its simplicity and robustness).

On Fri, Mar 8, 2019 at 10:08 AM Farahani1 notifications@github.com wrote:

Hi, I have changed this project in order to try domain adversarial learning in a regression problem. actually, I replaced the label classifier with a regressor. But it just makes the regression accuracy worse. Do you have any idea why it happens?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pumpikano/tf-dann/issues/30, or mute the thread https://github.com/notifications/unsubscribe-auth/AYM41-DXddJ2WTQgGrig0hOJjMmC8foQks5vUn0EgaJpZM4bliCM .

[Farahani1]

Well it’s not right to directly apply the loss functions of classification problem on regression problem as your not calculating the posterior of y in regression. Can you specify the loss you used? Instead I would suggest using Monte Carlo to generate the entropy of y and use it as a loss. Try to sample y accordingly to the latent variables for the given X and generate empirical p(y) (I use KDE for its simplicity and robustness). … On Fri, Mar 8, 2019 at 10:08 AM Farahani1 @.***> wrote: Hi, I have changed this project in order to try domain adversarial learning in a regression problem. actually, I replaced the label classifier with a regressor. But it just makes the regression accuracy worse. Do you have any idea why it happens? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#30>, or mute the thread https://github.com/notifications/unsubscribe-auth/AYM41-DXddJ2WTQgGrig0hOJjMmC8foQks5vUn0EgaJpZM4bliCM .

I used Mean Square Error of output(y) as the loss function for regression

[jaehongyoon]

For the domain adaptation, your loss should be more of a posterior. Therefore, using MSE is not ideal. Your mixing a squared loss with loss defined in form of divergence between representation of source and target.

I tried the approach and it worked for me in some cases. Basically what I did was normalize the data so that the two losses can be somewhat scaled similarly. Check the range of your MSE loss as it can be way off what you get from loss of domain predictor part.

Still, since your mixing up losses with different bases, it will have limits. Try to come up with the loss for y defined over the posterior.

On Fri, Mar 8, 2019 at 10:27 AM Farahani1 notifications@github.com wrote:

Well it’s not right to directly apply the loss functions of classification problem on regression problem as your not calculating the posterior of y in regression. Can you specify the loss you used? Instead I would suggest using Monte Carlo to generate the entropy of y and use it as a loss. Try to sample y accordingly to the latent variables for the given X and generate empirical p(y) (I use KDE for its simplicity and robustness). … <#m_-5902838529403855597m-4180455736861924419_> On Fri, Mar 8, 2019 at 10:08 AM Farahani1 @.***> wrote: Hi, I have changed this project in order to try domain adversarial learning in a regression problem. actually, I replaced the label classifier with a regressor. But it just makes the regression accuracy worse. Do you have any idea why it happens? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#30 https://github.com/pumpikano/tf-dann/issues/30>, or mute the thread https://github.com/notifications/unsubscribe-auth/AYM41-DXddJ2WTQgGrig0hOJjMmC8foQks5vUn0EgaJpZM4bliCM .

I used Mean Square Error of output(y) as the loss function for regression

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/pumpikano/tf-dann/issues/30#issuecomment-470967104, or mute the thread https://github.com/notifications/unsubscribe-auth/AYM417dyqPyYl020oyj3t7d04bqGP9urks5vUoFigaJpZM4bliCM .

[Farahani1]

I did not consider the point you mentioned, but now I almost got what you mean. I am gonna work on my project considering your guides. thanks for your help dear @jaehongyoon

[jaehongyoon]

Honored.

On Fri, Mar 8, 2019 at 11:27 AM Farahani1 notifications@github.com wrote:

I did not consider the point you mentioned, but now I almost got what you mean. I am gonna work on my project considering your guides. thanks for your help dear @jaehongyoon https://github.com/jaehongyoon

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/pumpikano/tf-dann/issues/30#issuecomment-470988799, or mute the thread https://github.com/notifications/unsubscribe-auth/AYM414UgRPMhHU-xi7umLj-1aiAGkYOIks5vUo9cgaJpZM4bliCM .

[Farahani1]

would you please explain more about your idea of Monte Carlo sampling and latent variables. Do you mean that I should use Expectation Maximization? What do you mean by posterior exactly? Do you mean conditional probability of y given x?

[jaehongyoon]

The basic idea is to employ variational inference technique to predict p(y|x). Since it’s hard to calculate it on dann architecture for regression problem, I would suggest to keep mean and std of weights as an variable and sample W as

W_i = mean_i + e * std_i

Where subscript i indicates the layer number and e is a random variable samples from N(0, 1).

At each inference point, you can sample multiples of weights using these means and stds and construct an empirical p(y|X). This posterior of y can be used to replace the p(y|X) calculated via crossentropy as in the original code. Hope this helps. I’m typing on my phone so please be aware of my crappy explanation.

On Sat, Mar 9, 2019 at 12:40 AM Farahani1 notifications@github.com wrote:

[image: Boxbe] https://www.boxbe.com/overview This message is eligible for Automatic Cleanup! (notifications@github.com) Add cleanup rule https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DtTMiNiS4YZtrEIYfZWp%252FmvTGZGQDRWfvMFQrpCHs7fo%253D%26token%3DK%252B%252FtXo2cPPNIRnzY4lx0TovT%252BudgS6f3jvkHTNMlguYusQfxsoYlosIMOh7iXQliK4IygLOmQoqsgWNosvm9vpcwEoErB6%252Fe1SJ67TQmuZxLmdmNH5DClkRDZuLdfxciY2LTzDgcp9Ovyv6MslOSZA%253D%253D&tc_serial=48597845167&tc_rand=1631818465&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001 | More info http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=48597845167&tc_rand=1631818465&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001

would you please explain more about your idea of Monte Carlo sampling and latent variables. Do you mean that I should use Expectation Maximization? What do you mean by posterior exactly? Do you mean conditional probability of y given x?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/pumpikano/tf-dann/issues/30#issuecomment-471147482, or mute the thread https://github.com/notifications/unsubscribe-auth/AYM41y-dHGmMi5dMO6Pxizz1-XnTygxQks5vU0YPgaJpZM4bliCM .