Some Questions About DaNN

[sunshining-bit]

    When I tested the model with my own data, I found that no matter whether the training loss "loss" increases the MMD loss "loss_mmd", the migration result of the source domain to the target domain does not change. This shows that MMD is not helpful to the result of domain adaptation.
    What parameters can be debugged in the code to highlight the effect of MMD on domain adaptation?
    The accuracy rate of the data provided by me is relatively high. The accuracy rate of the source domain prediction can reach about 96%, and the target domain migration prediction result is about 94% regardless of whether the MMD regular term is added or not. Will the effect of MMD decay in the case of high accuracy?
    Thank you.

[jindongwang]

Please provide more details of the problem, e.g., screen shot and your dataset description. Normally you need to tune the hyperparameters.

[sunshining-bit]

    Thanks for replying.     From my point of view, the basis for judging whether to add domain adaptation is whether to add the loss item of'loss_mmd'.     My data type is a tabular data set of .csv type.(Two categories)     The second column classifies (label) whether it is stable or not; the data after the fourth column are all feature quantities that characterize whether it is stable or not.

When the total loss of my mini-batch is loss=loss_c, I think that no domain adaptation item is added at this time. At this time, the prediction accuracy of the corresponding source domain should be high, and the prediction accuracy of the target domain should be low. The following figure shows the test results of the last iteration:

    When my mini-batch training total loss loss=loss_c+LAMBDA * loss_mmd, the test results are as follows:

    From the comparison of the above two pictures, after adding loss_mmd, the training loss of the source domain is slightly increased, and the test loss of the target domain is slightly reduced, which is in line with the trend of domain adaptation after mmd is added. However, the difference between the two losses is too small, resulting in no change in the accuracy of the settlement.     How to modify to greatly improve the accuracy of the test target domain judgment?     I think that only the hyperparameters related to mmd can be adjusted, which are the following two hyperparameters:

    But after I make the modification, the accuracy of the source domain and target domain may decrease at the same time.     How should I make adjustments, or do I need to replace the original training model of the single hidden layer feedforward neural network?     Do I need to try DAN or DDC?     Thank you. ------------------ 原始邮件 ------------------ 发件人: "jindongwang/transferlearning" @.>; 发送时间: 2021年5月10日(星期一) 上午10:55 @.>; 抄送: "○冰o激ゞ凌 @.**@.>; 主题: Re: [jindongwang/transferlearning] Some Questions About DaNN (#226)

Please provide more details of the problem, e.g., screen shot and your dataset description. Normally you need to tune the hyperparameters.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[jindongwang]

@sunshining-bit Can't read your comments... So many error codes

[sunshining-bit]

@jindongwang 对于乱码的事情我很抱歉，我是在QQ邮箱中填写附带有截图的回复，评论不支持图片所以导致了乱码。 从我的角度来看，判断是否添加域自适应的基础是是否添加“ loss_mmd”的损失项。我的数据类型是.csv类型的表格数据集。 第二列(标签)进行分类，之后的数据都是表征的特征量。 当我的小批量处理的总损失为loss = loss_c时，我认为此时未添加任何域适应项。 此时，对应的源域的预测精度应较高，而目标域的预测精度应较低。 代码如下：

    loss_c = criterion(y_src, target)   # loss_c=源域标签和target的交叉熵损失
    pred = y_src.data.max(1)[1]  # 获取最大对数概率的索引
    correct += pred.eq(target.data.view_as(pred)).cpu().sum()   # 正确判断个数叠加
    loss = loss_c
    optimizer.zero_grad()   # 将梯度初始化为零（因为一个batch的loss关于weight的导数是所有sample的loss关于weight的导数的累加和）
    loss.backward()   # 反向传播，计算当前梯度
    optimizer.step()   # 根据梯度更新网络参数

测试结果如下：

Epoch: [50/50], training loss: 0.096118, correct: [1484/1575], training accuracy: 94.2222% Test: total loss: 4.274431, correct: [814/900], testing accuracy: 90.4444%

当我的小批量训练总损失损失= loss_c + LAMBDA * loss_mmd时，此时引入领域自适应对应损失，代码如下：

    loss_c = criterion(y_src, target)   # loss_c=源域标签和target的交叉熵损失
    loss_mmd = mmd_loss(x_src_mmd, x_tar_mmd)   # MMD损失=源域数据MMD和目标域数据MMD组合的函数
    pred = y_src.data.max(1)[1]  # 获取最大对数概率的索引
    correct += pred.eq(target.data.view_as(pred)).cpu().sum()   # 正确判断个数叠加
    loss = loss_c + LAMBDA * loss_mmd   #损失 = loss_c + λ*MMD损失
    optimizer.zero_grad()   # 将梯度初始化为零（因为一个batch的loss关于weight的导数是所有sample的loss关于weight的导数的累加和）
    loss.backward()   # 反向传播，计算当前梯度
    optimizer.step()   # 根据梯度更新网络参数

测试结果如下：

Epoch: [50/50], training loss: 0.096120, correct: [1484/1575], training accuracy: 94.2222% Test: total loss: 4.274287, correct: [814/900], testing accuracy: 90.4444%

从以上的比较来看，添加loss_mmd后，源域的训练损耗略有增加，而目标域的测试损耗略有减少，这与添加mmd后域自适应的趋势相吻合。但是，两个损失之间的差异太小，导致结算精度没有变化。 如何修改以大大提高测试目标域判断的准确性？ 我认为只能调整与mmd相关的超参数，它们应该是以下两个超参数：

LAMBDA = 0.5 # 均衡系数λ GAMMA = 1e3

但是，当我进行修改后，源域和目标域的准确性可能会同时降低。我应该如何进行调整，或者需要替换单个隐藏层前馈神经网络的原始训练模型？ 我需要尝试DAN或DDC吗？ 谢谢。

[jindongwang]

如果你追求绝对精度，可以直接上DDC，相当于就是换一个网络。调参的话，主要调的参数有：learning rate, lambda, batch size等。

[sunshining-bit]

好的，谢谢东哥。

[onlinehuazai]

请问DANN这个域对抗损失一直下降不了，怎么回事