请问如何修改UniXcoder以支持Code-Refine下游任务？

[KaiHuangNIPC]

您好，我想要尝试使用UniXcoder实现code-refine任务，这是一个类似于神经机器翻译的下游任务。 我在实现时参考了UniXcoder的Code-generation和Code-summarization这两个下游任务，但是我的实现可能有些问题，表现在训练集上的loss正常收敛，但是xmatch和bleu指标效果并不理想(还不如CodeBERT和GraphCodeBERT)，甚至有时候模型的验证效果会随着训练epoch的增加而变得越来越差。我猜想是不是因为我在处理tokens的过程出现了问题，因为我在处理source_tokens和target_tokens时没有添加<mask0>，这是否会导致我的实现达不到正常的效果？麻烦您帮我看一下是否是这个原因，感谢！

这是我的详细代码：

这是训练日志记录：

epoch 0 loss 0.201: 100%|██████████| 6546/6546 [54:48<00:00,  1.99it/s]
02/02/2023 10:10:50 - INFO - __main__ -   
***** Running evaluation *****
02/02/2023 10:10:50 - INFO - __main__ -     Num examples = 6546
02/02/2023 10:10:50 - INFO - __main__ -     Batch size = 8
02/02/2023 10:13:11 - INFO - __main__ -     eval_ppl = 1.08751
02/02/2023 10:13:11 - INFO - __main__ -     global_step = 6547
02/02/2023 10:13:11 - INFO - __main__ -     train_loss = 0.201
02/02/2023 10:13:11 - INFO - __main__ -     ********************
02/02/2023 10:13:12 - INFO - __main__ -     Best ppl:1.08751
02/02/2023 10:13:12 - INFO - __main__ -     ********************
02/02/2023 11:10:45 - INFO - __main__ -     Calculate bleu
02/02/2023 11:10:46 - INFO - __main__ -     bleu-4 = 90.91 
02/02/2023 11:10:46 - INFO - __main__ -     xMatch = 2.6 
02/02/2023 11:10:46 - INFO - __main__ -     ********************
02/02/2023 11:10:46 - INFO - __main__ -     Best BLEU: 90.91
02/02/2023 11:10:46 - INFO - __main__ -     ********************

epoch 1 loss 0.0854: 100%|██████████| 6546/6546 [55:13<00:00,  1.98it/s]
02/02/2023 12:06:00 - INFO - __main__ -   
***** Running evaluation *****
02/02/2023 12:06:00 - INFO - __main__ -     Num examples = 6546
02/02/2023 12:06:00 - INFO - __main__ -     Batch size = 8
02/02/2023 12:08:23 - INFO - __main__ -     eval_ppl = 1.0802
02/02/2023 12:08:23 - INFO - __main__ -     global_step = 13093
02/02/2023 12:08:23 - INFO - __main__ -     train_loss = 0.0854
02/02/2023 12:08:23 - INFO - __main__ -     ********************
02/02/2023 12:08:25 - INFO - __main__ -     Best ppl:1.0802
02/02/2023 12:08:25 - INFO - __main__ -     ********************
02/02/2023 13:06:34 - INFO - __main__ -     Calculate bleu
02/02/2023 13:06:35 - INFO - __main__ -     bleu-4 = 90.64 
02/02/2023 13:06:35 - INFO - __main__ -     xMatch = 4.6 
02/02/2023 13:06:35 - INFO - __main__ -     ********************
02/02/2023 13:06:35 - INFO - __main__ -     Not Best BLEU, Now BLEU: 90.64, Best BLEU: 90.91
02/02/2023 13:06:35 - INFO - __main__ -     ********************

epoch 2 loss 0.0771: 100%|██████████| 6546/6546 [54:56<00:00,  1.99it/s]
02/02/2023 14:01:31 - INFO - __main__ -   
***** Running evaluation *****
02/02/2023 14:01:31 - INFO - __main__ -     Num examples = 6546
02/02/2023 14:01:31 - INFO - __main__ -     Batch size = 8
02/02/2023 14:03:54 - INFO - __main__ -     eval_ppl = 1.07865
02/02/2023 14:03:54 - INFO - __main__ -     global_step = 19639
02/02/2023 14:03:54 - INFO - __main__ -     train_loss = 0.0771
02/02/2023 14:03:54 - INFO - __main__ -     ********************
02/02/2023 14:03:56 - INFO - __main__ -     Best ppl:1.07865
02/02/2023 14:03:56 - INFO - __main__ -     ********************
02/02/2023 15:10:48 - INFO - __main__ -     Calculate bleu
02/02/2023 15:10:49 - INFO - __main__ -     bleu-4 = 86.8 
02/02/2023 15:10:49 - INFO - __main__ -     xMatch = 3.8 
02/02/2023 15:10:49 - INFO - __main__ -     ********************
02/02/2023 15:10:49 - INFO - __main__ -     Not Best BLEU, Now BLEU: 86.8, Best BLEU: 90.91
02/02/2023 15:10:49 - INFO - __main__ -     ********************

epoch 3 loss 0.0678: 100%|██████████| 6546/6546 [57:07<00:00,  1.91it/s]
02/02/2023 16:07:56 - INFO - __main__ -   
***** Running evaluation *****
02/02/2023 16:07:56 - INFO - __main__ -     Num examples = 6546
02/02/2023 16:07:56 - INFO - __main__ -     Batch size = 8
02/02/2023 16:10:21 - INFO - __main__ -     eval_ppl = 1.07621
02/02/2023 16:10:21 - INFO - __main__ -     global_step = 26185
02/02/2023 16:10:21 - INFO - __main__ -     train_loss = 0.0678
02/02/2023 16:10:21 - INFO - __main__ -     ********************
02/02/2023 16:10:23 - INFO - __main__ -     Best ppl:1.07621
02/02/2023 16:10:23 - INFO - __main__ -     ********************
02/02/2023 17:28:09 - INFO - __main__ -     Calculate bleu
02/02/2023 17:28:10 - INFO - __main__ -     bleu-4 = 80.77 
02/02/2023 17:28:10 - INFO - __main__ -     xMatch = 2.3 
02/02/2023 17:28:10 - INFO - __main__ -     ********************
02/02/2023 17:28:10 - INFO - __main__ -     Not Best BLEU, Now BLEU: 80.77, Best BLEU: 90.91
02/02/2023 17:28:10 - INFO - __main__ -     ********************

epoch 4 loss 0.0595: 100%|██████████| 6546/6546 [56:40<00:00,  1.93it/s]
02/02/2023 18:24:51 - INFO - __main__ -   
***** Running evaluation *****
02/02/2023 18:24:51 - INFO - __main__ -     Num examples = 6546
02/02/2023 18:24:51 - INFO - __main__ -     Batch size = 8
02/02/2023 18:27:14 - INFO - __main__ -     eval_ppl = 1.07617
02/02/2023 18:27:14 - INFO - __main__ -     global_step = 32731
02/02/2023 18:27:14 - INFO - __main__ -     train_loss = 0.0595
02/02/2023 18:27:14 - INFO - __main__ -     ********************
02/02/2023 18:27:16 - INFO - __main__ -     Best ppl:1.07617
02/02/2023 18:27:16 - INFO - __main__ -     ********************
02/02/2023 19:31:27 - INFO - __main__ -     Calculate bleu
02/02/2023 19:31:28 - INFO - __main__ -     bleu-4 = 86.79 
02/02/2023 19:31:28 - INFO - __main__ -     xMatch = 5.6 
02/02/2023 19:31:28 - INFO - __main__ -     ********************
02/02/2023 19:31:28 - INFO - __main__ -     Not Best BLEU, Now BLEU: 86.79, Best BLEU: 90.91
02/02/2023 19:31:28 - INFO - __main__ -     ********************

epoch 6 loss 0.049: 100%|██████████| 6546/6546 [56:40<00:00,  1.92it/s]
02/02/2023 21:23:53 - INFO - __main__ -   
***** Running evaluation *****
02/02/2023 21:23:53 - INFO - __main__ -     Num examples = 6546
02/02/2023 21:23:53 - INFO - __main__ -     Batch size = 8
02/02/2023 21:26:16 - INFO - __main__ -     eval_ppl = 1.07921
02/02/2023 21:26:16 - INFO - __main__ -     global_step = 45823
02/02/2023 21:26:16 - INFO - __main__ -     train_loss = 0.049
02/02/2023 21:26:16 - INFO - __main__ -     ********************
02/02/2023 22:38:49 - INFO - __main__ -     Calculate bleu
02/02/2023 22:38:50 - INFO - __main__ -     bleu-4 = 80.32 
02/02/2023 22:38:50 - INFO - __main__ -     xMatch = 5.7 
02/02/2023 22:38:50 - INFO - __main__ -     ********************
02/02/2023 22:38:50 - INFO - __main__ -     Not Best BLEU, Now BLEU: 80.32, Best BLEU: 90.91
02/02/2023 22:38:50 - INFO - __main__ -     ********************

[guoday]

我建议你加一下，要么调低一下学习率。但是我记得我在code-refine跑过unixcoder，好像也是不理想的。

[KaiHuangNIPC]

我建议你加一下，要么调低一下学习率。但是我记得我在code-refine跑过unixcoder，好像也是不理想的。

非常感谢您的回复。我通过调整发现<mask0>非常重要，通过添加这一标记符使得模型训练更加平稳，loss不再出现较大波动。此外我也发现了UniXcoder在基于完美缺陷定位的假设下，其repair accuracy超过了CodeBERT和GraphCodeBERT。

[guoday]

是的，因为预训练的时候，用的是。所以微调加上会好些

[KaiHuangNIPC]

您好，我还有一个问题需要请教您一下。我看到CodeBERT和GraphCodeBERT的参数量都是125M，请问UniXCoder的参数量是多少呢？

[guoday]

一样