Preprocessor step disposing numbers in (variable) names

[daveymathijssen]

Hello,

Thank you for your awesome work!

I have trained some models from scratch using a C# dataset. However, I have a question about the code2vec C# preprocessor. Some variables within the dataset samples contain a number to make them unique, such as variable_1 and variable_2. However, the SplitToSubtokens method removes these variable numbers during preprocessing. I am interested in the purpose of removing these numbers, because when I alter the SplitToSubtokens method and do not remove the numbers, all metrics seem to improve.

Best regards, Davey

[urialon]

Hi @daveymathijssen , Thank you for your interest in our work!

The C# constructor was contributed by researchers from Microsoft, so I am not sure why did they remove numbers. I have might removed number in the JavaExtractor as well, I'm not sure. But if it improves your metrics when you do not remove the numbers, that's great!

Best, Uri

[daveymathijssen]

HI @urialon, Thanks for your answer!

In the JavaExtractor, the same is happening in the normalizeName method. Do you have any clue why?

[urialon]

I'm guessing that at the time, We did not want to spend embeddings on numbers, as we had ~1M embedding vocabulary anyway. This is solved in newer models by segmenting tokens into subwords.

You can also check out our newer models

• code2seq: https://code2seq.org/
• PolyCoder: https://github.com/VHellendoorn/Code-LMs
• CodeBERT that we finetuned on multiple languages: https://github.com/neulab/code-bert-score#backend-model

Best, Uri

[daveymathijssen]

Thank you for your fast responses and insights!