search

sonoisa / code2vec

an implementation of "code2vec: Learning Distributed Representations of Code"
MIT License
29 stars 11 forks source link

Source code parsing pipeline #4

Open dhas opened 4 years ago

dhas commented 4 years ago

Hi @sonoisa,

I wasn't able to understand how you arrived at the dataset you provide in your code2vec/data directory. Could you clarify your source code parsing pipeline? If I understand correctly, you seem to have started with the parsed tokens serialized as JSON from http://groups.inf.ed.ac.uk/cup/codeattention/ and you have converted into *.txt in code2vec/data. Am I right?

Would you be able to add the code for doing this into the repo? I need to parse sources written in C which is why I'm seeking a clearer picture of parsing.

Thanks

wangyu1997 commented 4 years ago

I have the same problem.

dhas commented 4 years ago

@wangyu1997 - You may want to take a look at https://github.com/JetBrains-Research/astminer. They have a great parsing pipeline and a small implementation of Code2Vec, which can get you started

wangyu1997 commented 4 years ago

@dhas Thank you for reply, after i reviewing you code, I notice that the all variables in you terminal_idxs.txt are represented like "@var_xx", could you tell more about the detail? thanks!

sonoisa commented 4 years ago

Hi @dhas, @wangyu1997, I just uploaded a Jupyter Notebook (Scala script) to run the preprocessing.