Data Collection

[FernandoPieressa]

ToDo:

• [x] Filter current data by bug/fixes

• [ ] Saving file before change.

• [ ] Saving line after change.

• [ ] Given the files before change, lex with BPE and embeddings word2vec

[hsezhiyan]

@VHellendoorn does this plan sound good?

[VHellendoorn]

Sorta, some comments:

• Best not to web-scrape; the GH API limit is pretty low. Just share the code that reads a CSV file in the format I attach below and extracts the necessary data for each commit, and I will run it overnight on the collected data.
• For the file after change, please just extract the corresponding line number(s) based on the below format as well. Let's tentatively assume we extract one full set of meta-data per commit.
• BPE + Word2Vec are probably overkill for now; let's first get a basic model going that just learns the embeddings along with the rest of the model. No harm getting into BPE after that; Word2Vec I'm not sure about (might not be useful here).

Output format, where we will tentatively focus only on the special case where only one java file changed with one diff in that file removing and adding one line:

organization,project,commit,files_changed,java_files_changed,file,file_diffs,lines_removed,lines_added,line_rm_start,line_rm_end,line_add_start,line_add_end
a2888409,face2face,59f8472a9611ab1dd9f5f58de18f6a758aca9866,1,1,gate/src/main/java/gate/ClientMessage.java,2,1,1,59,59,59,59

[hsezhiyan]

@VHellendoorn sounds good.

We will be using the following command: git show REVISION:path/to/file to get the file before a particular commit.

Can you explain what the following values in your output represent: line_rm_start, line_rm_end, line_add_start, and line_add_end. In particular, are these lines in the precommit files or the postcommit files.

[VHellendoorn]

The rm lines are in the pre-commit file, the add lines in the post-commit.