[icodegen] Main Project Thread

[danaderp]

In this issue, we are going to describe tasks that have been implemented so far and the remaining tasks for the project.

Training Sheet: Link

RoadMap

Phase 1 [week1&2]

• [ ] Phase 1A: Model Training and Validation, Reporting Accuracy and Loss [RQ2.1] [Avi & Alvaro]

• [ ] Phase 1B: Dataset Organization in DVC, including testbeds for perturbation analysis. We will report statistical properties of the datasets and testbeds [RQ1] [Nathan & David]

• [ ] Phase 1C: Related work and Approach section Draft [David & Nathan]

  Phase 2 [week 3 & 4]

• [ ] Phase 2A: Model Training, Validation, and Sampling. Organize the samples in proper datasets in dvc. Report accuracy and loss of remaining models. [RQ2.1] [Avi]

• [ ] Phase 2B: Organization in classes and statistical validation of histogram of errors . Implementation of Causal Assessment [RQ2.2 & RQ3] [David]

• [ ] Phase 2C: Implementation of Overlapping Venn Diagramm [RQ2.3] [Nathan]

• [ ] Phase 2C: Empirical Desing (or experiment desing) draft [David & Nathan]

Phase 3 [week 5]

• [ ] Phase 3A: epsilonCodeGen and causalCodeGen experimentation on samples [Avi, Nathan, David]
• [ ] Phase 3B: Result Section Draft [David & Nathan]

Phase 4 [week 6]

• [ ] Phase 4A: In-the-wild Experimentation [RQ 2.4] [Nathan]
• [ ] Phase 4B: In the wild Result section [Nathan] and Introduction [David]
• [ ] Phase 4C: epsilonCodeGen and causalCodeGen experimentation (if apply) [Avi]

Phase 5 [Week 7]

• [ ] Phase 5A: Analysis of results and re-experimentations [Avi, Nathan, & David]
• [ ] Phase 5B: Discussion Section Draft [Nathan & David]
• [ ] Phase 5C: Editing [ALL]

Team Roles

Roles

• Model Trainer: Training and Sampling Models. Generate sample datasets.
• Experiment Developer: execution of experiments and reports
• Designer: Design of experiments and approach. Validate statistical soundness.
• Approach Developer: implementation and testing of the approach and statistical methods
• Analyst: analysis of results and validation

Assignments

• Avi: Experiment Developer & Model Trainer
• Alvaro: Model Trainer
• Nathan: Experiment Developer & Approach Developer & Analyst
• David: Designer, Approach Developer, and Analyst

[danaderp]

Internal Meeting (Minutes): 06.02.21

Purpose: Organize the status of the research, clarify the baselines, present new information, discuss the experiment design, and present the experience with unconditioned models.

1. Re-introduce the purpose and motivations of icodegen (by Nathan)
2. Quickly present new discoveries about interpretability techniques in Software Engineering and Code Generators (by Alvaro)
3. Introduce perturbation analysis: baselines and interventions (by David) [All discussing]
4. Research Question Definitions (by Nathan and David)
5. Identify Completed Tasks and Listed them
6. Identify To-Dos and Roles for icodegen
7. Plan tomorrow's meeting :)

Action Items

• [x] Nathan's inspection of experiments
• [ ] Alvaro's research of Tiny-GPT
• [x] Alvaro's mind-map of SOTA interpretability

[danaderp]

General Meeting Minutes: 06.03.21

Purpose: Present the plan of iCodege, the new advancements of the project, the plan of training models, and the interpretability background.

Agenda

• [x] Presenting Agenda and Goals of the meeting
• [x] Introducing Avi and Alvaro (by David)
• [x] Re-introduce purpose and motivations of icodegen (by Nathan)
• [x] Presenting SOTA Interpretability and SE Interpretability (by Alvaro)
• [x] Results of the first prototypes of RNNs and GPT (by Alvaro & Nathan)
• [x] Presenting icodegen approach (by David)
• [x] Presenting icodegen evaluation (by David & Nathan)
• [x] Introducing Road Map by weeks and Roles
• [x] Action Items ? To-Dos?

Notes & Action Items:

Insights:

• Kevin sees that the taxonomy of interpretability gives a motivation for our paper in the introduction. We should contemplate it to start writing parts of the introduction.
• Kevin wants us to put the EDA rhetoric into context with the approach. Like if we are using token distribution how does it help in the approach. In other words, we want the EDA supporting the epsilonCodeGen analysis.

Action Items

• [ ] We want a first draft of the paper with some motivation, approach section, and related work.
• [ ] An important threat to validity to include asap: Non tokenized models are the solution for future models.

[danaderp]

Summary Meeting 26.07.21

Main Notes

• Nathan made a very complete recap of icodegen. The purpose of the project, the techniques, and the employed datasets.
• David explained the 4 research questions:
• RQ1: [covariate analysis] What is the state of statistical properties of code datasets based on token distribution, method size, popular SE metrics, and information content?
• RQ2: [baselines, control, and in-the-wild] How good are generative models at predicting java code?
• RQ3: [all perturbations] To What extent is epsilonCodeGen affected by developer-based perturbations?
• RQ4: [sensitivity analysis] To what extent are the models stable under syntactic alterations?
• Results were presented for each research question. However, the results were mainly focused on Vanilla RNNs for buggy/non-buggy and commented/non-commented datasets.

Interesting Results

• Buggy and Non Buggy samples are perceived by the RNN as the same type of samples. No clear separation. Intuitively we want to observe that buggy samples perform worst.
• Commented and Non-Commented are also perceived by the RNN as the same type of samples. We can observe a subtle separation between the datasets giving an advantage to Commented code.

Action Items