Meeting Minutes for Week 6

JohnShiuMK commented 3 months ago

Sprint Planning - 2024-05-30 Week 6

Checklist

[x] #63 (to be done on Monday)
[x] #91 (waiting for Tiffany's replies)
[x] Prompt improvement by instructing GPT to distinguish standard case, edge case and error handling

System

[x] #111, #120
- John encountered some error in unit testing. To be fixed on Monday.
[x] #104, #109
- Orix, John to review and merge
[x] #115 (Orix to work on that after 104)
[x] #123
[x] #108
- formatting checklist / output using quarto-cli
- Meeting with Simon at 2pm on Monday
- For a checklist item, if none of the files satisfy, show only "Not Satisfied" / "None of test function fulfilled" in the Observations
- If a checklist item is partially / fully satisfied, show "Partial Satisfied" / "Satisfied" next to Observations, and keep only relevant test files and functions under the Function References.
- Move the hyperlink directly to the Functions and remove the Line Numbers
[x] (John, Tony) Debug n_files_test (#119 )
[x] #146
- Remove n_file_tested column in DataFrame
- Put it as a subheader, e.g. "N files are tested"
[x] #137
[x] (Optional, or week 8) Write docstrings for our system's functions (incl. test function)
[x] (Optional, or week 8) Write more (our system's) test functions
[x] (Optional) To consider parallelising multiple API calls to speed up
- [x] require #125
[x] #138
- to include test examples respectively for standard case, edge case and error handling as a context for GPT (defined in the checklist)
- to provide examples of good quality of test files + providing ground truth (defined in checklist)
- to define numpy format and provide example/skeleton
- to add an optional argument: giving all the docstrings of functions in project repo functions docstrings are more a user's responsibility
[x] #121

System Evaluation

[x] #130
[x] #133
- (Consistency) Consider show the F-test on the consistencies of per-item score as well

591 Requirement

[x] #97
- (Accuracy) For future users to contribute, to add Acknowledgement (refer to Tiffany's py-pkg Github repo)
[x] #98
[x] Writing final report
- Potential issues about our system to be stated in the final report:
  1. probably there is checklist item issue that requires improvement
  2. probably there is structure issue for the repo
  3. Add a conclusion whether we/users should trust our result in the Final Presentation based on the above methods

Others

Will have a separate presentation with Dr Rohan Alexander in June 24/25th morning

Future direction

Consider including good test examples and repo example with ground truth as a context for GPT
add a hyperlink to go to the part of the code that is relevant to the missing test (require taking project code base as a context)

tonyshumlh commented 3 months ago

Quarto Discussion 2024/06/03 Week 6

Build Quarto template to read the object and feed inside the template to render the report
Can insert code block in QMD to handle data, and call the variable_name in the textual sentences in the report.
Add reference in similar way as doing reference with Jupyter notebook and references.bib
Method 1: use a bash command to wrap 1) generate the test evaluation JSON object and dump to file; 2) Quarto doc read the JSON file and render into HTML/PDF report
[ ] Well document the schema and structure of LLM response, save the response as JSON object
Can use subprocess, Quarto template and JSON object to render the report

JohnShiuMK commented 2 months ago

go through finding reports
final report / presentation, why score low?
- data science approach (other ppl approach): literature review <-- Simon, tiff
- data science approach (our approach): LLM, correct?
pickle or json for response output
tests cases requirement
readthedocs for system doc, required?
openai usage

tonyshumlh commented 2 months ago

Mentor Meeting 2024/06/06 Week 6

Link the product with user persona in the Final Presentation
Simon is on campus on Jun 10th, 11th (Mon, Tue)
[ ] Do a second meeting on Mon/Tue and/or Wed for final presentation run-through with Simon
Presentation
There were a bit slide style consistency issue
The current product is POC and illustrative for researchers
[ ] Run code coverage for other repo (e.g. coverage.py)
Data Science method: 1) checklist; 2) LLM for specifc evaluation; 3) statistical analysis to evaluate LLM response
Further comparison/assessment: 1) GPT3.5 vs GPT4o
Presentation Flow:
1) Intro
2) WHY
3) Overview of Product
4) User Persona
- 1. General User -> show the output (e.g. score, report, ...) -> Demo
- 1. Scientist -> show the output (e.g. analysis on multiple run output)
- Source of Error: 1) bad prompt; 2) repo is unorthodox; 3) LLM model quality (Less Important for now)
- [ ] Use a jitter plot for the SD ratio plot and SD distribution plot instead of box plot for small number of repos
- Graphic evidence for 1): the SD are all high across all repo; Graph for 2): the SD are low across most repo, with some outliers repo; Graph for 3): compare the SD plot using difference models (e.g. 3.5 vs 4-o) to notice any change
- For unorthodox repo, deep dive into the reason why we think they are unorthodox
- Talk about the well designed response structure that allows researcher/scientists/developer to do their analysis on accuracy/consistency/other experiment
- 1. System developer
- They can use the well designed response for other development (e.g. web app, etc.)
  - [ ] Investigate on the number of test runs for GPT4-o -> investigate how many number of test runs required for SD to stablize
  - Confusion Matrix for accuracy measurement, show 1 that has very good result, 1 that is bad
  - Mention keep human feedback in the loop as future improvement
  - Test
  - 1) error handling
  - 2) validate the output if it is as expected
  - 3) mock the workflow with random files
Read the Doc is not necessary

tonyshumlh commented 2 months ago

Partner Meeting 2024/06/10 Week 6

Checklist
Checklist Requirement writing
(Good to have/Future Dev) How to guide users for consistent writing of effective Requirements
1) guideline to standardize the language; 2) refine the Requirement more based on consistency evaluation
System Dev
[ ] Add the unit for the score in evaluation report, e.g. score: 2.5/7 items
The test generator should focus on non-satisfying items
(good to have) based on the evaluation report and user input in the ML function(s) script files, add functionality to generate ref test cases for the non-satisfying items for that ML function(s)
Quarto export: 1) Do a check on Quarto package existence and give warning message & exception during export; 2) Add non-python external package spec into pyproject.toml
(Good to have) Containerize the package to include non-Python package dependencies
System Evaluation
We can show the Stat plot per 522 project
[ ] Change the stat spot from vertical to horizontal, add the Title to the y-axis as description
[ ] Need outline on what problem we are looking at -> show the graph -> point out the evidence
(Good to have)Add Experiment Github issue for improvement, e.g. few shot learning
[ ] Further breakdown the confusion matrix by checklist item/repo
(Good to have) write blog post to be posted in UBC MDS website