[x] Prompt improvement by instructing GPT to distinguish standard case, edge case and error handling
System
[x] #111, #120
John encountered some error in unit testing. To be fixed on Monday.
[x] #104, #109
Orix, John to review and merge
[x] #115 (Orix to work on that after 104)
[x] #123
[x] #108
formatting checklist / output using quarto-cli
Meeting with Simon at 2pm on Monday
For a checklist item, if none of the files satisfy, show only "Not Satisfied" / "None of test function fulfilled" in the Observations
If a checklist item is partially / fully satisfied, show "Partial Satisfied" / "Satisfied" next to Observations, and keep only relevant test files and functions under the Function References.
Move the hyperlink directly to the Functions and remove the Line Numbers
[x] (John, Tony) Debug n_files_test (#119 )
[x] #146
Remove n_file_tested column in DataFrame
Put it as a subheader, e.g. "N files are tested"
[x] #137
[x] (Optional, or week 8) Write docstrings for our system's functions (incl. test function)
[x] (Optional, or week 8) Write more (our system's) test functions
[x] (Optional) To consider parallelising multiple API calls to speed up
[x] require #125
[x] #138
to include test examples respectively for standard case, edge case and error handling as a context for GPT (defined in the checklist)
to provide examples of good quality of test files + providing ground truth (defined in checklist)
to define numpy format and provide example/skeleton
to add an optional argument: giving all the docstrings of functions in project repo
functions docstrings are more a user's responsibility
[x] #121
System Evaluation
[x] #130
[x] #133
(Consistency) Consider show the F-test on the consistencies of per-item score as well
Build Quarto template to read the object and feed inside the template to render the report
Can insert code block in QMD to handle data, and call the variable_name in the textual sentences in the report.
Add reference in similar way as doing reference with Jupyter notebook and references.bib
Method 1: use a bash command to wrap 1) generate the test evaluation JSON object and dump to file; 2) Quarto doc read the JSON file and render into HTML/PDF report
[ ] Well document the schema and structure of LLM response, save the response as JSON object
Can use subprocess, Quarto template and JSON object to render the report
Link the product with user persona in the Final Presentation
Simon is on campus on Jun 10th, 11th (Mon, Tue)
[ ] Do a second meeting on Mon/Tue and/or Wed for final presentation run-through with Simon
Presentation
There were a bit slide style consistency issue
The current product is POC and illustrative for researchers
[ ] Run code coverage for other repo (e.g. coverage.py)
Data Science method: 1) checklist; 2) LLM for specifc evaluation; 3) statistical analysis to evaluate LLM response
Further comparison/assessment: 1) GPT3.5 vs GPT4o
Presentation Flow:
1) Intro
2) WHY
3) Overview of Product
4) User Persona
General User -> show the output (e.g. score, report, ...) -> Demo
Scientist -> show the output (e.g. analysis on multiple run output)
Source of Error: 1) bad prompt; 2) repo is unorthodox; 3) LLM model quality (Less Important for now)
[ ] Use a jitter plot for the SD ratio plot and SD distribution plot instead of box plot for small number of repos
Graphic evidence for 1): the SD are all high across all repo; Graph for 2): the SD are low across most repo, with some outliers repo; Graph for 3): compare the SD plot using difference models (e.g. 3.5 vs 4-o) to notice any change
For unorthodox repo, deep dive into the reason why we think they are unorthodox
Talk about the well designed response structure that allows researcher/scientists/developer to do their analysis on accuracy/consistency/other experiment
System developer
They can use the well designed response for other development (e.g. web app, etc.)
[ ] Investigate on the number of test runs for GPT4-o -> investigate how many number of test runs required for SD to stablize
Confusion Matrix for accuracy measurement, show 1 that has very good result, 1 that is bad
Mention keep human feedback in the loop as future improvement
(Good to have/Future Dev) How to guide users for consistent writing of effective Requirements
1) guideline to standardize the language; 2) refine the Requirement more based on consistency evaluation
System Dev
[ ] Add the unit for the score in evaluation report, e.g. score: 2.5/7 items
The test generator should focus on non-satisfying items
(good to have) based on the evaluation report and user input in the ML function(s) script files, add functionality to generate ref test cases for the non-satisfying items for that ML function(s)
Quarto export: 1) Do a check on Quarto package existence and give warning message & exception during export; 2) Add non-python external package spec into pyproject.toml
(Good to have) Containerize the package to include non-Python package dependencies
System Evaluation
We can show the Stat plot per 522 project
[ ] Change the stat spot from vertical to horizontal, add the Title to the y-axis as description
[ ] Need outline on what problem we are looking at -> show the graph -> point out the evidence
(Good to have)Add Experiment Github issue for improvement, e.g. few shot learning
[ ] Further breakdown the confusion matrix by checklist item/repo
(Good to have) write blog post to be posted in UBC MDS website
Sprint Planning - 2024-05-30 Week 6
Checklist
System
n_files_test
(#119 )System Evaluation
591 Requirement
Others
Future direction