fixing final report

[JohnShiuMK]

• [x] General
  • [x] No need () around citation
  • [x] use figure capture and figure number, and reference the figure in the text
  • [x] add table caption
  • [x] in each capture: Analysis of ...., description of the graph
• [x] accuracy & consistency
  • in the discussion of the figure: the consistency of the ... variary by checklist and repo. ... Why we care about it? What the figure tries to show
  • We see this accuracy, the difference in consistency also has meaning. set up for the next thing. What does that look like on the larger scale.
  • 2 divergenting case --> When we examine predict consistency, we a bot of pattern, categoric and improve ... two diverging cases.
• [x] 3.5 vs 4o
  • drop 3.5-turbo, keep 4o
  • imply prompt engineering in this project, need to explore different structure (maybe prompt engineering is needed when switching) in moving from 3.5-turbo to 4o. What works well from one LLM version will work well in another LLM version.
  • run Qlib
  • run gpt-4 in addition to 4o
• [x] cost
  • cost usage: just write a sentence. Write a code block, model the cost --> cost in USD / # of line of code in test files.
  • who are and why are they using it?
• [x] Limitation & Future Improvement
  • .... Mentioned: As shown in {previous session} ...., tell how to solve them
• [x] checklist:
  • Mention: we are using expert... flexible enough to add checklist item / more specialized checklist items. Checklist is general and may not be covering all the thing, but we design to ....

[tonyshumlh]

• No need () around citation
• Read out loud and check the writing
• It is hard to keep the flow with the figures
• Use figure caption & figure number; for code & table, add table caption
• Rewrite the caption of analysis figures, e.g. more specific explanation
  • e.g. the result of FIXML varies by repo and by checklist, we saw it always underrates .....
• the meaning of having inaccuracy/inconsistency in checklist & model
• Set up at the end of accuracy part to introduce consistency part
• Consistency: 2 diverging cases -> we observe a broad range of patterns to be concluded into 2 categories, 2 examples are ....
• gpt 3.5 to 4o: the implication is maybe more prompt engineering is needed when switching LLMs which is complicated than we thought at first, have to run Qlib
• Add cost estimate session in the report, cost = fixed cost (y-intercept, e.g. system prompt, checklist, etc.) + line of codes * cost per line, add implications (e.g. who is paying for using FIXML, why the cost estimate is important per each person/role). carry out cost benefit analysis
• Future Improvement: refer to all the issues we discussed (e.g. shown in fig x) along the report, how to solve them
  • e.g. performance optimization points back to evaluation results
• Checklist Design: mentioned that it is flexible enough to add new items. How each component can facilitate Future Improvement

[jinyz8888]

@tonyshumlh. Usually cost-benefit analysis is a separate section in the proposal or report, not in the future improvement because it is for current project. For the project in the future, the team will do independent cost-benefit analysis based on the scope of their project. So I put it into after "Add cost estimate..."