[suggestion] Automated Unit Test Improvement using Large Language Models at Meta

[profnandaa]

Abstract

This paper describes Meta's TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination. We describe the deployment of TestGen-LLM at Meta test-a-thons for the Instagram and Facebook platforms. In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. During Meta's Instagram and Facebook test-a-thons, it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers. We believe this is the first report on industrial scale deployment of LLM-generated code backed by such assurances of code improvement.

Links

• https://arxiv.org/abs/2402.09171
• PDF: https://arxiv.org/pdf/2402.09171.pdf

Presenters

• [ ] Anthony Nandaa
• [ ] Bildan Urandu
• [ ] Timothy Kaboya

Timeframe

• ~April or May meetup~
• Now scheduled for July 9, 2024 12 noon GMT+3

[profnandaa]

Some related papers, that could be looked into separately:

• How to Understand Whole Software Repository?
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? -> project: https://github.com/princeton-nlp/SWE-bench