phodal
commented
5 months ago we have the similar conclusion to https://arxiv.org/abs/2402.06013 。
Summary:
-
Limited Understanding of Codebase Context: ChatGPT's comprehension of the entire codebase is restricted, potentially leading to missing dependencies and codependencies in its suggestions. Developers should understand the model's mechanics rather than treating it as a black box. The study revealed instances where suggested code changes introduced compiler errors or test failures, indicating the need for developers to verify suggestions.
-
Quality of Responses Depends on Prompts: The quality of ChatGPT responses is heavily influenced by the quality of the prompts. While it offers insightful suggestions for non-urgent issues like code style and comments, challenges arise when it misunderstands code fragments or receives incomplete code. Understanding common patterns and challenges in refactoring conversations can guide improvements in ChatGPT's capabilities.
-
Variability in Learning Settings: Developers use different learning settings with ChatGPT, including zero-shot, one-shot, and few-shot learning. This reflects the non-uniformity of developers' prompts and their varying expectations of the model's capabilities. Insights from these conversations can inform the integration of ChatGPT into development tools to align better with developers' expectations during refactoring tasks.
-
Need for Better Refactoring Descriptions: Developers should provide more specific descriptions when asking ChatGPT to refactor code. Adding the intent behind the refactoring and potential instructions can help ensure that the model's suggestions align with specific coding practices and styles targeted by the developer or their company policy.
Case suggestion:
Readability (22.13%) Usability (16.19%) Performance (10.25%) Maintainability (8.32%) Flexibility (8.32%)
the better refactor ideas suggestion for AI