This pull request includes several changes to enhance the functionality of the test framework in the src/agent module. The main updates involve improving the handling of test outputs, particularly for screenshots taken during test failures, and refactoring some utility functions.
Enhancements to Test Framework:
Screenshot Handling:
Added the check_for_screenshots function in src/agent/utils.py to find and encode screenshots taken during test failures. ([[1]](diffhunk://#diff-d296951d796f0ca48d77aa4eca27e060bca2660129539ab12bddc66a770b249aR2-R7), [[2]](diffhunk://#diff-d296951d796f0ca48d77aa4eca27e060bca2660129539ab12bddc66a770b249aL16-R52))
Updated the agent function in src/agent/completions.py to include screenshots in the context if tests fail. ([src/agent/completions.pyR167-R187](diffhunk://#diff-a8368bd8b80728800a31a574835c38f9f17086c0c609f85e33e85db4404f15d4R167-R187))
Test Prompt Improvements:
Modified the framework_prompt function in src/agent/completions.py to provide a more detailed structure for writing Playwright tests using the pytest-playwright plugin. ([src/agent/completions.pyL16-R28](diffhunk://#diff-a8368bd8b80728800a31a574835c38f9f17086c0c609f85e33e85db4404f15d4L16-R28))
Code Refactoring:
Imports Cleanup:
Removed unused imports and streamlined the import statements in src/agent/tools.py. ([src/agent/tools.pyL2-R7](diffhunk://#diff-e0b516691ff100add4484a056628cf60d1ad3e562e1540f1a63a67d44cf3efb6L2-R7))
Removed Redundant Code:
Removed the code that manually extracted and displayed images from trace.zip in the run_tests function in src/agent/tools.py. This functionality is now handled by the new check_for_screenshots function. ([src/agent/tools.pyL93-L113](diffhunk://#diff-e0b516691ff100add4484a056628cf60d1ad3e562e1540f1a63a67d44cf3efb6L93-L113))
These changes collectively improve the robustness and clarity of the test framework, making it easier to debug test failures by providing direct access to screenshots and simplifying the test-writing process.
This pull request includes several changes to enhance the functionality of the test framework in the
src/agentmodule. The main updates involve improving the handling of test outputs, particularly for screenshots taken during test failures, and refactoring some utility functions.Enhancements to Test Framework:
Screenshot Handling:
check_for_screenshotsfunction insrc/agent/utils.pyto find and encode screenshots taken during test failures. ([[1]](diffhunk://#diff-d296951d796f0ca48d77aa4eca27e060bca2660129539ab12bddc66a770b249aR2-R7),[[2]](diffhunk://#diff-d296951d796f0ca48d77aa4eca27e060bca2660129539ab12bddc66a770b249aL16-R52))agentfunction insrc/agent/completions.pyto include screenshots in the context if tests fail. ([src/agent/completions.pyR167-R187](diffhunk://#diff-a8368bd8b80728800a31a574835c38f9f17086c0c609f85e33e85db4404f15d4R167-R187))Test Prompt Improvements:
framework_promptfunction insrc/agent/completions.pyto provide a more detailed structure for writing Playwright tests using the pytest-playwright plugin. ([src/agent/completions.pyL16-R28](diffhunk://#diff-a8368bd8b80728800a31a574835c38f9f17086c0c609f85e33e85db4404f15d4L16-R28))Code Refactoring:
Imports Cleanup:
src/agent/tools.py. ([src/agent/tools.pyL2-R7](diffhunk://#diff-e0b516691ff100add4484a056628cf60d1ad3e562e1540f1a63a67d44cf3efb6L2-R7))Removed Redundant Code:
trace.zipin therun_testsfunction insrc/agent/tools.py. This functionality is now handled by the newcheck_for_screenshotsfunction. ([src/agent/tools.pyL93-L113](diffhunk://#diff-e0b516691ff100add4484a056628cf60d1ad3e562e1540f1a63a67d44cf3efb6L93-L113))These changes collectively improve the robustness and clarity of the test framework, making it easier to debug test failures by providing direct access to screenshots and simplifying the test-writing process.