Unrecognized arguments: --observe_synthetic_tests

[kgilpin]

https://github.com/getappmap/navie-benchmark/actions/runs/11546334168/job/32134543180#step:7:430

This option is provided in solve.yml, but not implemented in the solver code.

[github-actions[bot]]

Title: Implement --observe_synthetic_tests Argument in Solver Code

Problem: The --observe_synthetic_tests argument is specified in the solve.yml workflow file, but it is not implemented within the solver code. Consequently, when this argument is passed, an error occurs indicating that it is an unrecognized argument.

Analysis: The --observe_synthetic_tests argument is intended to modify the behavior of the solver, likely related to observing or handling synthetic tests. The usage of this argument suggests that the solver should conditionally observe or perform synthetic tests based on its presence and value. In the solve.yml, this argument is set as a boolean environment variable OBSERVE_SYNTHETIC_TESTS, which influences the solver's execution command.

To address this issue, the solver code must accept the --observe_synthetic_tests argument. The implementation should check for this argument and modify the solver's behavior accordingly. The solver should include appropriate handling to ensure that synthetic tests are either observed or ignored based on this parameter. The following considerations should be taken into account:

• Default behavior should be defined if the argument is not provided.
• Ensure compatibility with existing solver operations and workflows.

Proposed Changes:

1. File: solver/run_test.py:

   • Modify the ArgumentParser setup to include a new argument for --observe_synthetic_tests. This should be added as a boolean flag that, when present, alters the runtime behavior of the solver to observe synthetic tests.

2. File: solver/run_test.py:

   • Implement logic within the main function (or wherever applicable) to check for the --observe_synthetic_tests flag. If provided, adjust the behavior of the solver process to handle synthetic tests as per requirements.

3. File: solver/observe_test.py:

   • Ensure that the handling logic for observing test execution includes provisions for operating when synthetic tests are involved, if applicable. This may include configuring Docker runner instances, managing test patches, or logging outputs.

4. General Solver Integration:

   • Review how the observe behavior influences other components like RunTest and ObserveTest to confirm that the --observe_synthetic_tests logic is effectively integrated and does not disrupt existing flow.

By implementing these changes, the solver will correctly recognize the --observe_synthetic_tests argument, preventing errors related to unrecognized arguments and allowing the intended conditional behavior for synthetic tests to be observed during execution.