test_evaluate script does not work with variations

[Astor-Bizard]

I am trying to use the new feature for checking automatic tests, and it works smoothly on assignments without variations.

However, when using variations, every test ends up with 0 grade and the message "No test case found".
I followed the manual and created a directory for every variation. The compilation phase detects each one, but the execution phase yields the "No test case found" error.
On student side (or Test activity tab), evaluation works fine with the corresponding variations.

My guess at the moment is that it searches for a vpl_evaluate.cases on the parent directory, but for variations the vpl_evaluate.cases file is on the grandparent (?) directory.
EDIT: It comes from the VPL_VARIATION var, which is set to "-" during the execution of these tests. As my tests are set with variations, no test case corresponding to the "-" variation exists.

[jcrodriguez-dis]

Dear Astor,

Thank you for your detailed bug report regarding the issue with automatic tests using variations. Based on the information you provided, it appears that the problem arises during the execution phase of assignments with variations, where tests are incorrectly using VPL_VARIATION=-.

To further assist in resolving this problem, it would be helpful if you could provide the activity as a Moodle backup or as files with its directory structure.

Thank you once again for your support and patience. I appreciate your efforts in making VPL a better tool for everyone.

Best regards,

Juan Carlos

[jcrodriguez-dis]

Dear Astor,

I am pleased to inform you that the issue you reported concerning the tests of tests of VPL activities with variations has been successfully resolved. The root cause was indeed related to the handling of the VPL_VARIATION variable. The solution has been implemented in the master branch, ensuring that the VPL_VARIATION variable in each test of tests is now correctly assigned for the variation tested.

Best regards,

Juan Carlos Rodriguez

[Astor-Bizard]

Hi,

Thank you for your quick reaction. However, the problem still persists on my side.
I implemented a simple fix and did a pull request #166 that does it for me (it wasn't linked to this issue somehow, so here it is).

Best regards, Astor

[jcrodriguez-dis]

Dear Astor,

Upon careful review of your proposed solution and comparing it with my implementation, I have determined that our approaches are differents. Your method involves setting the VPL_VARIATION within the general vpl_environment.sh file prior to each solution's compilation. This strategy is effective, especially in scenarios where the compilation phase must be aware of the variation. However, if the compilation phase results in a compiled program rather than a script utilizing the vpl_environment.sh, the VPL_VARIATION will not accurately represent the current variation.

The current methodology addresses this by assigning the variation immediately before evaluating each solution, effectively ensuring that the VPL_VARIATION is correctly set for the evaluation phase. Nonetheless, it does not solve the issue of variation-dependent compilation.

It appears that a hybrid approach, integrating elements from both approachs, will be the good solution.

Best regards, Juan Carlos