hu-macsy / simexpal

Simplifying Experimental Algorithmics
https://simexpal.readthedocs.io
MIT License
17 stars 18 forks source link

Add Github Actions workflow and unit test for 'util.validate_setup_file()' and cli #89

Closed duylethanh closed 3 years ago

duylethanh commented 3 years ago

This PR contains the following:

The workflow currently uses the cross product of [ubuntu-latest, macos-latest] and [python3.6, python3.7] to run the unit tests The following screenshot is from a test pull request on my fork https://github.com/duylethanh/simexpal/pull/4:

GithubActions

Some things to consider:

duylethanh commented 3 years ago

PR updated. I made the requested changes. ~, while doing some refactoring to the new code. I also removed unused import statements.~

Are there any other simple but useful tests that we could add? Maybe running experiments via the Python API (= a fork launcher)? I guess we would have to copy the experiments directory to a temporary location for that to work.

I have added some cli tests via subprocess.check_call(). As we only verify the return code with this method we might run into similar problems like the following; simex e launch will launch the experiments or skip experiments if the instance is not available. Both actions will not cause an error and the program ends with return code 0.

This means, that we might want to parse the print() outputs of a subprocess and compare it to the expected output, to avoid different outcomes with the same return code.

edit: I removed the unrelated commits to keep this PR clean. I will make a separate PR for the minor fixes.