This pull request adds functional tests to the project to improve its reliability and catch bugs earlier in the development process. The tests cover the most critical experiments included in Cogment-Verse.
Motivation
Functional tests are an important part of any software development process, as they help to ensure that the software works as expected from an end user perspective. By adding functional tests to this project, we can catch bugs earlier and reduce the risk of regressions in future releases.
Testing
The functional tests were written using the pytest framework and cover several experiments, including at least one for each actor implementation. Additionally, a test for the execution of two consecutive experiments was added to simulate the training of a model, then testing its behavior in another experiment.
During functional tests, the original experiment configs are untouched. The entire config/ directory is copied to a temporary location (tests/functional/.tmp_config), where the functional test configs are also copied. This allows to have access to all configs required to run the basic services, actors and environments, while having the modified experiment configs to test a minimal experiment setup. All cogment_related artifacts are also in a separate working directory (tests/functional/.tmp_cogment_verse). Both temporary directories are cleaned out at the end of the test module.
The functional tests are added to the phase: test of the deployment process.
Documentation
Updated the development_setup.md with instructions to run functional tests.
Other changes
Separation between mlflow experiment tracker in simple experiment tracker (only using logs).
both experiment trackers are configurable from hydra.
mlflow tracker is the default.
test configs use the simple experiment tracker.
CogmentVerseError class. Used to catch when the mlflow server was not started or can't connect.
closes #74
Overview
This pull request adds functional tests to the project to improve its reliability and catch bugs earlier in the development process. The tests cover the most critical experiments included in Cogment-Verse.
Motivation
Functional tests are an important part of any software development process, as they help to ensure that the software works as expected from an end user perspective. By adding functional tests to this project, we can catch bugs earlier and reduce the risk of regressions in future releases.
Testing
The functional tests were written using the pytest framework and cover several experiments, including at least one for each actor implementation. Additionally, a test for the execution of two consecutive experiments was added to simulate the training of a model, then testing its behavior in another experiment.
During functional tests, the original experiment configs are untouched. The entire
config/directory is copied to a temporary location (tests/functional/.tmp_config), where the functional test configs are also copied. This allows to have access to all configs required to run the basic services, actors and environments, while having the modified experiment configs to test a minimal experiment setup. All cogment_related artifacts are also in a separate working directory (tests/functional/.tmp_cogment_verse). Both temporary directories are cleaned out at the end of the test module.The functional tests are added to the phase:
testof the deployment process.Documentation
Updated the
development_setup.mdwith instructions to run functional tests.Other changes
CogmentVerseErrorclass. Used to catch when the mlflow server was not started or can't connect.