Beginning of acceptance test framework

panahiparham / ml-experiment-definition

MIT License

2 stars 3 forks source link

Beginning of acceptance test framework #12

Closed yasuiniko closed 4 weeks ago

yasuiniko commented 1 month ago

This is the start of an acceptance test using mountaincar and softmaxAC. So far I have done:

Added tests/acceptance/softmaxAC_mc.py with a test that writes to a database using DefinitionPart and reads from it using ExperimentDefinition.
Added a unit test tests/test_generate_configurations.py, which tests the function in DefinitionPart.py.

yasuiniko commented 1 month ago

The tests are failing because Scheduler imports typing.Self, which was implemented in Python 3.11. The import is not from my code, it is from main. Should we stop testing python 3.10?

panahiparham commented 1 month ago

split styling changes and tests
maybe remove python 3.10 from testing?
make sure local run actually runs everything

yasuiniko commented 1 month ago

@andnp we need your advice here. The Github workflow tests for this PR are failing because of two issues

Scheduler imports typing.Self, which is only introduced with python 3.11. The workflow tests include python 3.10.
The user's experiment.py cannot import ml_experiment since it is not pip-installed during the github workflow, and ml_experiment is not in sys.path. See for more details: #20

I can fix both of these issues by editing .github/workflow/tests.yml, but I wanted to make sure you think this is a good idea first.

andnp commented 4 weeks ago

Yup, I think editing the workflow spec to solve both of these is a great strategy! I'm quite comfortable dropping 3.10 support

yasuiniko commented 4 weeks ago

Below tests are those on the main branch, the PR has an updated test.yaml and those pass when run locally through https://github.com/nektos/act. This fixes #20.