This PR adds an option to pytest, --runcost, that enables all tests marked with @pytest.mark.cost; if the option isn't present, those tests are skipped.
Specifically, it adds a configuration file tests/conftest.py that defines the --runcost CLI argument, the cost marker, and the logic that skips the tests marked cost unless --runcost was specified. It also decorates all tests in test/test_model_revision.py with the cost marker, since as far as I can tell they all use the OpenAI API.
Even though we intend to drop the tests that cost money or migrate them to a different testing framework, it was quick for me to implement it, so I figured it'd be a good stopgap until we move the tests. I also thought it would be good to establish the custom marker system for other uses.
Addresses #25.
This PR adds an option to pytest,
--runcost, that enables all tests marked with@pytest.mark.cost; if the option isn't present, those tests are skipped.Specifically, it adds a configuration file
tests/conftest.pythat defines the--runcostCLI argument, thecostmarker, and the logic that skips the tests markedcostunless--runcostwas specified. It also decorates all tests intest/test_model_revision.pywith thecostmarker, since as far as I can tell they all use the OpenAI API.Even though we intend to drop the tests that cost money or migrate them to a different testing framework, it was quick for me to implement it, so I figured it'd be a good stopgap until we move the tests. I also thought it would be good to establish the custom marker system for other uses.
(Credit to @d33bs for the idea.)