Rename test dirs

[ehmatthes]

When I started this project, I considered any test that didn't use remote resources a unit test. I considered test that use remote resources integration tests. Those are really integration tests, and e2e tests! Now that I have proper unit tests (in unit_tests_real/ :facepalm:), it's time to rename these.

Current tasks

• [x] Rename integration_tests/ -> e2e_tests/.
• [x] Set up bare pytest call to run all tests except e2e_tests.
• [x] Rename unit_tests/ -> integration_tests/.
• [x] Rename unit_tests_real/ -> unit_tests/.
• [x] Make sure all tests pass.
• [x] Consider a pytest --local arg that would run all tests in unit_tests/, and integration_tests/ (all tests that don't require remote resources). (not needed; bare pytest covers this now.)
• [x] Update docs just enough to let people run the kinds of tests they want to.
• [x] run all local tests on Windows.
  • [x] Needed to install toml.
• [x] Merge; no release required.
• [x] Move any unfinished tasks to separate issue.

Later tasks

• [ ] Write integration tests that cover an automate-all path, including one that's confirmed and one that's not confirmed?
• [ ] Consider a simple way to instantiate Command object for testing?
• [ ] Consider a simple way to instantiate Deployer object for testing, ie address arg issue that came up for _confirm_use_db() methods in #281?
• [ ] Consider local tests that cover automate-all paths?
• [ ] Pull common code out of e2e and integration tests?
• [ ] Add a check_log() function to each platform's integration test. (See fly_io integration test.)
• [ ] Add a line in integration test summary about log check results.

[ehmatthes]

Related tasks

• [ ] Add a check_log() function to each platform's integration test. (See fly_io integration test.)
• [ ] Add a line in integration test summary about log check results.

[ehmatthes]

make a tests/ dir

• [x] make tests/ dir
• [x] Make sure unit_tests_real/ pass.
• [x] Make sure unit_tests/ pass.

[ehmatthes]

integration_tests/ -> e2e_tests/

• [x] Rename the dir.
• [x] Rename obvious occurrences of the name.
• [x] Rename flag --integration-testing -> --e2e-testing.
• [x] Restructure root conftest.py. Want to run local tests, unless e2e tests is specified.
• [x] Run & debug.
• [x] Search integration, update comments to reflect e2e.

[ehmatthes]

bare pytest call runs all tests except e2e tests

• [x] Update root confest.py.
• [ ] Remove the None from unit_tests/conftest.py session.config.getoption()?

[ehmatthes]

parking lot

Message for canceling test run if it doesn't make sense for e2e tests:

if not check_valid_test_call():
    msg = "You must run either local tests or e2e tests, not both."
    msg += "\n  Either specify the tests you want to run, or cd"
    msg += "\n  into the appropriate directory and run tests from there."
    print(msg)

    sys.exit()

[ehmatthes]

Rename unit_tests/ -> integration_tests/

• [ ] Consider renaming --unit-testing flag to --local-testing, as it's used in integration tests.
  • [ ] Check if it's used in unit tests at all.
  • [ ] Consider whether there's any distinction between the two kinds of tests, and what their flags need to do.
  • [ ] Consider whether these flags can be removed entirely. Probably not; it basically tells simple_deploy not to create actual resources.
• [x] grep for "unit" in the entire integration_tests/ dir.

[ehmatthes]

Write integration tests that cover an automate-all path

Just trying this quickly, to see if it's easy to add.

• [ ] Fix: Seems to not be resetting between the automate and non-automated runs:

>       assert len(log_files) == 1   # update on friendly summary
E       AssertionError: assert 2 == 1
E        +  where 2 = len([PosixPath('

• [ ] Clean up the logic in run_simple_deploy().
• [ ] Look at confirmations for automate-all; I believe it's hanging on a confirmation?
• [ ] Clean up logic in get_confirmation() around testing.

Parking lot

• tests/integration_tests/conftest.py

@pytest.fixture(scope='module', autouse=True, params=[False, True])
def run_simple_deploy(reset_test_project, tmp_project, request):
    ...

    if request.param:
        cmd = f"python manage.py simple_deploy --automate-all"
    else:
        cmd = f"python manage.py simple_deploy"
    msp.call_simple_deploy(tmp_project, cmd, platform)

[ehmatthes]

Closing this issue

I'm way out of the mindset of this issue. Complete any critical open tasks so I can merge this, and resume overall development.

• [x] Unit tests pass?
• [x] Integration tests pass?
  • [x] Only Heroku tests fail.
  • [x] I don't think I have poetry installed, after updating all my Python versions.
  • [x] brew upgrade, brew upgrade pipx
  • [x] pipx uninstall poetry, rm ~/.local/bin/poetry, pipx install poetry.
• [x] E2E tests pass?
  • [x] Fly.io
  • [x] Platform.sh
  • [x] Heroku

[ehmatthes]

Update testing docs

• [x] Make a new "Testing" section in docs.
• [x] Make an overview page, a unit tests page, integration tests, and e2e tests.
• [x] Copy relevant information to each page.
  • [x] overview
  • [x] unit
  • [x] integration
    • [x] from "running tests"
    • [x] from "understanding tests"
  • [x] e2e
• [x] Remove old pages.