Both the original Python Page Objects framework (pre-"pysaunter") and the very latest version of pysaunter (0.12) present our QA group with the same reporting issue. We run the smoke tests sequentially on our staging environment (sometimes more than once if staging gets updated) and then once more on production in a big hurry right after we've pushed the new code there. If a test fails or times out, we give it a second chance by re-running it individually, before we spend any time investigating. The problem is that we don't know which test is producing the "E" or "F" until after the entire run has completed. We'd like to know immediately which test has produced the "E" or "F" so that we could be re-running that test asap.
Below is output from running the pysaunter 0.12 saucelabs examples that show the issue. Instead of ".F....", we'd like to see something more like:
Both the original Python Page Objects framework (pre-"pysaunter") and the very latest version of pysaunter (0.12) present our QA group with the same reporting issue. We run the smoke tests sequentially on our staging environment (sometimes more than once if staging gets updated) and then once more on production in a big hurry right after we've pushed the new code there. If a test fails or times out, we give it a second chance by re-running it individually, before we spend any time investigating. The problem is that we don't know which test is producing the "E" or "F" until after the entire run has completed. We'd like to know immediately which test has produced the "E" or "F" so that we could be re-running that test asap.
Below is output from running the pysaunter 0.12 saucelabs examples that show the issue. Instead of ".F....", we'd like to see something more like:
test_incorrect_login . test_incorrect_login_fails F test_incorrect_login_from_csv . test_incorrect_login_from_sqlite3 . test_incorrect_login_with_random_username_and_password . test_incorrect_login_with_soft_assert .
adobe-MacBookPro:saucelabs mmaypump$ pysaunter.py -f login ================================================================== test session starts =================================================================== platform darwin -- Python 2.7.2 -- pytest-2.1.1 collected 6 items
scripts/LoginExample.py .F....
======================================================================== FAILURES ======================================================================== /Users/mmaypump/Desktop/PYSAUNTER-20110920/examples/saucelabs/scripts/LoginExample.py:93: AssertionError: 'Incorrect username or password.' != 'This message is deliberately incorrect to trigger a failed test.' ---------------------- generated xml file: /Users/mmaypump/Desktop/PYSAUNTER-20110920/examples/saucelabs/logs/2011-09-20-35-32.xml ----------------------- ========================================================== 1 failed, 5 passed in 101.52 seconds ========================================================== adobe-MacBookPro:saucelabs mmaypump$