VOLTTRON / openeis

Other
10 stars 15 forks source link

conflict between unit tests #3

Open dmlorenzetti opened 9 years ago

dmlorenzetti commented 9 years ago

There is a conflict between the unit tests. Running each test individually, they all pass. When run all at once, only the first one passes.

The conflict may relate to duplicate names on the output tables.

The following steps reproduce the problem, on both Mac and Windows machines, running in branch "1.x" (use changeset 7c57c071917b02b1e8ec5410f09b91292e2cf5e6 for reference):

(1) Verify that the first three unit tests work when run individually:

py.test applications/utest_applications/utest_daily_summary/test_daily_summary.py applications/utest_applications/utest_daily_summary/test_daily_summary.py .....

py.test applications/utest_applications/utest_energy_signature/test_energy_signature.py applications/utest_applications/utest_energy_signature/test_energy_signature.py ..

py.test applications/utest_applications/utest_heat_map/test_heat_map.py applications/utest_applications/utest_heat_map/test_heat_map.py ....

(2) Verify that all unit tests after the first (daily_summary) fail when run all at once:

py.test --tb=short applications/utest_applications/utest_daily_summary/test_daily_summary.py ..... applications/utest_applications/utest_energy_signature/test_energy_signature.py FF applications/utest_applications/utest_heat_map/test_heat_map.py FFFF etc.

(3) Remove or rename the first unit test, so that it will not execute:

mv applications/utest_applications/utest_daily_summary/test_daily_summary.py hold_tds.py

(4) Verify that what used to be the second unit test (energy_signature) now runs, while all else fail:

py.test --tb=short applications/utest_applications/utest_energy_signature/test_energy_signature.py .. applications/utest_applications/utest_heat_map/test_heat_map.py FFFF etc.

(5) Put things back to rights:

mv hold_tds.py applications/utest_applications/utest_daily_summary/test_daily_summary.py

The failures from lines like the following (this one from energy_signature.py): self.out.insert_row(LOAD_VS_OAT_TABLE_NAME, { "oat": x['oat'][0], "load": x['load'][0] }) with an error message like the following: E TypeError: 'load' is an invalid keyword argument for this function

Note that some later tests fail for other reasons. We are aware of, and working on, those applications.

fstshrk commented 9 years ago

Are you clearing the output tables as part of the text fixture before starting the next test?

The process should be:

1) Set up test 2) Run test 3) Clean up test artifacts 4) Go to step 1 if there are more tests to run.

Thanks

Bora

On 10/20/2014 4:07 PM, David M. Lorenzetti wrote:

There is a conflict between the unit tests. Running each test individually, they all pass. When run all at once, only the first one passes.

The conflict /may/ relate to duplicate names on the output tables.

The following steps reproduce the problem, on both Mac and Windows machines, running in branch "1.x" (use changeset 7c57c07 https://github.com/VOLTTRON/openeis/commit/7c57c071917b02b1e8ec5410f09b91292e2cf5e6 for reference):

(1) Verify that the first three unit tests work when run individually:

py.test
applications/utest_applications/utest_daily_summary/test_daily_summary.py
applications/utest_applications/utest_daily_summary/test_daily_summary.py
.....

py.test
applications/utest_applications/utest_energy_signature/test_energy_signature.py
applications/utest_applications/utest_energy_signature/test_energy_signature.py
..

py.test
applications/utest_applications/utest_heat_map/test_heat_map.py
applications/utest_applications/utest_heat_map/test_heat_map.py ....

(2) Verify that all unit tests after the first (daily_summary) fail when run all at once:

py.test --tb=short
applications/utest_applications/utest_daily_summary/test_daily_summary.py
.....
applications/utest_applications/utest_energy_signature/test_energy_signature.py
FF
applications/utest_applications/utest_heat_map/test_heat_map.py FFFF
etc.

(3) Remove or rename the first unit test, so that it will not execute:

mv
applications/utest_applications/utest_daily_summary/test_daily_summary.py
hold_tds.py

(4) Verify that what used to be the second unit test (energy_signature) now runs, while all else fail:

py.test --tb=short
applications/utest_applications/utest_energy_signature/test_energy_signature.py
..
applications/utest_applications/utest_heat_map/test_heat_map.py FFFF
etc.

(5) Put things back to rights:

mv hold_tds.py
applications/utest_applications/utest_daily_summary/test_daily_summary.py

The failures from lines like the following (this one from energy_signature.py): self.out.insert_row(LOAD_VS_OAT_TABLE_NAME, { "oat": x['oat'][0], "load": x['load'][0] }) with an error message like the following: E TypeError: 'load' is an invalid keyword argument for this function

Note that some later tests fail for other reasons. We are aware of, and working on, those applications.

— Reply to this email directly or view it on GitHub https://github.com/VOLTTRON/openeis/issues/3.

Help us find a cure for canine cancer. Please donate at: http://www.wearethecure.org/friends/skipper

hashstat commented 9 years ago

What I found is that when py.test is run directly the database is not setup and cleared correctly. To fix that, I run it through openeis which, for each test, sets up a temporary database in memory and inserts the fixtures. Try running the tests using openeis pytest.

I also found that to pass files and certain options to pytest, it must be done after a -- option, like so: openeis pytest -- applications/utest_applications/utest_daily_summary/test_daily_summary.py. Please give it a try and let us know how it goes.

Brandon

dmlorenzetti commented 9 years ago

Running openeis pytest gives me the same errors as described above (e.g., daily_summary passes and energy_signature fails, but removing daily_summary allows energy_signature to pass).

-Dave