facebookincubator / cinder

Cinder is Meta's internal performance-oriented production version of CPython.
https://trycinder.com
Other
3.49k stars 121 forks source link

Add Lazy Imports test suite framework #84

Closed chuanzzx closed 2 years ago

chuanzzx commented 2 years ago

Summary

Test Plan

Notes: All these tests were executed in cinder/3.10. This is a crashy branch, so somethings might not work here.

1. Test the behavior in test_lazy_imports.py

We will cover the testing of these four features in this section.


When I remove @unittest.skipIfLazyImportsIsDisabled (line 8) and @unittest.skipIfLazyImportsIsEnabled (line 12) in test_lazy_imports.py, no matter this test is executed with or without Lazy Imports, the unit test will fail.

image

With Lazy Imports

Run

./python -L -m test test_lazy_imports

Expected results

It will fail in test_lazy_imports_is_disabled because Lazy Imports is enabled.

image

Without Lazy Imports

Run

./python -m test test_lazy_imports

Expected results It will fail in test_lazy_imports_is_enabled because Lazy Imports is not enabled.

image

When I add @unittest.skipIfLazyImportsIsDisabled (line 8) and @unittest.skipIfLazyImportsIsEnabled (line 12) into test_lazy_imports.py, the unit test can be passed with or without Lazy Imports.

image(same as the original test_lazy_imports.py )

With Lazy Imports

Run

./python -L -m test test_lazy_imports

Expected results

image

Without Lazy Imports

Run

./python -m test test_lazy_imports

Expected results

image

2. Test the framework

We add a new target, test_with_lazy_imports, in Makefile, so that we can easily validate the test suite with Lazy Imports by using make test_with_lazy_imports.

Test suite with Lazy Imports

If we run make test_with_lazy_imports, we would like to make every testing execute with Lazy Imports. To validate this, I remove some code in test_lazy_imports.py. We only leave test_lazy_imports_is_enabled.

image

Then, we can observe if it still can pass test_lazy_imports.py. If it passes, all the tests are executed with Lazy Imports. Otherwise, make test_with_lazy_imports failed.

Run

make test_with_lazy_imports

Expected results We expected test_lazy_imports did not fail during the validation. That is, it didn't show in the tests failed list.

image

Test suite without Lazy Imports

This test suite existed in the code. We could use make test to execute it, and the results should be the same as before.

chuanzzx commented 2 years ago

Updated the reasons in Lib/test/test_lazy_imports.py.