Features in Swift can be experimental, upcoming or base features. During their lifetime features evolve from one category to other categories. Usually experimental features are only available in "asserts" compilers, but they can also be experimental features in "non asserts" compilers. Upcoming and base features are available in both "asserts" and "non asserts" compilers.
When coding a new feature, the feature normally will start as an experimental feature. In the past this has required to mark the tests that use that feature with REQUIRES: asserts to avoid the test failing when testing in a "non asserts" compilers. This requisite is not always follow, and we forget to add those REQUIRES: asserts from time to time. This causes breakages for people that test the "non asserts" compilers. For some experimental features that are available in production compilers the REQUIRES: asserts is not even needed, which can make adding those lines work against one intentions. When the feature graduates from experimental to upcoming or base, we sometimes forget to update the the related tests to remove those REQUIRES: asserts, so a "non asserts" compiler will not actually execute those tests, and bugs can be introduced by mistake in those compilers.
The changes in this PR introduce a system that aims to simplify testing those experimental features and avoid some of the problems noted above.
The first change is take the canonical Features.def and transform its contents in something that LLVM Lit can create available_features for. This is done abusing the Clang preprocessor to transform the .def file into a Python file that can be loaded by lit.site.cfg during testing. This is done for each build and will pick up changes in the Features.def as they happen, so it will always be up-to-date. Additionally it understand when features as available depending on "asserts" or "non asserts" compilers, and will not incorrectly require an "asserts" compiler for non-production features, or let experimental features be tested in a "non asserts" compiler.
The second part of the change is keeping the tests up-to-date with the features they are testing, so each test that uses a feature is marked as such. This is done with a test itself, which greps through the existing tests, checks for the usage of -enable-experimental-feature or -enable-upcoming-feature and warns the user about the missing REQUIRES: lines (failing the test suite, so nobody can submit a test that skips the requirements).
Finally, the last change is modifying a huge number of tests to follow the new rules. All the tests that currently use -enable-experimental-feature or -enable-upcoming-feature have been annotated with REQUIRES: lines, and the (now unnecessary) REQUIRES: asserts have been removed.
Features in Swift can be experimental, upcoming or base features. During their lifetime features evolve from one category to other categories. Usually experimental features are only available in "asserts" compilers, but they can also be experimental features in "non asserts" compilers. Upcoming and base features are available in both "asserts" and "non asserts" compilers.
When coding a new feature, the feature normally will start as an experimental feature. In the past this has required to mark the tests that use that feature with
REQUIRES: asserts
to avoid the test failing when testing in a "non asserts" compilers. This requisite is not always follow, and we forget to add thoseREQUIRES: asserts
from time to time. This causes breakages for people that test the "non asserts" compilers. For some experimental features that are available in production compilers theREQUIRES: asserts
is not even needed, which can make adding those lines work against one intentions. When the feature graduates from experimental to upcoming or base, we sometimes forget to update the the related tests to remove thoseREQUIRES: asserts
, so a "non asserts" compiler will not actually execute those tests, and bugs can be introduced by mistake in those compilers.The changes in this PR introduce a system that aims to simplify testing those experimental features and avoid some of the problems noted above.
The first change is take the canonical
Features.def
and transform its contents in something that LLVM Lit can createavailable_features
for. This is done abusing the Clang preprocessor to transform the.def
file into a Python file that can be loaded bylit.site.cfg
during testing. This is done for each build and will pick up changes in theFeatures.def
as they happen, so it will always be up-to-date. Additionally it understand when features as available depending on "asserts" or "non asserts" compilers, and will not incorrectly require an "asserts" compiler for non-production features, or let experimental features be tested in a "non asserts" compiler.The second part of the change is keeping the tests up-to-date with the features they are testing, so each test that uses a feature is marked as such. This is done with a test itself, which greps through the existing tests, checks for the usage of
-enable-experimental-feature
or-enable-upcoming-feature
and warns the user about the missingREQUIRES:
lines (failing the test suite, so nobody can submit a test that skips the requirements).Finally, the last change is modifying a huge number of tests to follow the new rules. All the tests that currently use
-enable-experimental-feature
or-enable-upcoming-feature
have been annotated withREQUIRES:
lines, and the (now unnecessary)REQUIRES: asserts
have been removed.