swiftlang / swift

The Swift Programming Language
https://swift.org
Apache License 2.0
67.37k stars 10.34k forks source link

[test] Improve testing of Swift features #76740

Open drodriguez opened 1 day ago

drodriguez commented 1 day ago

Features in Swift can be experimental, upcoming or base features. During their lifetime features evolve from one category to other categories. Usually experimental features are only available in "asserts" compilers, but they can also be experimental features in "non asserts" compilers. Upcoming and base features are available in both "asserts" and "non asserts" compilers.

When coding a new feature, the feature normally will start as an experimental feature. In the past this has required to mark the tests that use that feature with REQUIRES: asserts to avoid the test failing when testing in a "non asserts" compilers. This requisite is not always follow, and we forget to add those REQUIRES: asserts from time to time. This causes breakages for people that test the "non asserts" compilers. For some experimental features that are available in production compilers the REQUIRES: asserts is not even needed, which can make adding those lines work against one intentions. When the feature graduates from experimental to upcoming or base, we sometimes forget to update the the related tests to remove those REQUIRES: asserts, so a "non asserts" compiler will not actually execute those tests, and bugs can be introduced by mistake in those compilers.

The changes in this PR introduce a system that aims to simplify testing those experimental features and avoid some of the problems noted above.

The first change is take the canonical Features.def and transform its contents in something that LLVM Lit can create available_features for. This is done abusing the Clang preprocessor to transform the .def file into a Python file that can be loaded by lit.site.cfg during testing. This is done for each build and will pick up changes in the Features.def as they happen, so it will always be up-to-date. Additionally it understand when features as available depending on "asserts" or "non asserts" compilers, and will not incorrectly require an "asserts" compiler for non-production features, or let experimental features be tested in a "non asserts" compiler.

The second part of the change is keeping the tests up-to-date with the features they are testing, so each test that uses a feature is marked as such. This is done with a test itself, which greps through the existing tests, checks for the usage of -enable-experimental-feature or -enable-upcoming-feature and warns the user about the missing REQUIRES: lines (failing the test suite, so nobody can submit a test that skips the requirements).

Finally, the last change is modifying a huge number of tests to follow the new rules. All the tests that currently use -enable-experimental-feature or -enable-upcoming-feature have been annotated with REQUIRES: lines, and the (now unnecessary) REQUIRES: asserts have been removed.

drodriguez commented 1 day ago

@swift-ci please test

drodriguez commented 1 day ago

@swift-ci please test

tshortli commented 1 day ago

This seems like a good idea to me.

drodriguez commented 15 hours ago

@swift-ci please test

drodriguez commented 6 hours ago

@swift-ci please test