Tracking Issue for compiletest documentation

[jieyouxu]

Context

compiletest is the main test harness of rustc and other tools inside this repository, but there is a significant lack of documentation for it. We should write some documentation for compiletest. There are some documentation in rustc-dev-guide, but compiletest is a lot more complex than that.

Areas lacking documentation

• [ ] High-level architecture and overview
• [ ] Interaction between bootstrap <-> compiletest <-> tests
• [ ] compiletest-as-a-binary: CLI interface options and docs
• [ ] Filtering tests
• [ ] How are test modes and test suites created, registered, built and run
  • When can tests be ignored? cached test results and their invalidation: dependency stamps
  • Some test mode / test suite tests can be run in parallel / concurrently:
    • Concurrency assumptions: what if the test modifies external state?
• [ ] Test mode and test suite specific behavior, assumptions and quirks
  • UI tests: auxiliary crates and bins, compile-flags, compare modes, assumptions and conventions on stdout/stderr output, blessing
  • run-make and run-make-fulldeps tests: how the whole build pipeline is setup, concepts, how tests are built and executed, provided env vars, assumed dependencies
  • codegen tests: filecheck, revisions magic behavior
  • coverage tests: how to bless?
  • ui-fulldeps have some non-trivial behavior: e.g. https://github.com/rust-lang/rust/pull/126073#discussion_r1629843686
• [ ] Error handling strategy/design and diagnostics
• [ ] Interaction with CI
• [ ] Test directive collection, parsing and validation
  • Which directives can be used in which test suites?
  • What do the directives even mean/do?
• [ ] Test output normalization special syntax
  • e.g. const CWD: &str = "{{cwd}}";
• [ ] Setting current_dir when running tests

Related issues

• https://github.com/rust-lang/rust/issues/125585
• https://github.com/rust-lang/rust/issues/125587
• https://github.com/rust-lang/rust/issues/126092 (we should document what debuggers we assume for debuginfo tests)

Unresolved questions

• Where do we keep compiletest docs?
  • Maybe alongside the source? Maybe a mdBook? rustc-dev-guide?
  • Ideally we would like this easy to read and reference and improve.
• How to we keep compiletest documentation update-to-date?
  • Periodic triage?

Documentation improvement history

TODO

[Zalathar]

• 126180