search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
97.84k stars 12.66k forks source link

Tracking Issue for porting `run-make` tests to use Rust #121876

Open jieyouxu opened 7 months ago

jieyouxu commented 7 months ago

We want to stop run-make tests from relying on make, and improve the run-make tests so that they are more accessible to rustc contributors by allowing the tests to be written in Rust (see

40713 for context). PR #113026 was merged to address this, and now we are able to write run-make

tests in Rust recipes. We would like your help to port over existing run-make tests still using Makefiles to use Rust recipes instead.

If you would like to work on porting one of the run-make tests, please link to this issue in your PR and leave a comment to claim the test (or multiple tests). If you are stuck, please don't hesitate to open a thread on Rust's Zulip.

run-make is the most flexible fallback test kind, but some run-make tests could be migrated to become e.g. ui tests instead. Please check if you can convert the run-make test into other test kinds before porting!

When you try to port a test, also consider:

Context: Rust recipes?

PR #113026 adds basic infrastructure support to write run-make tests using small Rust programs, called recipes.

We aim to eliminate the dependency on make and Makefiles for building run-make-style tests. Makefiles are replaced by recipes (rmake.rs). The PR implements running run-make recipes in 3 steps:

  1. We build the support library run_make_support which the rmake.rs recipes depend on as a tool lib.
  2. We build the recipe rmake.rs and link in the support library.
  3. We run the recipe to build and run the tests.

rmake.rs is basically a replacement for Makefile, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired.

The infrastructure implemented by the PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example run-make tests ported over to the new infrastructure:

You likely will find that you would need to improve the API of the support library, and extend the functionality of the support library.

Common traps and pitfalls, and tips and tricks

Tests that need porting

5225225 commented 7 months ago

claiming exit-code, looks simple enough

abhay-51 commented 7 months ago

claiming the alloc tests (no oom handling, no rc, no sync)

high-cloud commented 7 months ago

claiming the hir-tree

0xhiro commented 7 months ago

claiming archive-duplicate-names

JoverZhang commented 7 months ago

Hi, I want to claim the no-input-file. And I'll add the diff support.

Oneirical commented 6 months ago

Claimed core-no-fp-fmt-parse in my PR, #123180.

Oneirical commented 6 months ago

Claimed version in my PR, https://github.com/rust-lang/rust/pull/123297.

Zalathar commented 4 months ago

I’ve been making good progress in clearing out run-make-fulldeps:

There’s one left that I’ve also written a patch for, but it’s a bit more complex so I’ll have to figure out whether it’s worthwhile.

EDIT: PR filed for the final test! (#126111)

Zalathar commented 4 months ago

All of the run-make-fulldeps tests have been ported to ui-fulldeps (or removed), so I've filed #126155 to remove the suite entirely.

lolbinarycat commented 4 months ago

claiming llvm-outputs

lolbinarycat commented 4 months ago

llvm-outputs has been migrated https://github.com/rust-lang/rust/pull/126211

Zalathar commented 4 months ago

I ported one of the run-make tests that actually just delegates to a Python script (#126231), though at the last moment I noticed that I forgot one piece, so it's still a work in progress.

Zalathar commented 4 months ago

I also have a port of tests/run-make/libtest-json that I've been working on recently, but haven't filed a PR for yet.

EDIT: Along the way I ran into some compiletest issues:

Rejyr commented 4 months ago

Claiming comment-section.

Kobzol commented 3 months ago

@0xhiro Not sure if you're still working on it, but just FYI, archive-duplicate-names is being ported in https://github.com/rust-lang/rust/pull/127825.

lolbinarycat commented 2 months ago

claiming extern-flag-disambiguates

lolbinarycat commented 2 months ago

extern-flag-disambiguates ported in #128099

lolbinarycat commented 2 months ago

resolve-rename is listed twice, once as completed, and once as not completed (in reality, it has been completed)

Kobzol commented 2 months ago

Thanks! Fixed.

ChrisDenton commented 2 months ago

I don't think tools.mk should be on the list because it doesn't contain any actual tests to be ported (as far as I'm aware). It's just a utility script that's used by actual tests. Though maybe it's useful to note it as to be deleted once the final test is ported?

jieyouxu commented 2 months ago

I don't think tools.mk should be on the list because it doesn't contain any actual tests to be ported (as far as I'm aware). It's just a utility script that's used by actual tests. Though maybe it's useful to note it as to be deleted once the final test is ported?

Err yeah, the list was generated via ls lol, I think I'll keep the entry around until we finally get rid of it.

lolbinarycat commented 2 months ago

claiming extern-fn-reachable to try the new symbol iterator functions

lolbinarycat commented 2 months ago

claiming symbol-mangling-hashed

Oneirical commented 2 months ago

incr-prev-body-beyond-eof and suspicious-library are already ported, they should be checked off.

Kobzol commented 2 months ago

Thanks, fixed. They were included twice in the list.

Oneirical commented 2 months ago

rust-lld and rust-lld-custom-target are already ported, they should be checked off.

Kobzol commented 2 months ago

Thanks, fixed.

lolbinarycat commented 2 months ago

cat-and-grep-sanity-check can probably be deleted along with tools.mk once migration is complete

Oneirical commented 2 months ago

valid-print-requests in the checklist does not exist.

Kobzol commented 2 months ago

Thanks! It was turned into a UI test in https://github.com/rust-lang/rust/pull/124333.

Zalathar commented 2 months ago

Explicitly claiming libtest-junit alongside libtest-json, since they're very similar, and I've decided to go with a straightforward rmake port for now.

petrochenkov commented 2 months ago

@Oneirical @jieyouxu Tests doing codegen for specific targets need //@ needs-llvm-components directives. These are the tests that are currently missing the directives and failing if LLVM is built without various aarch64/arm/thumb components or wasm.

    [run-make] tests\run-make\print-target-list
    [run-make] tests\run-make\print-to-output
    [run-make] tests\run-make\print-cfg
    [run-make] tests\run-make\target-without-atomic-cas
jieyouxu commented 1 month ago

@Oneirical @jieyouxu Tests doing codegen for specific targets need //@ needs-llvm-components directives. These are the tests that are currently missing the directives and failing if LLVM is built without various aarch64/arm/thumb components or wasm.

    [run-make] tests\run-make\print-target-list
    [run-make] tests\run-make\print-to-output
    [run-make] tests\run-make\print-cfg
    [run-make] tests\run-make\target-without-atomic-cas

Thanks for the heads-up, I'll revisit them.

Zalathar commented 1 week ago

Summary of the current remaining Makefiles:

Test name PR Problem/status
branch-protection-check-IBT #126720 Never actually runs due to a bug in the Makefile itself!
cat-and-grep-sanity-check #129156 Meta test; can be deleted when no remaining Makefiles use $(CGREP)
extern-fn-reachable #128314 PR has outstanding concerns; waiting on author?
incr-add-rust-src-component #128562 Windows symlink horrors? #129155
issue-84395-lto-embed-bitcode #128562 Doesn't actually run anywhere? #126180, #126375
jobserver-error #128789 Waiting for review? Looks tricky
libs-through-symlinks #129011 Waiting for review?
split-debuginfo #128754 Very large rewrite of a very large Makefile; waiting for review?
symbol-mangling-hashed #128567 Blocked on extern-fn-reachable?
translation #129011 Same as libs-through-symlinks