Turn stdarch into a subtree

[RalfJung]

stdarch is one of the few remaining big submodules in the rustc repo. And it's painful -- stdarch depends on a lot of very internal rustc implementation details, but refactoring / improving those details always requires a complicated multi-stage process to ensure stdarch keeps building. It would be so much easier if the two could be updated in lock-step, but with a submodule that's not very practical. I ran into this twice just recently: as part of https://github.com/rust-lang/rust/issues/128738, and as part of https://github.com/rust-lang/rust/pull/131349. To ensure our continued ability of maintaining and improving rustc, we should fix this situation.

Miri, clippy, portable-simd, RA are all using subtrees, using either git subtree or josh. git subtree has some bad interacts with git blame due to the way it mirrors the changes between the repo. josh has the downside that many of the merge commits from the rustc repo end up being mirrored into the subrepo (see https://github.com/josh-project/josh/issues/1328). I don't care very strongly which is used, though the general trend has been away from git subtree and towards josh (RA recently migrated, clippy decided to migrate but the effort seems currently stalled). If you pick josh, I can help with the migration. (I have some migration notes here but those are for a migration from git subtree to josh. Migrating from submodules will look a bit different.) You'll likely want some maintainer scripts in the stdarch repo to simplify pushing and pulling changes from and to the rustc repo; cargo xtask is probably the easiest way to set that up (that's what RA uses).

Last time I brought this up with @Amanieu, they were concerned about testing changes that land via rustc, since not the full stdarch test suite runs in rustc CI. I don't know how much testing in rustc CI is required to alleviate these concerns -- would be good to get a clear list of requirements. And then I hope someone can get motivated to work on this so that stdarch can stop being a roadblock for rustc evolution. :)

[lu-zero]

I think the main issue is writing the logic that triggers the stdarch CI bits only when something is changed on here.

git subtree seems available out of box, not sure how bad is the git blame issue or how difficult is to setup josh w/out docker so I cannot tell which is nicer to use. I noticed that git-subrepo seems available.

[RalfJung]

I think the main issue is writing the logic that triggers the stdarch CI bits only when something is changed on here.

Generally we don't do this in rust-lang CI since it is rather fragile -- it is easy to miss some file that can make tests fail, and then it can be tricky to debug when a failure actually got introduced. CI should be stateless -- it should ensure that the repo satisfies certain checks, in a repeatable way that does not depend on past history. Miri and clippy for example get tested on rustc CI each time, not just when their subtrees change.

how difficult is to setup josh w/out docker

I use josh without docker, cargo install is all it takes.

git subtree seems available out of box

AFAIK one has to use a patched version to make it usable with large repos like rustc, otherwise it is way too slow.

I noticed that git-subrepo seems available.

I have not tried this one. It seems to be squashing the subrepo history in the main repo into a single large commit for each pull, whereas git subtree and josh both preserve individual subrepo commits.

[RalfJung]

Cc @calebzulawski @flip1995 @lnicola (for portable-simd, clippy, RA) in case they have anything to add to the choice of tool here.

[calebzulawski]

My only comment is that I often find myself deleting the branch and recreating it when something goes a little awry with git-subtree. I haven't tried josh yet but it seems possibly easier to use.

[lu-zero]

Then I guess josh is a winner :)