tlakollo commented 2 years ago

Move linker repo to runtime repo

This issue tracks progress and discussion for moving the dotnet/linker repo into dotnet/runtime repo.

Progress

[x] Selected the location of the repository as runtime\src\tools\linker
[x] Ability to transfer Linker history to runtime using the consolidation script
[x] Add cecil as an arcade package in order to avoid having a submodule
- [x] Rename package to DotNet.Cecil
[x] Be able to run dotnet format to apply runtime formatting to linker repo
[x] Create subsets (tools.linker & tools.linkertests) in order to be able to run and test the repo
[x] Run in CI
[x] Add additional label
[x] Rename linker to illink
[x] Build a package from the new location or use live bits

Follow-up work

Motivation

Code Sharing. At this moment NativeAOT and linker share a lot of code via a straight copy. NativeAOT consumes the ILLink.Shared project which is a shared project between linker, trimming analyzer and NativeAOT. It also shares the same testing infrastructure and even shares additional files that are not included in the shared project since they are only shared between linker and NativeAOT (not for the trimming analyzer). The plan is that once we move the linker repository to runtime we can take advantage of having only one ILLink.Shared project, only one testing infrastructure, and we can improve code sharing between linker and NativeAOT, even having the same formats and analyzers applied to the codebases improve the current state of code sharing.
Versioning history. Backward compatibility of tools like linker and NativeAOT is an ongoing issue (see https://github.com/dotnet/linker/issues/3029) having both repositories in the runtime simplifies the solution for both of the tools.

Select the location for the linker project

So far there have been some locations that seem relevant

src\coreclr\tools\ folder, although it would be closer to the AOT tools linker is not strictly a part of the coreclr
Create a new folder under src\tools for tools that are agnostic to the clr. Likely the best option.
Maybe somewhere in the mono partition?

Avoiding having a cecil submodule

Part of the linker workflow is to have a .gitmodules file that specifies a mono/cecil module, for runtime this won't fit the runtime needs. So we need to maybe ship cecil packages from mono/cecil and consume them in the runtime to later be used by linker

Linker style of code format

Currently, linker repo uses mono style of code. This makes already codesharing with NativeAOT more difficult since we cannot share code as is. Linker already transition from the mono space to the dotnet space, which leaves in the air if it should still use the mono guidelines. Once moving to the runtime repo and giving that we want to share code, the most likely solution is to just use runtime format for that repo. Note: mono directories under runtime still use the mono style of code guidelines.

Legs for execution

Need a definition on when should ci build linker and run linker tests. Most likely will remain as a subset just like currently clr.tools and clr.toolstests, and run only if changes in the linker repo are made. For source build, building and testing the linker repo seems inevitable unless we create a different leg that only runs linker.

Add an additional label

Add an additional label to runtime to identify linker issues

Keep a history of the linker changes

There are some scripts that were used to consolidate different repositories into runtime, although they are tailored to work with certain repositories we could configure it to keep the history from linker (see https://github.com/dotnet/consolidation)

Migration of github issues

Previous runtime consolidation work had the help of GitHub to migrate all issues from the different repos to the new dotnet/runtime repo, and then lock down creation of additional issues, such that the repos will no longer be used for issue management. They also inhibit the creation of PRs to the main branch, effectively making old repos an archive for read-only review of history. Plan is to do the same for linker

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

tlakollo commented 2 years ago

cc @dotnet/linker-contrib @dotnet/ilc-contrib

tlakollo commented 2 years ago

cc @marek-safar @radekdoulik

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-meta See info in area-owners.md if you want to be subscribed.

Issue Details

# Move linker repo to runtime repo This issue tracks progress and discussion for moving the dotnet/linker repo into dotnet/runtime repo. ## Select the location for the linker project So far there have some locations that seem relevant - `src\coreclr\tools\` folder, although it would be closer to the AOT tools linker is not strictly a part of the coreclr - Create a new folder under `src\tools` for tools that are agnostic to the clr. Likely the best option. - Maybe somewhere in the mono partition? ## Keep a history of the linker changes There are some scripts that were used to consolidate different repositories into runtime, although they are tailored to work with certain repositories we could configure it to keep the history from linker (see https://github.com/dotnet/consolidation) ## Avoiding having a cecil submodule Part of the linker workflow is to have a `.gitmodules` file that specifies a `mono/cecil` module, for runtime this won't fit the runtime needs. So we need to maybe ship cecil packages from mono/cecil and consume them in the runtime to later be used by linker ## Linker style of code format Currently, linker repo uses mono style of code. This makes already codesharing with NativeAOT more difficult since we cannot share code as is. Linker already transition from the mono space to the dotnet space, which leaves in the air if it should still use the mono guidelines. Once moving to the runtime repo and giving that we want to share code, the most likely solution is to just use runtime format for that repo. Note: mono directories under runtime still use the mono style of code guidelines. ## Legs for execution Need a definition on when should ci build linker and run linker tests. Most likely will remain as a subset just like currently clr.tools and clr.toolstests, and run only if changes in the linker repo are made. For source build, building and testing the linker repo seems inevitable unless we create a different leg that only runs linker. ## Add an additional label Add an additional label to runtime to identify linker issues

Author:	tlakollo
Assignees:	-
Labels:	`area-Meta`
Milestone:	8.0.0

jkotas commented 2 years ago

Create a new folder under src\tools

That looks better to me.

we need to maybe ship cecil packages from mono/cecil and consume them in the runtime to later be used by linker

We need to make sure that this is compatible with our servicing policies and source build. If we were to keep cecil in separate repo, I expect that we would need to move it to dotnet/cecil at least.

vitek-karas commented 2 years ago

Moving Cecil to runtime repo is problematic. So far we've used mono/cecil as only a way to ship predictable bits and have a way to quickly fix issues without waiting for upstream merge. But we always send all our changes upstream and most of the time mono/cecil and upstream are identical. If we were to move cecil into runtime repo, it would have to be a mirror, which seems impractical.

marek-safar commented 2 years ago

We need to make sure that this is compatible with our servicing policies and source build. If we were to keep cecil in separate repo, I expect that we would need to move it to dotnet/cecil at least.

I don't think mono/ org should be a problem for source build (we have or had sources used by source build there before) but I have no objection to creating another fork under dotnet/ org. We just need to be careful about versioning of the internal cecil package/dlls.

ghost commented 1 year ago

Tagging subscribers to this area: @agocke, @sbomer, @vitek-karas See info in area-owners.md if you want to be subscribed.

Issue Details

# Move linker repo to runtime repo This issue tracks progress and discussion for moving the dotnet/linker repo into dotnet/runtime repo. ## Progress - [x] Selected the location of the repository as runtime\src\tools\linker - [x] Ability to transfer Linker history to runtime using the consolidation script - [x] Add cecil as an arcade package in order to avoid having a submodule - [x] Rename package to DotNet.Cecil - [x] Be able to run dotnet format to apply runtime formatting to linker repo - [x] Create subsets (tools.linker & tools.linkertests) in order to be able to run and test the repo - [x] Run in CI - [x] Add additional label - [x] Rename linker to illink - [x] Build a package from the new location or use live bits ## Follow-up work - [ ] https://github.com/dotnet/runtime/issues/78050 - [x] https://github.com/dotnet/runtime/issues/78059 - [x] https://github.com/dotnet/runtime/issues/78334 - [ ] https://github.com/dotnet/runtime/issues/77868 - [ ] https://github.com/dotnet/runtime/issues/78344 - [x] https://github.com/dotnet/runtime/issues/80319 - [x] https://github.com/dotnet/runtime/pull/80407 - [x] https://github.com/dotnet/runtime/pull/80406 ## Motivation - **Code Sharing**. At this moment NativeAOT and linker share a lot of code via a straight copy. NativeAOT consumes the ILLink.Shared project which is a shared project between linker, trimming analyzer and NativeAOT. It also shares the same testing infrastructure and even shares additional files that are not included in the shared project since they are only shared between linker and NativeAOT (not for the trimming analyzer). The plan is that once we move the linker repository to runtime we can take advantage of having only one ILLink.Shared project, only one testing infrastructure, and we can improve code sharing between linker and NativeAOT, even having the same formats and analyzers applied to the codebases improve the current state of code sharing. - **Versioning history**. Backward compatibility of tools like linker and NativeAOT is an ongoing issue (see https://github.com/dotnet/linker/issues/3029) having both repositories in the runtime simplifies the solution for both of the tools. ## Select the location for the linker project So far there have been some locations that seem relevant - `src\coreclr\tools\` folder, although it would be closer to the AOT tools linker is not strictly a part of the coreclr - Create a new folder under `src\tools` for tools that are agnostic to the clr. Likely the best option. - Maybe somewhere in the mono partition? ## Avoiding having a cecil submodule Part of the linker workflow is to have a `.gitmodules` file that specifies a `mono/cecil` module, for runtime this won't fit the runtime needs. So we need to maybe ship cecil packages from mono/cecil and consume them in the runtime to later be used by linker ## Linker style of code format Currently, linker repo uses mono style of code. This makes already codesharing with NativeAOT more difficult since we cannot share code as is. Linker already transition from the mono space to the dotnet space, which leaves in the air if it should still use the mono guidelines. Once moving to the runtime repo and giving that we want to share code, the most likely solution is to just use runtime format for that repo. Note: mono directories under runtime still use the mono style of code guidelines. ## Legs for execution Need a definition on when should ci build linker and run linker tests. Most likely will remain as a subset just like currently clr.tools and clr.toolstests, and run only if changes in the linker repo are made. For source build, building and testing the linker repo seems inevitable unless we create a different leg that only runs linker. ## Add an additional label Add an additional label to runtime to identify linker issues ## Keep a history of the linker changes There are some scripts that were used to consolidate different repositories into runtime, although they are tailored to work with certain repositories we could configure it to keep the history from linker (see https://github.com/dotnet/consolidation) ## Migration of github issues Previous runtime consolidation work had the help of GitHub to migrate all issues from the different repos to the new dotnet/runtime repo, and then lock down creation of additional issues, such that the repos will no longer be used for issue management. They also inhibit the creation of PRs to the main branch, effectively making old repos an archive for read-only review of history. Plan is to do the same for linker

Author:	tlakollo
Assignees:	tlakollo
Labels:	`area-Tools-ILLink`
Milestone:	8.0.0

agocke commented 1 year ago

Closing as complete

dotnet / runtime