[darc/maestro++] Darc should be buildable from source

[dagood]

I noticed issues about using Octokit https://github.com/dotnet/arcade-services/issues/2580 and VSTS C# libs https://github.com/dotnet/arcade-services/issues/2579 in Darc. This sounds like a great idea, but I want to make sure it's implemented in a way that leaves Darc able to build from source.

If Octokit and VSTS APIs can build from source, ok, but it would be preferable not to build them at all. The builds take time, and the capabilities won't be used in source-build.

/cc @alexperovich @jcagme

---

I haven't looked into every one, but these are unknowns when it comes to source-build:

• Microsoft.AspNet.WebApi.Client
• Microsoft.Azure.KeyVault
• Microsoft.Azure.Services.AppAuthentication
• Microsoft.IdentityModel.Clients.ActiveDirectory
• Microsoft.Rest.ClientRuntime
• Microsoft.Rest.ClientRuntime.Azure
• Microsoft.TeamFoundation.DistributedTask.Common.Contracts
• Microsoft.TeamFoundationServer.Client
• Microsoft.VisualStudio.Services.Client
• Microsoft.VisualStudio.Setup.Configuration.Interop
• Newtonsoft.Json.Bson
• Octokit
• YamlDotNet.Signed

An easy way I see to do this is to split code that needs to use these dependencies into separate csproj's, condition the dependency, and condition some small part of the code to link in (or not) web client capability. (Microsoft.DotNet.Maestro.Client is an example of something that's already split off!) The property ExcludeFromSourceBuild is used to skip a project during source-build.

I can imagine something fancier being done, where there's a real extensibility point on Darc. The user could choose to download some extra packages (or something) to enable web client capabilities.

[alexperovich]

I agree. Does darc even need to be buildable from source? What darc functionality would be used in the source-build scenario?

[dagood]

source-build will use darc for https://github.com/dotnet/arcade/issues/427 "Create a product source build layout from a repo sha" but even if that's done "before" the build from source, I'm pretty sure we'll be using darc to flow dependencies between repos. /cc @mmitche

[jcagme]

@dagood I know you mentioned that you'll try building Arcade from source in the following two weeks, I assume this includes darc. Would you consider any potential issues in the process to be preview (12/14) blockers?

[dagood]

Doing a "proper" build from source (no prebuilts) isn't a blocker for 3.0 preview, so no.

That said, the sooner we can build Darc from source rather than downloading it, the better, to avoid building up debt that ends up delaying the delivery of a fully source-built .NET Core. (And yeah, Darc is part of the required toolset, so it's included.) When I try building Arcade from source I'm planning to file issues to get this on the radar.

[dagood]

With Darc in particular, I don't see any source-buildability issues yet beyond the dependency stuff I already mentioned in the issue description, so I just added some more info and more detail on the design suggestion.

[markwilkie]

@dagood - do you feel this requirement is fulfilled yet? If not, what do you see is missing?

[dagood]

AFAIK none of the work discussed in this thread has been done.

This was discussed in a email thread ~11/29 "Source-buildability of Darc's local capabilities" with @ChadNedzlek, @mmitche, and @jcagme, so the stuff to do is still known.

[markwilkie]

@mmitche - what's needed for Preview II?

[dagood]

From the source-build perspective, this is not needed for preview 2. We'll need it to be able to deliver source-build on new distros without prebuilts, which I don't think is tied to any specific 3.0.0 milestones.

[ChadNedzlek]

Is "darc" necessary to do anything more than fetching the correct commits/source blobs? Presumably that could be run on something other than the target computer (since it's not actually "the build"... it's just getting the source code), then that thing that's created would be either shipped off to distros like RedHat or built on the target computer... but at that point nothing in darc would be necessary. If there is no prebuilt stuff, darc won't be able to run anyway, so it doesn't help in that scenario either (since it requires a full functioning dotnet core to run).

Darc is a pretty intense mess of dependencies and pulling it apart is going to be painful and awkward and difficult to maintain. I'm hoping we can find a more sustainable solution that doesn't drag the web service into some dependency cycle.

[dagood]

Is "darc" necessary to do anything more than fetching the correct commits/source blobs?

Yes, when I mentioned "I'm pretty sure we'll be using darc to flow dependencies between repos." I'm referring to this:

• Darc is meant to be the only way developers and other tools like Maestro++ interact and alter version/dependency files as well as bootstrapping files and scripts in arcade participating repos.
• Darc's operations range from altering version/dependency files to creating PRs in specified repos.

Currently we use the Repo API (specifically Package Version Props) to do this, and from the above it sounds like using Package Version Props won't be supported. (And we might still need Darc for a standard way to flow zips through?)

Presumably that could be run on something other than the target computer (since it's not actually "the build"... it's just getting the source code), then that thing that's created would be either shipped off to distros like RedHat or built on the target computer... but at that point nothing in darc would be necessary.

That's correct, to me the reliance on Microsoft-prebuilt bits to do the trivial-seeming thing of gathering the source makes community maintenance less sustainable, but I don't have justification for this put together right now.

If there is no prebuilt stuff, darc won't be able to run anyway, so it doesn't help in that scenario either (since it requires a full functioning dotnet core to run).

This isn't the case: the N-1 SDK and Runtime aren't prebuilts. When there is no N-1, first-time bootstrapping has specific exceptions, but we can only use stuff we can build, not e.g. Octokit and Azure packages. If we can bring those into source-build, sweet, but that seems like a lot more maintenance cost than cutting them out of the build.

If you're interested, EngineeringPlanLinuxDistroRepos.md should clarify the requirements and N-1/bootstrap process. (It looks like it hasn't been updated for the switch from 2.1 => master yet.)

Darc is a pretty intense mess of dependencies and pulling it apart is going to be painful and awkward and difficult to maintain. I'm hoping we can find a more sustainable solution that doesn't drag the web service into some dependency cycle.

From what I've seen so far, I think the concerns are a bit overstated, but I think it's reasonable to wait for me to give it a try or something. Keep in mind that we aren't asking for this for preview2 or RTM.

[mmitche]

@dagood I don't think this is required for preview 2, right?

[dagood]

Correct, not needed for preview2.

[riarenas]

@dagood is this required for .net 5 rtm?

[dagood]

is this required for .net 5 rtm?

Nope.

(I think we (source-build) should actually consider extracting the subset of darc clone functionality we actually need and put it in a dotnet/arcade build task, which would end up resolving this. More info at https://github.com/dotnet/source-build/issues/1571.)

[riarenas]

Thanks @dagood. The idea seems interesting. Let me know what you decide to see if we can close this or if we should re-examine for future releases.

[dagood]

Is "darc" necessary to do anything more than fetching the correct commits/source blobs?

Yes, when I mentioned "I'm pretty sure we'll be using darc to flow dependencies between repos." I'm referring to this:

• Darc is meant to be the only way developers and other tools like Maestro++ interact and alter version/dependency files as well as bootstrapping files and scripts in arcade participating repos.
• Darc's operations range from altering version/dependency files to creating PRs in specified repos.

Currently we use the Repo API (specifically Package Version Props) to do this, and from the above it sounds like using Package Version Props won't be supported. (And we might still need Darc for a standard way to flow zips through?)

I noticed this when reading this thread back and I think I should give an update. It turns out that this didn't happen: we kept on using Package Version Props, and that infra has been maintained. We only use darc clone when creating a source tarball, not when building it, so building darc from source is not strictly necessary to build .NET from source without prebuilts. But I think this is an ugly technicality. Having to use a proprietary tool to create a usable source tarball from a Git tag isn't in the spirit of building from source, IMO.

Let me know what you decide to see if we can close this or if we should re-examine for future releases.

I think we can close it when we no longer depend on the prebuilt darc, whether that's via a build task or the original plan. I think it's worth keeping it open until then to keep track of this.

[premun]

We have made a decision last year that building darc from source is not a requirement