[SR-13396] Support for vendoring dependencies

[05262b81-54a9-4fe1-bf6a-96f8042de10e]

Previous ID	SR-13396
Radar	rdar://problem/67360651
Original Reporter	@lilyball
Type	New Feature

Additional Detail from JIRA

| | | |------------------|-----------------| |Votes | 5 | |Component/s | Package Manager | |Labels | New Feature | |Assignee | None | |Priority | Medium | md5: 4f474f63e95170f27f10251b004d6fd2

Issue Description:

As near as I can tell, Swift Package Manager has no support for vendoring dependencies. The closest it has is the ability to declare that individual packages are mirrored at different URLs. But what I really want is to be able to place the working copies for my dependencies at a user-controlled path, along with graceful handling of having existing working copies but no associated git dirs exist when being asked to resolve dependencies. The idea being I can then check the working copies into source control but ignore the git checkouts, and SPM would happily use the existing working copies without invoking network access as long as it's up-to-date with the lockfile.

The reason for wanting this is twofold:

1. If I git remote update my project prior to going offline, I should be able to check out a previously-unseen commit that changes my dependencies without running into trouble because of my lack of internet connection.

2. My company's security policy requires packages sourced from package managers to either be vendored or to come from an internal mirror, in order to insulate ourselves from malicious third parties making changes to existing package releases (including the git commit in the lockfile helps avoid silently adopting malicious changes but would cause a build failure which isn't good either). SPM does support mirroring, but it has to be configured independently for each URL, which means if we want to mirror we either have to script our usage of SPM to inspect all the transitive dependencies and complain if any are not configured with a mirror, or we have to rewrite all packages when we mirror them to our internal host to change the URLs in their Package.swift file. And mirroring doesn't solve the offline usage issue.

We already vendor CocoaPods, but the lack of SPM vendoring support is a major roadblock to migrating from CocoaPods to SPM.

Xcode of course also needs to support the ability to vendor packages.

---

The explicit behavior I want from SPM is:

• Some way to instruct SPM to vendor packages. This would make sense as a swift package config directive. For Xcode, presumably a configuration toggle in the "Swift Packages" build tab for the project, or possibly in the Shared Workspace Settings because this setting should probably apply to all projects in the workspace.

• When asked to vendor to a specific path, on first dependency resolution, SPM would create a directory at that path (if it doesn't exist), put all of the dependency working trees in that directory but without a .git file (it can check out the git repos elsewhere and set core.worktree to point at the vendored path), and place some sort of manifest file in the directory to declare what the work trees represent. This manifest would ideally be text in a merge-friendly format (possibly just a clone of Package.resolved).

• On subsequent dependency resolutions, it would read the vendored manifest, and if that matches Package.resolved and if every referenced dependency exists in the vendored path, then it assumes the vendored dependency is valid and does not do network access, or even touch the associated git repo (since the git repo may not even exist).

• If the manifest does not match the Package.resolved it would run the normal dependency resolution process that would exist if vendoring wasn't being used.

I'm guessing this behavior is similar to what it does already with non-vendored dependencies, except the .build dir today has a sqlite3 manifest.db file, which is not suitable for checking into VCS (and I don't know what it actually does). It also has a dependencies-state.json file which does actually look like the sort of text-only manifest I'm talking about, except it's JSON with no newlines so again it's not suited to merging (I'm not actually sure how mergeable Package.resolved itself is, since that's also JSON, but hopefully SPM has explicit ordering of the JSON in Package.resolved to ensure stable output that's more likely to be mergeable). Also the checkouts today are full git copies, clones of another local repo; I don't actually know why it uses a separate git clone for the checkout, though at least it's using alternates to share objects.

Anyway the point is what it does today isn't suitable for checking into VCS, and I have no idea what SPM would do if I threw away the repositories and manifest.db but kept the checkouts and the dependencies-state.json file. And I can't do this with Xcode anyway as Xcode stores all this stuff in derived data.

[typesanitizer]

@swift-ci create

[swift-ci]

Comment by Chris Ballinger (JIRA)

When compiling via `xcodebuild` on the command line, this is possible by passing the `-clonedSourcePackagesDirPath ExamplePath/` flag. As far as I'm aware there is no way to configure this for compiling within Xcode itself though, just by overriding your entire DerivedData destination.