Support for Xcode project Swift PM Dependencies

mc7h commented 3 years ago

What would you like Renovate to be able to do?

At present renovate doesn't help manage Swift dependencies from the perspective of an Xcode project. Large Xcode projects can have many dependencies and transitive dependencies. Annoyingly, Xcode projects don't handle these dependencies via a Package.swift file, like Swift packages themselves do.

Did you already have any implementation ideas?

Dependencies are tracked in project.pbxproj.

/* Begin XCRemoteSwiftPackageReference section */
               A74E51122637DA2D0018AC52 /* XCRemoteSwiftPackageReference "sqlite" */ = {
                       isa = XCRemoteSwiftPackageReference;
                       repositoryURL = "https://github.com/shareup/sqlite.git";
                       requirement = {
                               kind = upToNextMajorVersion;
                               minimumVersion = 15.0.1;
                       };
               };
/* End XCRemoteSwiftPackageReference section */

Resolved packages can be found in app.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved

github-actions[bot] commented 3 years ago

Hi there,

The Renovate team needs your help! Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on minimal reproductions to understand what is needed.

We may close the issue if you have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

rarkins commented 3 years ago

This sounds like something Renovate could do, but we'll need to understand the file formats and process better e.g.

does the resolved file need to be refreshed every time the project.pbxproj is updated?
what is the relationship between Package.swift and project.pbxproj, if any?

mc7h commented 3 years ago

I think the real challenge will lie in the fact that there doesn't appear to be a Public specification for .pbxproj so any solution would need to be based on observation alone and could be fragile. That said, there are Projects that work with the .pxproj files successfully (e.g https://github.com/CocoaPods/Xcodeproj)

No, the Package.resolved file doesn't need to be refreshed every time the project.pbxproj is updated; but it will be changed when a new Dependency is added (and resolved) or when Dependencies are updated through the Xcode UI.
project.pbxproj is a fairly-huge file. It describes almost every aspect of an Xcode project. For dependencies, rather than having the usual Package.swift -> Package.resolved relationship Apple has chosen to embed all thePackage.swift content to the project.pbxproj in the section: XCRemoteSwiftPackageReference. From what I can gather, the Package.resolved format does not differ from what we already have with Swift packages.

Sherlouk commented 2 years ago

I've started to pick this up. There's a couple nuances that we need to tackle, one of which is that the existing SPM solution doesn't handle the resolved file where it's a lot more necessary for Xcode (as it uses it as a source of truth and doesn't update automatically as frequently as a Swift Package framework).

Depending on the rangeStrategy, we can get away with either updating just the project file, or we need to update both the project file and the resolved file.

It's going to take a bit of messing to get it perfect and to support all permutations of package type.

Sherlouk commented 2 years ago

Filling out the template for a new manager. Reproduction: https://github.com/JamesSherlouk/renovate-xcode/

Did you read our documentation on adding a package manager?

[x] I've read the adding a package manager documentation.

Basics

Name of package manager

swift-xcode (Swift Package Manager for Xcode)

I have also considered naming this simply xcode (as there is no other package manager native to Xcode) or swift merging functionality with the existing manager.

What language does this support?

Swift

How popular is this package manager?

According to the iOS Developer Survey ran in 2020 (source), Swift Package Manager is the leading package manager for newer and personal projects with CocoaPods taking a small lead in business applications. I believe it is fair to say that Swift Package Manager is constantly growing within the community, and due to it's first-party integrations and support will (if it hasn't already) overtake CocoaPods shortly.

Does this language have other (competing?) package managers?

[x] Yes
[ ] No

Developers in the Apple ecosystem have several choices including CocoaPods (which Renovate supports), Carthage, and Swift Package Manager (SPM). SPM is the only first-party package manager supported by Apple for use with iOS and other Apple platforms.

Package File Detection

What type of package files and names does it use?

Unlike the existing SPM support started in #3911, when included in Xcode projects there is no Package.swift file. The dependency definitions are included in the Xcode project file which, thanks Apple, uses an undocumented structure.

What fileMatch pattern(s) should be used?

The matched file should point to the Xcode project definition which which is called project.pbxproj and sits within the project's .xcodeproj directory. The default configuration includes a regex for this.

Is it likely that many users would need to extend this pattern for custom file names?

[ ] Yes
[x] No

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

It is almost definitely true that this will pickup Xcode project files that do not specify Swift packages in them, though I wouldn't call this a "false hit" as they are genuine package definitions, it just happens that they don't have any.

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

The project file can include local references, but these are ignored by the manager as they're not linked as Swift packages which is what it looks for.

Is there a reason why package files need to be parsed together (in serial) instead of independently?

Nope, each project file can be parsed independently.

What format/syntax is the package file in?

[ ] JSON
[ ] TOML
[ ] YAML
[x] Custom (explain below)

Apple uses a proprietary format which is undocumented. It may change at any point. There are a number of other libraries which do attempt to parse it, but I didn't believe them to be a good fit for this project as it would dramatically increase the complexity for the small part we need.

How do you suggest parsing the file?

[ ] Off the shelf parser
[x] Using regex
[ ] Custom-parsed line by line
[ ] Other

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, dev dependencies, etc?

[ ] Yes, production and development dependencies
[x] No, all dependencies are treated the same
[x] Technically we could...?

It is possible with a fair bit more work to identify which Xcode "target" each dependency is tied to. A dependency can be in one or more targets. Each target has a "product type" which does tell us whether it's designed for testing, or is for the application. By our definition, a testing dependency is a developer dependency.

I do not believe it's worth the effort for this initial investigation but could be something to explore in the future to provide more control to users.

List all the sources/syntaxes of dependencies that can be extracted

Dependencies can be added with a valid Git URL, this can be in the SSH format or HTTPS format, and can be from any provider (GitHub, GitLab, Bitbucket, etc).

Dependencies may have one of a few different 'requirements' for versioning:

upToNextMajorVersion
upToNextMinorVersion
exactVersion
branch
revision (Git SHA)
versionRange

Describe which types of dependencies above are supported and which will be implemented in future

With current assumptions:

For exactVersion and revision, the resolved code cannot be changed and as such will always be ignored by Renovate. For branch, the resolved revision (SHA) may be changed but is out of scope for this initial PR. For the other three, we can use SemVer in order to identify newer versions which fall within the provided ranges.

Versioning

What versioning scheme does the package file(s) use?

Semantic Versioning

Does this versioning scheme support range constraints, e.g. `^1.0.0` or `1.x`?

[ ] Supports range constraints (e.g ^1.0.0 or 1.x)
[x] No

Basic constraints are supported (as documented above), but is not as elaborate as some other package managers support.

Is this package manager used for applications, libraries, or both? If both, is there a way to tell which is which?

[ ] Applications
[ ] Libraries
[x] Both (explain how to tell which is which)

While the package manager can be used for both, practically speaking, there is no need to tell the difference as they are handled identically.

If ranges are supported, are there any cases when Renovate should pin ranges to exact versions if rangeStrategy=auto?

There is no necessity to pin ranges, but (as per the question below) depending on the range strategy we may approach things slightly differently. This should be a user preference.

Lookup

Is a new datasource required? Provide details

[ ] Yes, provide details.
[x] No.

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Nope, dependencies are linked to a source control platform such as GitHub or GitLab. All the information is in the source URL.

Do the package files have any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc) that should be used in the lookup procedure?

No, the package file format is linked to the Xcode version used to generate it (and subsequently committed into source control). While this could practically change in the future (forcing us to rework how our RegExs work) but given the structure has mostly remained the same for a decade, this isn't a major concern.

Will users need the ability to configure language or other constraints using Renovate config?

No, if we do need to switch extraction in the future, we would do this based on the "objectVersion" which is stored in the file. No user configuration is required.

Artifacts

Are lock files or checksum files used? Are they mandatory?

Xcode generates a "Package.resolved" file which is in a JSON format, this informs Xcode on which version to download given the requirements listed in the project definition file.

Depending on our approach, updating the pins in this file may be required in order to ensure Xcode actually uses the newer version.

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

Xcode does not, currently, provide a tool to update a single dependency. There is a command xcodebuild -resolvePackageDependencies which (under some circumstances) updates all dependencies, but this is not reliable.

It is more likely that we will manually alter the JSON file ourselves to pin the correct version.

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or env? Do you recommend the cache be kept or disabled/ignored?

N/A

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

N/A

viceice commented 2 years ago

are there any transitive dependencies which need to be resolved for the JSON file?

we really should update both files to have it fully functional.

also we can't use any tool to update the lockfile which isn't at least working on Linux, otherwise it would only supported by self-hosted renovate on apple MacOS.

Sherlouk commented 2 years ago

are there any transitive dependencies which need to be resolved for the JSON file?

If we're going down the route of updating both files, then we will need to handle transitive dependencies too. This is where we open pandora's box of having version conflicts (when two dependencies rely on different versions of the same dependency).

I'm not sure what the best way to handle this is. We have no dependency tree file to use for grouping.

Is it acceptable to just update them each individually, and rely on the user's CI/config to manage incompatibilities? Do we have other managers we can use for examples here?

Example Package.resolved

{
  "pins" : [
    {
      "identity" : "alamofire",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/Alamofire/Alamofire.git",
      "state" : {
        "revision" : "63dfa86548c4e5d5c6fd6ed42f638e388cbce529",
        "version" : "5.6.0"
      }
    },
    {
      "identity" : "cocoalumberjack",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/CocoaLumberjack/CocoaLumberjack.git",
      "state" : {
        "revision" : "e518eb6e362df327574ba5e04269cd6d29f40aec",
        "version" : "3.7.2"
      }
    }
  ]
}

we really should update both files to have it fully functional.

Agreed. It increases complexity, but I think it's better to have a more well-rounded solution which is also more resilient to Xcode shenanigans.

we can't use any tool to update the lockfile which isn't at least working on Linux

Yup, this was fully expected and is one of the many reasons we shouldn't rely on xcodebuild to manage the lock file.

federicopolesello commented 2 years ago

Yeah good work! Keep in mind one important thing in my opinion: if you have an xcworkspace the Package.resolved file is created and updated under the .xcworkspace folder (.xcworkspace/xcshareddata/swiftpm/Package.resolved) and the old one under the original .xcodeproj folder will be ignored. In my case I'm still having the xcworkspace for some reasons.

Sherlouk commented 2 years ago

Aye so there's two files we need to look at: • Package.resolved • pbxproj

Both of these really need to be updated to work reliably.

I've just gotten back from Norway so hopefully going to tackle this again - planning to go back to basics a little as trying to reuse the existing solution (which I'm finding has a few flaws) is causing problems.

rarkins commented 1 year ago

@Sherlouk do you still plan to move this forward?

Sherlouk commented 1 year ago

Has fallen down my priority list a bit, as I have a homebrew workaround which is working for me - but I do definitely want to support or lead this piece of work.

From when I looked at it, it does appear that there will need to be a bit of a bigger change to the existing Swift solution in order to make it better at handling these different formats.

sebastianbaar commented 1 year ago

@Sherlouk any updates regarding this issue? Or could you at least share your homebrew workaround? Would be nice to see renovate being able to be used for iOS projects using SPM.

Sherlouk commented 1 year ago

This issue continues to be the only issue I don't mark as done in GitHub! The only thing stopping me from reaching inbox zero

I still fully intend on getting back to this - just time is a precious commodity and one I don't have excess amounts in right now. Apologies! Hopefully soon 🙏🏼

sebastianbaar commented 1 year ago

@Sherlouk I totally understand this! I just wanted to get some feedback regarding this issue. Thank you very much for your answer.

garethlloyd1994 commented 1 year ago

Has this been any update on this/any workarounds that we can use?

Sherlouk commented 11 months ago

Absolutely agree, having Renovate have full support for Swift including Xcode projects would be an amazing feature.

Dependabot now has Swift support but just like Renovate limits this to Package.swift, this is presumably just down to the complexities involved with adding Xcode project support (partially documented in this thread, but also just from experience in trying to add support earlier in the year).

Apple don't make this easy, especially to do it right where you need to update multiple files some with proprietary formatting (which we only know through extensive trial and error).

I'd love to get back to this, but please do not feel like this ticket has to wait for me either if somebody has the time and intrigue to get it done.

While far from comparable, I do know a number of teams who simply run xcodebuild -resolvePackageDependencies -disablePackageRepositoryCache via their CI on a periodic basis which includes all updates at once.

rarkins commented 11 months ago

Comments which are like "I'd really like to see this feature", "Any update?" or "Yes you should really do this feature, would be a shame if someone else did it first and nobody uses Renovate anymore" will be hidden as Spam because they do not help the project in any way while wasting maintainer time with unnecessary notifications.

If you really want to see the feature, you can help identify any remaining requirements gaps or write the code. Thank you to @Sherlouk for doing so, so far! If you're eager for the feature but no intention of contributing, please stick to GitHub's +1 and subscribe features and stop creating noise.

markst commented 3 weeks ago

I'm not sure if there's much impact to the pbx using XcodeProj to parse then save out with modifications.

It would be possible to traverse the dependencies using: https://github.com/tuist/XcodeProj/blob/main/Sources/XcodeProj/Objects/SwiftPackage/XCRemoteSwiftPackageReference.swift#L13

I wonder about hosting the XcodeProj binary on NPM so it can be used as a dependency. https://www.npmjs.com/search?q=XcodeProj

oonoo commented 3 weeks ago

Not directly an answer to a question here, but I was triggered by @markst mentioning XcodeProj, which we use in the company I work for. The weird workaround I built to get Renovate support for the Xcode projects of the company I work for:

A Swift package that has all dependencies that are only in Xcode projects. This let's Renovate create an merge request even for these.
Configure a CI job that starts whenever the Renovate bot created a commit for updating a Swift dependency. This CI job will do all of the following steps.
Get the information what dependencies are changed (and the "target" version) via the API of the CI platform (request to get the merge request description and parse that)
A little self-created tool based on tuist/XcodeProj takes the information from the previous step and updates the dependencies in all Xcode projects in the repository
Running xcodebuild to update the Package.resolved file to have the new version of the dependency

Our renovate bot runs on Linux, so the last command to xcodebuild is probably impossible there because it needs macOS. But it would be cool if all the other steps could be done by Renovate itself. 🙏

viceice commented 3 weeks ago

I'm not sure if there's much impact to the pbx using XcodeProj to parse then save out with modifications.

It would be possible to traverse the dependencies using: https://github.com/tuist/XcodeProj/blob/main/Sources/XcodeProj/Objects/SwiftPackage/XCRemoteSwiftPackageReference.swift#L13

I wonder about hosting the XcodeProj binary on NPM so it can be used as a dependency. https://www.npmjs.com/search?q=XcodeProj

if you can compile it to wasm^1, then we can consider using it.

renovatebot / renovate

Support for Xcode project Swift PM Dependencies #9735