search

swiftlang / swift-package-manager

The Package Manager for the Swift Programming Language
Apache License 2.0
9.75k stars 1.35k forks source link

[SR-8262] Swift Package Manager fails at checkout when dependent repository contains Git LFS objects #5351

Open swift-ci opened 6 years ago

swift-ci commented 6 years ago
Previous ID SR-8262
Radar rdar://problem/42246102
Original Reporter yifei (JIRA User)
Type Bug
Additional Detail from JIRA | | | |------------------|-----------------| |Votes | 2 | |Component/s | Package Manager | |Labels | Bug | |Assignee | None | |Priority | Medium | md5: 2218c2c1d09b919e983b204e24d25058

Issue Description:

The error is in the format of: (XXXXX being the name of LFS-tracked file)

Error downloading object: XXXXX (hash): Smudge error: Error downloading XXXXX (hash): batch request: missing protocol: "/path/to/project/.build/repositories/dependency.git/info/lfs"

error: external filter 'git-lfs filter-process' failed
fatal: XXXXX: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

This happens during the "cloneCheckout" phase i.e. invoking "git clone --shared". If I disable the LFS filter by editing the global .gitconfig, the project builds fine.

aciidgh commented 6 years ago

@swift-ci create

jtbandes commented 2 years ago

Has there been any update on this issue? There's a similar discussion at https://forums.swift.org/t/swiftpm-with-git-lfs/42396.

I am running into this issue as well with https://github.com/foxglove/mcap. This dependency fails during checkout: .package(url: "https://github.com/foxglove/mcap", branch: "main"),

Creating working copy for https://github.com/foxglove/mcap
error: Couldn’t check out revision ‘2dad37cad3c7875842b44f3fe927e2dcfdecf7d4’:
    Downloading docs/img/0398582-hero.png (669 KB)
    Error downloading object: docs/img/0398582-hero.png (266f9d3): Smudge error: Error downloading docs/img/0398582-hero.png (266f9d3dd91c1bd739e1d79ae4abd459f71e8d4c661656745504ea52913467b5): error transferring "266f9d3dd91c1bd739e1d79ae4abd459f71e8d4c661656745504ea52913467b5": [0] remote missing object 266f9d3dd91c1bd739e1d79ae4abd459f71e8d4c661656745504ea52913467b5
    Errors logged to /Users/jacob/Desktop/testpackage/.build/checkouts/mcap/.git/lfs/logs/20220923T112418.927225.log
    Use `git lfs logs last` to view the log.
    error: external filter 'git-lfs filter-process' failed
    fatal: docs/img/0398582-hero.png: smudge filter lfs failed

A simple git clone succeeds.

shoaibsadanaut commented 2 years ago

Is there any progress on it or someone has a solution?

neonichu commented 2 years ago

We're talking about an Xcode specific issue here, correct? I tried reproducing with https://github.com/foxglove/mcap and swift build and it seems to work fine for me.

jtbandes commented 2 years ago

Nope, it's not an Xcode specific issue. swift build within the repository works fine when you manually clone it. The problem is with swiftpm's automatic cloning.

Try creating an empty directory and add this Package.swift:

// swift-tools-version:5.5
import PackageDescription

let package = Package(
  name: "demo",
  products: [],
  dependencies: [
    .package(url: "https://github.com/foxglove/mcap", branch: "main"),
  ],
  targets: []
)

Then run swift build:

$ swift build
Fetching https://github.com/foxglove/mcap from cache
Fetched https://github.com/foxglove/mcap (2.03s)
Fetching https://github.com/apple/swift-collections from cache
Fetched https://github.com/apple/swift-collections (0.76s)
Fetching https://github.com/apple/swift-algorithms from cache
Fetched https://github.com/apple/swift-algorithms (0.33s)
Computing version for https://github.com/apple/swift-algorithms
Computed https://github.com/apple/swift-algorithms at 1.0.0 (0.32s)
Fetching https://github.com/apple/swift-numerics from cache
Fetched https://github.com/apple/swift-numerics (0.60s)
Computing version for https://github.com/apple/swift-numerics
Computed https://github.com/apple/swift-numerics at 1.0.2 (0.33s)
Creating working copy for https://github.com/apple/swift-algorithms
Working copy of https://github.com/apple/swift-algorithms resolved at 1.0.0
Creating working copy for https://github.com/apple/swift-collections
Working copy of https://github.com/apple/swift-collections resolved at 418378107c87a4b312e29a51f773ce0e4e12e199
Creating working copy for https://github.com/apple/swift-numerics
Working copy of https://github.com/apple/swift-numerics resolved at 1.0.2
Creating working copy for https://github.com/foxglove/mcap
error: Couldn’t check out revision ‘14fa18afb83e504ae03397fd4e9292a6c83a7e80’:
    Downloading docs/img/0398582-hero.png (669 KB)
    Error downloading object: docs/img/0398582-hero.png (266f9d3): Smudge error: Error downloading docs/img/0398582-hero.png (266f9d3dd91c1bd739e1d79ae4abd459f71e8d4c661656745504ea52913467b5): error transferring "266f9d3dd91c1bd739e1d79ae4abd459f71e8d4c661656745504ea52913467b5": [0] remote missing object 266f9d3dd91c1bd739e1d79ae4abd459f71e8d4c661656745504ea52913467b5
    Errors logged to /Users/jacob/Desktop/swifttest/.build/checkouts/mcap/.git/lfs/logs/20221006T101212.79192.log
    Use `git lfs logs last` to view the log.
    error: external filter 'git-lfs filter-process' failed
    fatal: docs/img/0398582-hero.png: smudge filter lfs failed
neonichu commented 2 years ago

Interesting, this does not happen for me. I ended up with the same commit as you, but nothing in .git/lfs:

❯ git show --oneline -s
14fa18afb (HEAD, origin/main, origin/HEAD, main) fix link in README (#636)

/tmp/foo/.build/checkouts/mcap heads/main
❯ ls -ld .git/lfs
".git/lfs": No such file or directory (os error 2)

I'm not really familiar with git-lfs, is there any global config required for it?

jtbandes commented 2 years ago

I'm not really familiar with git-lfs, is there any global config required for it?

Yes, normally users will run git lfs install once, as described here: https://git-lfs.github.com

It's a very common thing to have installed, since many projects use Git LFS.

What do you see when you look at /tmp/foo/.build/checkouts/mcap/docs/img/0398582-hero.png? I'm guessing you have only the LFS pointer file, which is what you'd get if you don't have LFS installed.

jtbandes commented 2 years ago

Also note that some folks on Discourse have dug deeper into why this happens: https://forums.swift.org/t/swiftpm-with-git-lfs/42396/13

neonichu commented 2 years ago

Thanks, this helps!

jtbandes commented 2 years ago

Thanks for looking into this! 🙌 Just wondering if there’s been any update so far on a possible resolution?

jtbandes commented 2 years ago

I tried adding a .lfsconfig file to the repo, as recommended in the Discourse thread, and this seems to have worked around the issue for swift build (like the small example I gave you), but it still doesn't work when adding the dependency in Xcode...

Example project that fails to build in Xcode on my machine: TestPackage.zip

image
briancordanyoung commented 2 years ago

Based on this thread and the forum discussion, I came to the same blocking errors.

Linking git-lfs did work in the sense that Xcode used it but reported a different error downloading various files that were checked in with lfs. The error included "Smudge error: Error downloading..."

> cd /Applications/Xcode-14.1.0.app/Contents/Developer
> ln -s /opt/homebrew/bin/git-lfs git-lfs

Adding a .lfsconfig (with an updated path for my repo) did not help with me when running swift build or an Xcode package resolution. I did have some GitHub authentication errors until I removed the commit that added the file.

The only work around I have is to manually clone the repo and include it as a local package—which of course negates all advantages of using a dependency manager. 😫

I did file FB11803523 for Xcode.

jtbandes commented 2 years ago

Not sure if it helps you, but I believe the .lfsconfig file has to be present on the repo's default branch. When SPM clones the package it starts with the default branch and then checks out whichever branch/commit you requested.

This did ultimately end up working well enough for me, I was able to use xcodebuild in CI (but not the Xcode app): https://github.com/foxglove/MCAPSpotlightImporter/actions/runs/3380647041/jobs/5613664270

However, the .lfsconfig file is causing some other issues for developers working with the repo... 😓

iosdevben commented 1 year ago

Is there any update to this?

twprzybysz commented 1 year ago

+1

nicolascabaret commented 1 year ago

+1

scosman commented 1 year ago

One option for this I wrote up here (which only applies if the LFS files are something optional for your package, like test results): https://stackoverflow.com/questions/67457594/swift-package-manager-does-not-work-with-git-lfs/77336212#77336212

koliyo commented 1 year ago

The workaround from @briancordanyoung seem to work, but the Xcode path has changed

cd /Applications/Xcode.app/Contents/Developer/usr/bin
ln -s /opt/homebrew/bin/git-lfs git-lfs
yangkx1024 commented 1 year ago
image

Seems the workaround conflict with SIP, it's weird to include XCode folder in SIP.

koliyo commented 1 year ago

Are you sure it is SIP @yangkx1024? Wonder why the symlinking was allowed for me, I have not disabled SIP

csrutil status

System Integrity Protection status: enabled.
yangkx1024 commented 1 year ago

I disabled the SIP and the ln command success, but now I stuck with the git-lfs issue, same as this post: https://forums.swift.org/t/swiftpm-with-git-lfs/42396/13

jtbandes commented 10 months ago

@neonichu if you learned anything about the root cause of this issue or possible workaround, I’m sure many of us would be interested to hear it, thanks :)

dipidoo commented 6 months ago

I am using the following workaround:

iosdevben commented 2 months ago

Can we get an update on this please? It's been open for 6 years but seems like a critical barrier preventing 100% adoption of Swift PM.

briancordanyoung commented 2 months ago

@iosdevben I would love to have this fixed. But I would not let it prevent you from adopting SPM. I've been using this work around all this time and the benefits of SPM have made it completely worth it.