search

swiftlang / swift-package-manager

The Package Manager for the Swift Programming Language
Apache License 2.0
9.76k stars 1.35k forks source link

[SR-662] swiftpm doesn't correctly order .a files when constructing swiftc args #5431

Closed swift-ci closed 8 years ago

swift-ci commented 8 years ago
Previous ID SR-662
Radar None
Original Reporter jaybuff (JIRA User)
Type Bug
Status Closed
Resolution Done

Attachment: Download

Additional Detail from JIRA | | | |------------------|-----------------| |Votes | 2 | |Component/s | Package Manager | |Labels | Bug | |Assignee | @mxcl | |Priority | Medium | md5: f13d5ad12d638b307720fac644c5388f

relates to:

Issue Description:

It looks like llbuild doesn't consider order to pass .a files to swiftc.

$ swift --version
Swift version 2.2-dev (LLVM 3ebdbb2c7e, Clang f66c5bb67b, Swift 42591f7cba)
Target: x86_64-unknown-linux-gnu

# build a package that depends on the swift package manager
$ cat Package.swift
import PackageDescription

let package = Package(
    name: "swift-ld-bug",
    dependencies: [
        .Package(url: "https://github.com/apple/swift-package-manager.git", Version!("0.1.0")),
    ]
)

# try to use the sys.Path method provided by swift pm
$ cat Sources/main.swift
import sys
let path = Path.join("usr", "bin")
print("path is \(path)")

# the build fails with linker issues: sys can't find methods in libc and POSIX
$ swift build
Compiling Swift Module 'libc' (3 sources)
Compiling Swift Module 'PackageDescription' (3 sources)
Linking Library:  .build/debug/libc.a
Compiling Swift Module 'POSIX' (22 sources)
Linking Library:  .build/debug/POSIX.a
Compiling Swift Module 'sys' (9 sources)
Linking Library:  .build/debug/sys.a
Linking Library:  .build/debug/PackageDescription.a
Compiling Swift Module 'dep' (9 sources)
Linking Library:  .build/debug/dep.a
Compiling Swift Module 'swiftbuild' (3 sources)
Linking Executable:  .build/debug/swift-build
Compiling Swift Module 'swiftldbug' (1 sources)
Linking Executable:  .build/debug/swift-ld-bug
/home/jaybuff/swift-ld-bug/.build/debug/sys.a(Path.swift.o): In function `_TFE3sysSSg10isAbsoluteSb':
/home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0/Sources/sys/Path.swift:194: undefined reference to `_TFE4libcSS9hasPrefixfSSSb'
/home/jaybuff/swift-ld-bug/.build/debug/sys.a(Path.swift.o): In function `_TZFV3sys4Pathg4homeSS':
/home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0/Sources/sys/Path.swift:48: undefined reference to `_TF5POSIX6getenvFSSGSqSS_'
/home/jaybuff/swift-ld-bug/.build/debug/sys.a(Path.swift.o): In function `_TFE3sysSS7abspathfzT_SS':
/home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0/Sources/sys/Path.swift:189: undefined reference to `_TF5POSIX6getcwdFzT_SS'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: build had 1 command failures
error: exit(1): ["/home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swift-build-tool", "-f", "/home/jaybuff/swift-ld-bug/.build/debug/swift-ld-bug.o/llbuild.yaml"]

# run that build with verbose output and see the swiftc command it's using:
$ swift build -v
/home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swiftc --driver-mode=swift -I /home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/lib/swift/pm -L /home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/lib/swift/pm -lPackageDescription /home/jaybuff/swift-ld-bug/Package.swift
/usr/bin/git -C /home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0 config --get remote.origin.url
/usr/bin/git -C /home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0 config --get remote.origin.url
/home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swiftc --driver-mode=swift -I /home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/lib/swift/pm -L /home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/lib/swift/pm -lPackageDescription /home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0/Package.swift
/usr/bin/git -C /home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0 config --get remote.origin.url
/home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swiftc --driver-mode=swift -I /home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/lib/swift/pm -L /home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/lib/swift/pm -lPackageDescription /home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0/Package.swift
/home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swift-build-tool -v -f /home/jaybuff/swift-ld-bug/.build/debug/SwiftPackageManager.o/llbuild.yaml
/home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swift-build-tool -v -f /home/jaybuff/swift-ld-bug/.build/debug/swift-ld-bug.o/llbuild.yaml
/home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swiftc -o /home/jaybuff/swift-ld-bug/.build/debug/swift-ld-bug /home/jaybuff/swift-ld-bug/.build/debug/swift-ld-bug.o/Sources/main.swift.o -g /home/jaybuff/swift-ld-bug/.build/debug/dep.a /home/jaybuff/swift-ld-bug/.build/debug/libc.a /home/jaybuff/swift-ld-bug/.build/debug/PackageDescription.a /h
ome/jaybuff/swift-ld-bug/.build/debug/POSIX.a /home/jaybuff/swift-ld-bug/.build/debug/sys.a -L/usr/local/lib
/home/jaybuff/swift-ld-bug/.build/debug/sys.a(Path.swift.o): In function `_TFE3sysSSg10isAbsoluteSb':
/home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0/Sources/sys/Path.swift:194: undefined reference to `_TFE4libcSS9hasPrefixfSSSb'
/home/jaybuff/swift-ld-bug/.build/debug/sys.a(Path.swift.o): In function `_TZFV3sys4Pathg4homeSS':
/home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0/Sources/sys/Path.swift:48: undefined reference to `_TF5POSIX6getenvFSSGSqSS_'
/home/jaybuff/swift-ld-bug/.build/debug/sys.a(Path.swift.o): In function `_TFE3sysSS7abspathfzT_SS':
/home/jaybuff/swift-ld-bug/Packages/swift-package-manager-0.1.0/Sources/sys/Path.swift:189: undefined reference to `_TF5POSIX6getcwdFzT_SS'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: build had 1 command failures
/usr/bin/which clang++
error: exit(1): ["/home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swift-build-tool", "-v", "-f", "/home/jaybuff/swift-ld-bug/.build/debug/swift-ld-bug.o/llbuild.yaml"]

# what if we we run the swiftc, but put sys.a before POSIX.a and libc.a in the args:
$ /home/jaybuff/swift-2.2-SNAPSHOT-2016-01-11-a-ubuntu15.10/usr/bin/swiftc -o /home/jaybuff/swift-ld-bug/.build/debug/swift-ld-bug /home/jaybuff/swift-ld-bug/.build/debug/swift-ld-bug.o/Sources/main.swift.o -g /home/jaybuff/swift-ld-bug/.build/debug/sys.a /home/jaybuff/swift-ld-bug/.build/debug/dep.a /home/jaybuff/swift-ld-bug/.build/debug/libc.a /home/jaybuff/s
wift-ld-bug/.build/debug/PackageDescription.a /home/jaybuff/swift-ld-bug/.build/debug/POSIX.a  -L/usr/local/lib

# it worked!
$  .build/debug/swift-ld-bug
path is usr/bin

The list of .a files and their order is passed from swiftc to clang++, which is finally passed to ld. ld's man page says that order matters:

Normally, an archive is searched only once in the order that it is specified on the command line. If a symbol in that archive is needed to resolve an undefined symbol referred to by an object in an archive that appears later on the command line, the linker would not be able to resolve that reference.

mxcl commented 8 years ago

This is a package manager bug as we decide the order.

swift-ci commented 8 years ago

Comment by Jay Buffington (JIRA)

Ah, yes, looks like the bug is somewhere around here (from Sources/dep/llbuild.swift)

                // Target dependencies are *already* reverse sorted.

                let libsInThisPackage = target.dependencies
                    .filter{ $0.type == .Library }
                    .map{ $0.productFilename }
                    .map{ Path.join(params.prefix, $0) }
swift-ci commented 8 years ago

Comment by Shmuel Kallner (JIRA)

I'd like to add that this bug will only show itself on certain Unix based platforms. In particular on OSX you won't see this issue because LD will scan the list of .a flles multiple times.

One solution for this on Linux albeit not optimal is to place all of the .a's in a group using the -( and -) LD parameters around the .a files. This may not be optimal as the LD documentation claims there is a performance penalty for doing this.

mxcl commented 8 years ago

We should be able to order them correctly

mxcl commented 8 years ago

Confirmed this still occurs after the testing refactor. Looking into it.

swift-ci commented 8 years ago

Comment by Shmuel Kallner (JIRA)

We have found that the issue seems to crop up if one adds "extra" dependencies beyond the minimal list needed.

By that I mean, if A is dependent on B and B is dependent on C, then in theory A can directly import C, even though A is not directly dependent on C and that dependency may not have been listed in A's dependency list at all. If however C gets listed as a dependency of A and by chance it is listed in the dependency list after B, then the problem occurs. The solution is to make sure that such lists of dependencies are in reverse order, i.e. in the case above C is listed before B

mxcl commented 8 years ago

Actually, good news, I didn't correctly verify this bug. It seems to work with latest swiftpm:

Let me know if I'm wrong.

swift-ci commented 8 years ago

Comment by Jay Buffington (JIRA)

Works for me 👍