Add tensorflow/swift-apis as a SwiftPM dependency.

dan-zheng commented 3 years ago

Motivation

This enables building tensorflow/swift-models using stock toolchains from swift.org/download.

swift build will clone and build tensorflow/swift-apis as a regular SwiftPM dependency. Eventually, we would like to stop releasing custom toolchains bundled with pre-installed tensorflow/swift-apis.

Build instructions

It is possible to build tensorflow/swift-apis and dependencies like tensorflow/swift-models using stock toolchains by installing pre-built X10 libraries (currently available only for macOS and Windows).

After installing (e.g. to $HOME/Library on macOS), build with SwiftPM via the following:

$ swift build -Xcc -I$HOME/Library/tensorflow-2.4.0/usr/include -Xlinker -L$HOME/Library/tensorflow-2.4.0/usr/lib -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN

swift test is known not to work on macOS for tensorflow/swift-apis and dependencies due to SR-14008: Library not loaded: /usr/lib/swift/libswift_Differentiation.dylib.

Testing

Before merging, let's verify that swift build, swift run, and swift test works for swift.org/download toolchains across platforms:

[x] macOS: swift-DEVELOPMENT-SNAPSHOT-2021-01-04-a-osx.pkg
- swift run and swift test currently both fail due to SR-14008:

```console $ swift run Benchmarks -Xcc -I$HOME/Library/tensorflow-2.4.0/usr/include -Xlinker -L$HOME/Library/tensorflow-2.4.0/usr/lib -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN ld: warning: dylib (/Users/danielzheng/Library/tensorflow-2.4.0/usr/lib/libx10.dylib) was built for newer macOS version (10.15) than being linked (10.13) ld: warning: dylib (/Users/danielzheng/Library/tensorflow-2.4.0/usr/lib/libx10.dylib) was built for newer macOS version (10.15) than being linked (10.13) [2/2] Linking Benchmarks * Build Completed! dyld: Library not loaded: @rpath/libx10.dylib Referenced from: /Users/danielzheng/swift-models/.build/x86_64-apple-macosx/debug/Benchmarks Reason: image not found [1] 78739 abort swift run Benchmarks -Xcc -I$HOME/Library/tensorflow-2.4.0/usr/include ``` ```console $ swift test -Xcc -I$HOME/Library/tensorflow-2.4.0/usr/include -Xlinker -L$HOME/Library/tensorflow-2.4.0/usr/lib -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN 2021-01-08 07:11:00.826 xctest[78981:2111298] The bundle “swift-modelsPackageTests.xctest” couldn’t be loaded because it is damaged or missing necessary resources. Try reinstalling the bundle. 2021-01-08 07:11:00.826 xctest[78981:2111298] (dlopen_preflight(/Users/danielzheng/swift-models/.build/x86_64-apple-macosx/debug/swift-modelsPackageTests.xctest/Contents/MacOS/swift-modelsPackageTests): Library not loaded: /usr/lib/swift/libswift_Differentiation.dylib Referenced from: /Users/danielzheng/swift-models/.build/x86_64-apple-macosx/debug/swift-modelsPackageTests.xctest/Contents/MacOS/swift-modelsPackageTests Reason: image not found) ```

[ ] Ubuntu 18.04: swift-DEVELOPMENT-SNAPSHOT-2021-01-04-a-ubuntu18.04.tar.gz
[ ] Windows 10: swift-DEVELOPMENT-SNAPSHOT-2021-01-04-a-windows10.exe

novakov-alexey-zz commented 3 years ago

I can confirm the same run failure locally:

 swift-linear-regression % swift run -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN -Xcc -I/Library/tensorflow-2.4.0/usr/include -Xlinker -L/Library/tensorflow-2.4.0/usr/lib
dyld: Library not loaded: @rpath/libx10.dylib
  Referenced from: /Users/Alexey_Novakov/dev/git/swift-linear-regression/.build/x86_64-apple-macosx/debug/libTensorFlow.dylib
  Reason: image not found
zsh: abort      swift run -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN -Xcc  -Xlinker

philipturner commented 2 years ago

I can confirm the same run failure locally:

 swift-linear-regression % swift run -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN -Xcc -I/Library/tensorflow-2.4.0/usr/include -Xlinker -L/Library/tensorflow-2.4.0/usr/lib
dyld: Library not loaded: @rpath/libx10.dylib
  Referenced from: /Users/Alexey_Novakov/dev/git/swift-linear-regression/.build/x86_64-apple-macosx/debug/libTensorFlow.dylib
  Reason: image not found
zsh: abort      swift run -Xswiftc -DTENSORFLOW_USE_STANDARD_TOOLCHAIN -Xcc  -Xlinker

That may be a different failure. _Differentiation should now load. You could try copying the libx10.dylib binary into the Swift build products folder after compiling. In addition, it should not be necessary to pass libx10 into the linker settings for swift build or swift test, at least on macOS.

tensorflow / swift-models