search

tensorflow / swift

Swift for TensorFlow
https://tensorflow.org/swift
Apache License 2.0
6.12k stars 608 forks source link

Is it possible to use the S4TF toolchain with GitHub actions? #369

Open rickwierenga opened 4 years ago

rickwierenga commented 4 years ago

Hi, I wondered if it's possible to use the toolchain for GitHub action tests. If no, is there another option for CI tests? Thanks.

brettkoonce commented 4 years ago

you could set up jenkins locally and trigger it on the update, then post back to github.

dan-zheng commented 4 years ago

@rickwierenga: could you please clarify your use case involving GitHub actions and CI?

Currently, Swift for TensorFlow Ubuntu toolchains are published nightly at fixed Google Cloud Storage download links. There's no GitHub hook for when new nightly toolchains are released.

For reference, here's the Dockerfile that tensorflow/swift-apis uses as CI. It clones and installs the latest nightly Ubuntu toolchain and runs swift build/swift test for various repositories.

rickwierenga commented 4 years ago

Thanks. I'd like to run tests for an open source S4TF project I'm working on.

The docker file seems interesting. I'll see if I can use that.

rickwierenga commented 4 years ago

@dan-zheng I checked if I could get that to work, but it crashes with:

CMake Error: The source directory "/swift-apis" does not appear to contain CMakeLists.txt.
277
Specify --help for usage, or press the help button on the CMake GUI.
278
The command '/bin/sh -c cmake -G Ninja -D CMAKE_BUILD_TYPE=Release -D CMAKE_Swift_COMPILER=/swift-tensorflow-toolchain/usr/bin/swiftc -D USE_BUNDLED_CTENSORFLOW=YES -D TensorFlow_INCLUDE_DIR=/swift-tensorflow-toolchain/usr/lib/swift/linux/x86_64/modulemaps/CTensorFlow -D TensorFlow_LIBRARY=/swift-tensorflow-toolchain/usr/lib/swift/linux/libtensorflow.so -B /BinaryCache/tensorflow-swift-apis -S /swift-apis' returned a non-zero code: 1

See https://github.com/rickwierenga/s4tfpackage/runs/432779938 for the full log. It's weird because that repo seems to contain a CMakeLists file: https://github.com/tensorflow/swift-apis/blob/master/CMakeLists.txt. I'm have very little experience with Docker so excuse me if this is a dumb question.

brettkoonce commented 4 years ago

@rickwierenga i get the same error when trying to run sudo docker build https://raw.githubusercontent.com/tensorflow/swift-apis/master/Dockerfile but if you do a git clone https://github.com/tensorflow/swift-apis.git and then cd swift-apis and sudo docker build . that seems to work.

rickwierenga commented 4 years ago

Strange... The swift-apis repo should not be a dependency for the Dockerfile to work because that's the whole point of having Docker in the first place. Would you happen to know if the Dockerfile could be updated?

dan-zheng commented 4 years ago

@dan-zheng I checked if I could get that to work, but it crashes with:

CMake Error: The source directory "/swift-apis" does not appear to contain CMakeLists.txt.
277
Specify --help for usage, or press the help button on the CMake GUI.
278
The command '/bin/sh -c cmake -G Ninja -D CMAKE_BUILD_TYPE=Release -D CMAKE_Swift_COMPILER=/swift-tensorflow-toolchain/usr/bin/swiftc -D USE_BUNDLED_CTENSORFLOW=YES -D TensorFlow_INCLUDE_DIR=/swift-tensorflow-toolchain/usr/lib/swift/linux/x86_64/modulemaps/CTensorFlow -D TensorFlow_LIBRARY=/swift-tensorflow-toolchain/usr/lib/swift/linux/libtensorflow.so -B /BinaryCache/tensorflow-swift-apis -S /swift-apis' returned a non-zero code: 1

Sorry, tensorflow/swift-apis's CI (Dockerfile) breaks occasionally! It should pass most of the time - when it's broken, our top priority is to fix it.

The CMake error should be fixed now. It occurred at the time because CMake support was just being added and was unstable.

My request: could you please clarify what you mean by "how to use the S4TF toolchain with GitHub actions?" The question currently isn't particularly actionable - a specific request would be helpful!

rickwierenga commented 4 years ago

Thanks. I will check it out as soon as I get the chance (probably Sunday) and I'll let you know how it goes.

could you please clarify what you mean by "how to use the S4TF toolchain with GitHub actions?" The question currently isn't particularly actionable - a specific request would be helpful!

William Zhang (@WilliamHYZhang) and I are building an open source project where we would like to run tests online before merging PRs. For most swift packages this is possible through the Swift Action (https://github.com/actions/starter-workflows/blob/master/ci/swift.yml), but it understandably fails for packages requiring third party toolchains. The Dockerfile seems like a great solution for this, if it works :)

saeta commented 4 years ago

@rickwierenga if you get a nice setup, let's figure out how to document it so others can learn from and benefit from your configuration too!

xanderdunn commented 4 years ago

Here is a simple example of what you want to do @rickwierenga: https://github.com/xanderdunn/swift-action

It's much simpler and faster than using the swift-apis Dockerfile, which is getting and building many things. Just grabbing the pre-built binary is sufficient for a GitHub action.