Configure build on travis-ci

[chrhansen]

It's for my branch https://travis-ci.org/chrhansen/tensorflow.rb, will enable for this repository once the build works. I think that a couple fresh eyes on the .travis.yml file and the build errors https://travis-ci.org/chrhansen/tensorflow.rb we should be able to get this working quickly.

Good sources of info:

• https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/ci_build/Dockerfile.cpu
• https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/ci_build
• node-tensorflow got Travis-CI working, .travis.yml, some build tools
• saiprashanths/dl-docker

[chrhansen]

@sebastiandeutsch, here's my current status and a couple useful links! Happy to hear any suggestions or questions.

[geoffreylitt]

@chrhansen I'm playing with this a bit. I tried making a couple updates to your travis-cy-python branch (changes at my branch).

These helped get a few steps further, but my latest build on that branch still ends up failing at the same step as your last build on the travis-ci branch, where it can't find numpy.

I plan to keep digging at some point today or tomorrow but wanted to share intermediate progress in case it's helpful.

[geoffreylitt]

Also, have you thought about using Docker for Travis? If we added a Dockerfile to this repo, we could potentially:

• have the Docker "quickstart" dev environment version-controlled along with code (which it currently is not)
• share setup logic between the dev environment and the CI environment

[chrhansen]

@geoffreylitt I think the issue is that the Python that TensorFlow is using as default is whatever is returned by which python (https://github.com/tensorflow/tensorflow/blob/master/configure#L25) and on Travis-CI (Linux Trusty environment) that is in the /opt/... dir, however apt-get and pip are installing somewhere else. In the first ./configure-question I tried to specify /usr/lib/python2.7 which is where numpy gets installed, but I don't think it's quite correct.

At this point I think the issue is to figure out how use and install in the proper Python dir. Also, note that Travis have certain ways that default Python should be used https://docs.travis-ci.com/user/languages/python#Default-Python-Version. Eg perhaps numpy et al. installs need to go in the packages-block for it to work...

I made a new travis-ci-python branch in my fork so I wouldn't pollute this PR till I made some progress.

As you can see in my links above, I did consider using a Docker image, but I decided to wait for now as that could be another long road and I wanted the test environment to be build the same (or similar) way we describe in the README.md.

[geoffreylitt]

@chrhansen I agree that your diagnosis of the problem seems probable. This link looks like it could be helpful -- I think setting up virtualenv or using conda instead of pip might be helpful. Unfortunately I'm spoiled by rubygems :grin: and am not familiar enough with python dependency management to have good instincts for what is best.

re: Docker, it totally makes sense to try for the simplest solution first, but I'm also wondering if Docker might end up being easier to get working. Because there's already a working Docker image, if @nethsix could share the Dockerfile backing that image, I imagine it might be as simple as pointing Travis to the Dockerfile. Of course it might not be that simple, but if this route is getting painful it would be worth a try. And the payoff at the end of shared dev/test environment would be nice.

[nethsix]

@geoffreylitt That is a good suggestion. However, when I created the Docker image, I only partially used the Dockerfile. The reason:

• Docker was having problem compiling bazel, which is pretty similar to what @chrhansen encountered with TravisCI. Thus I partially created the Docker image up to the step of bazel compilation and from there access the Docker image through shell and run commands within it and then did Docker commits to complete the Docker image

Having gone through this exercise, maybe I can now create the image entirely from Dockerfile. Please give me a day to do it. I'll report back to you guys tomorrow.

[nethsix]

@geoffreylitt @chrhansen

• I have created a the Dockerfile in tools/docker/ (the directory structure is based of tensorflow/tensorflow repo) https://github.com/Arafatk/tensorflow.rb/pull/33
• I used CircleCI to build a docker image using the Docker file and ran specs inside the Docker container https://github.com/Arafatk/tensorflow.rb/pull/35

CircleCI successfully built everything, and ran the specs, but it takes 45 mins.

Currently the circle.yml config is set to only run when someone pushes to a branch with 'circle_ci' in the branch name. I left it at that because it currently takes ~45 mins because it is building everything from scratch. I am not sure if Circle CI is smart enough, to re-build only changes. Maybe there is a way to set it up. I will look into it. Alternatively, it is easy to pre-built a Docker image (with all prereqs and Tensorflow shared library) and then during CI, we just compile the ruby-tensorflow code in the pre-build Docker image.

[chrhansen]

@nethsix that's great! I've added a comment over in https://github.com/Arafatk/tensorflow.rb/pull/35#issuecomment-235699079.

[chrhansen]

Closing this as @nethsix got a build to work on Circle-CI #41 instead.