Travis + Docker improvements

[dnoliver]

Similar to https://github.com/intel-iot-devkit/mraa/pull/785

This changes are intended to improve the current build infrastructure with Docker in Travis. Previously, there was a single Docker image that contains every build tool for C/C++, Node, Java, and Python. this make each Travis builder to install unnecessary packages to build a single target. Adding Android dependencies, made that image a ~4.5Gb image, so I tried to change that.

Now there is a Dockerfile hierarchy. There is a "base" image that provides C/C++ build dependencies, and other common dependencies. The "python" image install python2 and python3. The "node" image install an specific Node.js version. And the "java" image install jdk and jre. The build process is now a little bit more complex (you need to build the "base" image first, and then the other ones), but the final image size and build times are improved, which is nice :). All the Dockerfiles are now located in the "docker" folder.

There is an important change over there: using volumes. Previously, docker build copied all the sources from the host to the image, and then build the targets using the sources in the image. Now it is using a volume to mount all the sources into the running container. This has the following results:

1. The final image size is reduced, the build is faster
2. there is no image rebuild when the sources changes! it just mount the sources from the host to the container and use them.
3. the "build" folder is also available in the host file system, so you don't need to re-compile everything from scratch. Make decide which file is changed and only recompile what its needs
4. We can publish those images, and use them as ready-to-use build environments.

Android things build is still WIP. I had a problem with cmake using mraa and mraajava libs installed with apt-get. I figure out a workaround for it, and was able to generate android things packages for mraa. This is implemented in https://github.com/dnoliver/upm/tree/dnoliver/improve-docker-build-android branch.

Additionaly, I removed the bacnet lib install files, and implemented them in the Dockerfile

@Propanu @bjbeare @pylbert Android Things build is not here yet, but this is the foundation to make that happen.

[dnoliver]

@Propanu if I run travis lint for upm .travis.yml file it tells me:

dnoliver@dnoliver-vm:~/Git/upm$ travis lint
Warnings for .travis.yml:
[x] in compiler section: illegal value gcc-6, defaulting to gcc
[x] in compiler section: illegal value gcc-4.8, defaulting to gcc

I will fix that in this PR

[dnoliver]

@bjbeare and @vintummala confirmed that Android Things packages works! I just need to add android things as a target in travis now

[dnoliver]

@Propanu I fixed the travis matrix in ee7a2606830447840624ea7da0b4d92e83c255e. Before that, it was always running with the default gcc version (5.x I think).

Now we have them running with gcc-4.8 and gcc-6. gcc-4.8 has a problem with libopenzwave, so it fails. gcc-6 works fine.

Should we keep gcc-4.8?

[bjbeare]

The compiler version for this library isn't critical for us.

From: Nicolas Oliver [mailto:notifications@github.com] Sent: Thursday, July 13, 2017 1:55 PM To: intel-iot-devkit/upm upm@noreply.github.com Cc: Beare, Bruce J bruce.j.beare@intel.com; Mention mention@noreply.github.com Subject: Re: [intel-iot-devkit/upm] Travis + Docker improvements (#580)

@Propanuhttps://github.com/propanu I fixed the travis matrix in ee7a2606830447840624ea7da0b4d92e83c255e. Before that, it was always running with the default gcc version (5.x I think).

Now we have them running with gcc-4.8 and gcc-6. gcc-4.8 has a problem with libopenzwave, so it fails. gcc-6 works fine.

Should we keep gcc-4.8?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/intel-iot-devkit/upm/pull/580#issuecomment-315199378, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABIn9SboGRtyrqcahmy82mITD59ow3zQks5sNoQYgaJpZM4OUtMl.

[Propanu]

Heh, so that's why builds "worked" for GCC 4.8, I was seeing this before when trying builds by hand too. I'm perfectly fine with GCC 5 & 6 builds only since they would cover Ubuntu 16.04 and newer. With 4.8 one of the default linker flags messes things up either with the libopenzwave build or ours. Should be pretty easy to fix later on in case anyone needs to build with GCC 4.8 though.

[dnoliver]

@Propanu I think this is ready to merge!

[dnoliver]

@Propanu I will just need to configure the Sonar Qube project, so we can get a report for the master branch. I hope I don't need any other authorization

[Propanu]

Yeah, feel free to add the required variables to the Travis build environment for the project.

[dnoliver]

@Propanu travis is configured to run sonar scans. I will merge this into master branch, doing a rebase first of all the commits

[dnoliver]

I was checking why SONAR_TOKEN was not available in this PR, and I found this documentation about travis env vars:

Defining encrypted variables in .travis.yml #

Before adding sensitive data such as API credentials to your .travis.yml you need to encrypt it. Encrypted variables are not available to untrusted builds such as pull requests coming from another repository.

So, that is why SONAR_TOKEN is not available in this PR. I will merge this into master so I can check if sonar scans works

[dnoliver]

I am still on training wheels with this manual merges. I miss clicking the "merge" button and letting Github do the job.

I merged this PR into master https://github.com/intel-iot-devkit/upm/commit/ab4eeea61ed94e58d2de001958a9f823070b31f8

For reference, this is how I did it:

git clone https://github.com/intel-iot-devkit/upm.git

curl https://patch-diff.githubusercontent.com/raw/intel-iot-devkit/upm/pull/580.patch > 580.patch

git am -s 580.patch

git push

[Propanu]

There are a few ways you can achieve that. Here's another way of doing it, based on the command line suggestion Github shows:

git checkout -b newpatches
git pull https://github.com/dnoliver/upm.git dnoliver/improve-docker-build
git rebase master
git checkout master
git merge newpatches
git push

The reason I prefer this flow is that I can easily change the 3rd line to git rebase -i master and that allows me to squash, sign-off or edit commits in a PR. This might also come in handy if there are issues with the merge, or when someone opens a PR from a diverging master instead of a branch:

git fetch https://github.com/username/upm.git master && git cherry-pick <SHA>