Replace puppet 'docker' with containers

Our intention with our docker containers, was to use them for unit testing our application as a whole, while testing the validity of the puppet scripts, used to build our development environment. However, this basis is not really valid, since we implement the docker puppet environment, which is beginning to change from the vagrant environment. This means, our docker containers are no longer checking the validity of puppet logic, used to build our development environment. And since the requirements of docker, and vagrant is not always a one-to-one relationship, we won't always be able to reuse the exact puppet script(s) between the vagrant and docker puppet environments.

Additionally, running puppet in docker, is similarly flawed to #2932. Therefore, we will eliminate our puppet implementation, within our docker containers, used for unit testing. This means we'll remove entirely the docker puppet environment, create an equal number of dockerfile's, as the number of puppet modules defined in our vagrant puppet environment, and adjust our .travis.yml, respectively.

Our current build, and run instance of our browserify container:

root@trusty64:/vagrant# docker build -f dockerfile/browserify.dockerfile -t jeff1evesque/ml-browserify:0.7 .
Sending build context to Docker daemon  76.01MB
Step 1/7 : FROM node:9
 ---> aa3e171e4e95
Step 2/7 : ENV ROOT_PROJECT /var/machine-learning
 ---> Running in 774b755c836d
Removing intermediate container 774b755c836d
 ---> e57684123c31
Step 3/7 : COPY src/package.json src/jsx $ROOT_PROJECT/
 ---> 8818cbce7bc1
Step 4/7 : RUN mkdir -p $ROOT_PROJECT/interface/static/js
 ---> Running in b96518135fc9
Removing intermediate container b96518135fc9
 ---> 0f11d7925a8a
Step 5/7 : WORKDIR $ROOT_PROJECT/src
Removing intermediate container 485088ac4294
 ---> c53cdb79c31e
Step 6/7 : RUN npm install
 ---> Running in fca1ca308321
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN react-rangeslider@2.2.0 requires a peer of react@^0.14.0 || ^15.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN reactjs@1.0.0 license should be a valid SPDX license expression

added 350 packages in 17.153s
Removing intermediate container fca1ca308321
 ---> e015854a0e48
Step 7/7 : CMD ["npm", "run-script", "watch:jsx"]
 ---> Running in 2269243305a7
Removing intermediate container 2269243305a7
 ---> 60e9c64671cd
Successfully built 60e9c64671cd
Successfully tagged jeff1evesque/ml-browserify:0.7
root@trusty64:/vagrant# docker run --hostname browserify --name browserify -it jeff1evesque/ml-browserify:0.7

> reactjs@1.0.0 watch:jsx /var/machine-learning
> watch 'npm run build:js' src/jsx'

sh: 1: Syntax error: Unterminated quoted string
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! reactjs@1.0.0 watch:jsx: `watch 'npm run build:js' src/jsx'`
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the reactjs@1.0.0 watch:jsx script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-04-14T12_07_32_214Z-debug.log

Our browserify container builds:

root@trusty64:/vagrant# docker build -f dockerfile/browserify.dockerfile -t jeff1evesque/ml-browserify:0.7 .
Sending build context to Docker daemon  76.03MB
Step 1/7 : FROM node:9
 ---> aa3e171e4e95
Step 2/7 : ENV ROOT_PROJECT /var/machine-learning
 ---> Running in 1f0daa4ce8be
Removing intermediate container 1f0daa4ce8be
 ---> 92526a603a03
Step 3/7 : COPY src/package.json src/jsx $ROOT_PROJECT/
 ---> 7f7a8123d532
Step 4/7 : RUN mkdir -p $ROOT_PROJECT/interface/static/js
 ---> Running in 91e39c4e6aeb
Removing intermediate container 91e39c4e6aeb
 ---> df6e0684a17f
Step 5/7 : WORKDIR $ROOT_PROJECT/src
Removing intermediate container 286e66fbda0d
 ---> cd87270cdf32
Step 6/7 : RUN npm install
 ---> Running in 67ff9da6db9b
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN react-rangeslider@2.2.0 requires a peer of react@^0.14.0 || ^15.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN reactjs@1.0.0 license should be a valid SPDX license expression
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.1.3 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 402 packages in 19.592s
Removing intermediate container 67ff9da6db9b
 ---> 88d792a81663
Step 7/7 : CMD ["npm", "run", "watch:jsx"]
 ---> Running in 1d50411d244a
Removing intermediate container 1d50411d244a
 ---> 13c8633cca72
Successfully built 13c8633cca72
Successfully tagged jeff1evesque/ml-browserify:0.7

Additionally, it will run without error:

root@trusty64:/vagrant# docker run --hostname browserify --name browserify -d jeff1evesque/ml-browserify:0.7
2294ecf797f96c705e84fedd00a4063d0a3f0c90264eaa2deab7eb45d0ac1d09
root@trusty64:/vagrant#
root@trusty64:/vagrant#
root@trusty64:/vagrant# docker ps -a
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS                           PORTS               NAMES
2294ecf797f9        jeff1evesque/ml-browserify:0.7   "npm run watch:jsx"      3 seconds ago       Up 2 seconds                                         browserify
d2850ce0daa4        jeff1evesque/ml-sass:0.7         "/bin/sh -c 'node-sa…"   33 hours ago        Up 33 hours                                          sass
root@trusty64:/vagrant#

Our browserify container is capable of compiling our jsx scripts:

root@trusty64:/vagrant# docker run --hostname browserify --name browserify -d jeff1evesque/ml-browserify:0.7
8956b39ec92514f157d02ef48cfe5bb36a5c489a96e0d85dfb43f90bda09f5f1
root@trusty64:/vagrant# docker ps -a
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS                     PORTS               NAMES
8956b39ec925        jeff1evesque/ml-browserify:0.7   "npm run watch:jsx"      3 seconds ago       Up 2 seconds                                   browserify
d2850ce0daa4        jeff1evesque/ml-sass:0.7         "/bin/sh -c 'node-sa…"   35 hours ago        Up 35 hours                                    sass
root@trusty64:/vagrant# docker exec -it browserify /bin/bash
root@browserify:/var/machine-learning/src# ls ../interface/static/js/
root@browserify:/var/machine-learning/src# touch jsx/content.jsx
root@browserify:/var/machine-learning/src# ls -l ../interface/static/js/
total 0
-rw-r--r-- 1 root root 0 Apr 14 15:01 content.js.tmp-browserify-65896583659321983362
jsot@browserify:/var/machine-learning/src# ls -l ../interface/static/js/content.js
-rw-r--r-- 1 root root 3112539 Apr 14 15:01 ../interface/static/js/content.js
root@browserify:/var/machine-learning/src#

However, we need to initiate the onchange npm watcher. Therefore, we'll likely need to trigger the initial compile, via some touch equivalent, after the entire rancher build succeeds. This can likely be accomplished, by putting the corresponding logic, at the very end of the install_rancher script.

We simplified our sass container, then verified it was able to compile:

root@trusty64:/vagrant# docker run --hostname sass --name sass -d jeff1evesque/ml-sass:0.7
ca181cd8b6c1978746edf9c20ef203e6b23672c7f4e19ae81ae5f954cb77f99d
root@trusty64:/vagrant# docker exec -it sass /bin/bash
root@sass:/var/machine-learning/src# ls -l /var/machine-learning/interface/static/css
total 0
root@sass:/var/machine-learning/src# touch /var/machine-learning/src/scss/style.scss
root@sass:/var/machine-learning/src# ls -l /var/machine-learning/interface/static/css
-rw-r--r-- 1 root root 168520 Apr 14 20:35 style.css

After running a fresh install_rancher, we notice 10 minutes after installation, all our containers are Active, and running, except our gunicorn webserver instances:

nginx-error

Upon visiting the corresponding webserver-web logs, via the rancher gui:

Removing bad service instance: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\"/factory.py\\" to rootfs \\"/mnt/sda1/var/lib/docker/aufs/mnt/c6406b2121ad361d34e9e13c0d87a269e127b2783136602b225e70b2f105f6cf\\" at \\"/mnt/sda1/var/lib/docker/aufs/mnt/c6406b2121ad361d34e9e13c0d87a269e127b2783136602b225e70b2f105f6cf/var/machine-learning/factory.py\\" caused \\"not a directory\\"\"" : Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Upon visiting the corresponding webserver-api logs, via the rancher gui:

Expected state running but got error: Error response from daemon: pull access denied for jeff1evesque/ml-webserver0.7, repository does not exist or may require 'docker login'

Note: each rancher instance, repeats their respective errors, as shown above.

The above errors could coincide with the fact that we ran our install_rancher, before we refactored the script (i.e. 4d90765). So, container provisioning did not execute. Specifically, we removed the conditional logic, which attempted to detect whether docker was running in docker toolbox; in hopes to prepare for varying docker execute syntax. But, this was found to be unnecessary, since earlier within the install_rancher, we specify the docker-machine env, which designates the location endpoint, where the docker command should execute against. So, we can proceed by either attempting these commands manually, or rerunning our install_rancher.

We'll also need to add additional volumes for each database container instance, to allow each database to be persistent. Lastly, when all containers are properly wired up, we'll need to reconfigure our unit-tests, to account for the changes we have made in this issue. This way our travis ci will not incorrectly indicate a broken build, for successive commits.

When we attempt to run the following provisioning commands:

docker exec mariadb -d sh -c "\
    cd $ROOT_PUPPET/code/modules/mariadb/scripts && \
    python setup_tables.py $ROOT_PUPPET/puppet"

docker exec mongodb -d sh -c '\
    /usr/bin/mongod --fork --config /etc/mongod.conf && \
    cd /root/build && \
    ./create-mongodb-users'

docker exec browserify -d sh -c "\
    touch $ROOT_PROJECT/src/jsx/content.jsx"

docker exec sass -d sh -c "\
    touch $ROOT_PROJECT/src/scss/style.scss"

We receive the following errors:

Error: No such container: mariadb
Error: No such container: mongodb
Error: No such container: browserify
Error: No such container: sass

So, we'll need to investigate whether the following docker-machine properties, need to be adjusted:

$ docker-machine env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="C:\Users\jeff1evesque\.docker\machine\machines\default"
export DOCKER_MACHINE_NAME="default"
export COMPOSE_CONVERT_WINDOWS_PATHS="true"
# Run this command to configure your shell:
# eval $("C:\Program Files\Docker Toolbox\docker-machine.exe" env)

The docker-compose provides the command directive, which passes the respective value, to the ENTRYPOINT argument, of the corresponding dockerfile. Therefore, the syntax is simplified, by relocating the provisioning syntax from our install_rancher, to the docker-compose.development.yml. This means, before testing our adjusted implementation, we'll need to recreate the respective images, then push the update, to our dockerhub repository.

Note: additional references for passing the command arguments to the dockerfile ENTRYPOINT:

https://stackoverflow.com/a/30064175

In general, the docker-compose.yml allows for relative paths, via version: 3 (not currently supported by rancher), and via syntax such as ./, or ${PWD}. However, rancher does not honor these approaches, and requires by default, an absolute path, when implementing volumes, from the host to the container.

So, we'll need to adjust our install_rancher, to acquire the current $PWD, and perform string matching, and replacing. Specifically, we'll rename our current rancher-compose.development.yml as rancher-template.development.yml. Our install_rancher will copy the former file as the latter, then perform string replacing, to build a full absolute path to corresponding files and directories, needed in the volumes.

53cba71: now, we need to build logic within our install_rancher, before executing rancher stacks, such that rancher-template.development.yml is copied as rancher-compose.development.yml, while ensuring the PREPATH keyword is replaced by the value of $(PWD) from our install_rancher. However, we need to ensure if the host is windows, that proper treatment is performed to enforce /c/style/path. Additionally, the adjusted PREPATH cannot contain the : special character. So, this can be stripped out.

Now, we are able to guaranty that local host files, and directories, can be mounted into the rancher browserify container:

browserify

Furthermore, we were able to run touch jsx/content.jsx, from the same above rancher shell terminal, for MLStack-browserify-1. However, the corresponding webcompiler, did not behave as expected, since it compiled a 0KB temporary file, as shown on the following host windows explorer:

browserify

We were able to touch the contents of our sass container:

sass

Then, view the corresponding compiled result on the host machine:

sass

So, we can attempt to apply dos2unix, recursively on the src/jsx directory, from the earlier browserify container. Then, follow up with a touch src/jsx/content.jsx. If the webcompiler succeeds, then we'll have to build this step into the script directive of the package.json. Otherwise, we'll likely need to investigate whether browserify and it's dependencies are properly wired up.

Running the following browserify command manually, within a pure browserify docker container, created via docker build (versus rancher), succeeded:

root@browserify:/var/machine-learning/src# browserify -t [ babelify --presets env,stage-2,react ] /var/machine-learning/src/jsx/content.jsx -o /var/machine-learning/interface/static/js/content.js

Specifically, the corresponding asset was compiled:

root@browserify:/var/machine-learning/src# ls -l /var/machine-learning/interface/js
total 3044
-rw-r--r-- 1 root root 3115336 Apr 18 00:39 content.js

Running the browserify command manually with the rancher terminal also succeeds:

root@browserify:/var/machine-learning/src# ls -l ../interface/static/js/
total 0
root@browserify:/var/machine-learning/src# npm install -g browserify
/usr/local/bin/browserify -> /usr/local/lib/node_modules/browserify/bin/cmd.js
+ browserify@16.2.0
added 135 packages in 7.18s

   ╭─────────────────────────────────────╮
   │                                     │
   │   Update available 5.6.0 → 5.8.0    │
   │     Run npm i -g npm to update      │
   │                                     │
   ╰─────────────────────────────────────╯

root@browserify:/var/machine-learning/src# browserify -t [ babelify --presets env,stage-2,react ] /var/machine-learning/src/jsx/content.jsx -o /var/machine-learning/interface/static/js/content.js
root@browserify:/var/machine-learning/src# ls -l ../interface/static/js/total 3044
-rwxrwxrwx 1 node staff 3115335 Apr 18 11:44 content.js

Additionally, we noticed the corresponding compiled file on the host:

browserify

So, we'll likely need to refactor our implementation, by possibly removing the onchange implementation from our package.json. We know it's possible to manually install only the browserify npm package globally, and have the desired result (without a watcher). Additionally, it seems that an inotifywait directive, with a dockerfile worked for others:

https://ryanfb.github.io/etc/2015/09/24/dependencies_for_docker-compose_with_inotify.html

Therefore, we may simply migrate the installation of browserify from the package.json, to install globally with our corresponding browserify.dockerfile, as well remove the scripts directive from package.json, by implementing corresponding logic similar to the above example.

Our current reimplemented npm browserify.dockerfile properly performs an initial compile:

root@trusty64:/vagrant# docker run --hostname browserify --name browserify -it jeff1evesque/ml-browserify:0.7
[...DOS2UNIX-TRACE-OMITTED...]
> reactjs@1.0.0 build:browserify /var/machine-learning/src
> browserify -t [ babelify --presets env,stage-2,react ] /var/machine-learning/src/jsx/content.jsx -o /var/machine-learning/interface/static/js/content.js

Then, in a second terminal window, we notice our content.jsx was created:

root@browserify:/var/machine-learning/src# date
Fri Apr 20 22:14:27 UTC 2018
root@browserify:/var/machine-learning/src# ls -l ../interface/static/js/
total 3044
-rw-r--r-- 1 root root 3114845 Apr 20 22:14 content.js
root@browserify:/var/machine-learning/src# touch jsx/import/result/results.jsx
root@browserify:/var/machine-learning/src# ls -l ../interface/static/js/
total 3044
-rw-r--r-- 1 root root 3114845 Apr 20 22:14 content.js
-rw-r--r-- 1 root root       0 Apr 20 22:16 content.js.tmp-browserify-06997664755896493283
root@browserify:/var/machine-learning/src# ls -l ../interface/static/js/
total 3044
-rw-r--r-- 1 root root 3114845 Apr 20 22:16 content.js

Now, we need to test our improved npm implementation within rancher.

Our browserify container keeps rebooting in rancher. However, before rebooting, the following Standard Error can be obtained from the rancher GUI:

4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/reducer/layout.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/reducer/login.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/reducer/page.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/store.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/result/current-result.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/result/results.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/session-type/data-append.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/session-type/data-new.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/session-type/model-generate.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/session-type/model-predict.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/svg/svg-books.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/svg/svg-home.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/svg/svg-pencil-note.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/svg/svg-user.jsx to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-email.js to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-file.js to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-float.js to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-int.js to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-password.js to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-string.js to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-url.js to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/package.json to Unix format ...
4/20/2018 6:29:39 PMdos2unix: converting file /var/machine-learning/src/jsx/README.md to Unix format ...
4/20/2018 6:30:34 PMSetting up watches.  Beware: since -r was given, this may take a while!
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/content.jsx to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/animation/animate.jsx to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/content/home-page.jsx to Unix format ...
4/20/2018 6:30:34 PMWatches established.
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/content/login.jsx to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/content/register.jsx to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/formatter/transpose.js to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/general/ajax-caller.js to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/general/breakpoints.js to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/general/colors.js to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/general/range-slider.jsx to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/general/spinner.jsx to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/general/submit-button.jsx to Unix format ...
4/20/2018 6:30:34 PMdos2unix: converting file /var/machine-learning/src/jsx/import/input-data/supply-dataset-file.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/input-data/supply-dataset-url.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/input-data/supply-predictors.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/kernel/sv-kernel.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/layout/analysis.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/layout/login.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/layout/page.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/layout/register.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/model/model-type.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/header-menu.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/menu-items/current-result.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/menu-items/home.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/menu-items/login.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/menu-items/menu.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/menu-items/register.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/menu-items/results.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/nav-bar.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/navigation/user-menu.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/action/current-result.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/action/login.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/action/logout.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/action/page.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/analysis-layout.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/current-result.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/data-append.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/data-new.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/header-menu.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/home-page.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/login-link.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/login.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/model-generate.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/model-predict.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/page.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/range-slider.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/register-link.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/register.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/results.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/review-results-link.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/container/user-menu.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/reducer/data.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/reducer/layout.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/reducer/login.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/reducer/page.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/redux/store.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/result/current-result.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/result/results.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/session-type/data-append.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/session-type/data-new.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/session-type/model-generate.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/session-type/model-predict.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/svg/svg-books.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/svg/svg-home.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/svg/svg-pencil-note.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/svg/svg-user.jsx to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-email.js to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-file.js to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-float.js to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-int.js to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-password.js to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-string.js to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/import/validator/valid-url.js to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/package.json to Unix format ...
4/20/2018 6:30:35 PMdos2unix: converting file /var/machine-learning/src/jsx/README.md to Unix format ...

Our current implementation succeeds within rancher:

rancher

Additionally, the changes are available within the host:

compiled

However, we haven't successfully devised a mechanism to ensure an initial build.

Our browserify container sufficiently compiles during initial build:

browserify

As well as having the compiled asset available on the host:

compiled

Note: prior to the above touch, content.js was created at 11:18pm, with respect to the initial build.

Our sass container sufficiently compiles during initial build:

scss

As well as having the compiled asset available on the host:

compiled

Note: prior to the above touch, style.css was created at 8:02am, with respect to the initial build.

Our mariadb properly builds, and uses it's corresponding entrypoint bash script:

root@trusty64:/vagrant# docker build -f dockerfile/mariadb.dockerfile -t jeff1evesque/ml-mariadb:0.7 .Sending build context to Docker daemon  79.88MB
Step 1/8 : FROM jeff1evesque/ml-base:0.7
 ---> b41e451df1a8
Step 2/8 : ENV ENVIRONMENT=docker    PUPPET=/opt/puppetlabs/bin/puppet    ROOT_PUPPET=/etc/puppetlabs    MODULES=$ROOT_PUPPET/code/modules    CONTRIB_MODULES=$ROOT_PUPPET/code/modules_contrib
 ---> Running in 14572b399c00
Removing intermediate container 14572b399c00
 ---> f3afad1a1872
Step 3/8 : COPY puppet/environment/$ENVIRONMENT/modules/mariadb $MODULES/mariadb
 ---> 01d2d6e66a38
Step 4/8 : COPY hiera $ROOT_PUPPET/puppet/hiera
 ---> db6aff8a2e57
Step 5/8 : RUN $PUPPET apply -e 'class { mariadb: }'    --modulepath=$CONTRIB_MODULES:$MODULES    --confdir=$ROOT_PUPPET/puppet
 ---> Running in 2ef393bcd580
Warning: Unknown variable: 'python::params::provider'. at /etc/puppetlabs/code/modules_contrib/python/manifests/init.pp:56:25
Warning: Unknown variable: 'python::params::source'. at /etc/puppetlabs/code/modules_contrib/python/manifests/init.pp:61:25
Warning: Unknown variable: 'redis_version'. at /etc/puppetlabs/code/modules/mariadb/manifests/dependency.pp:13:9
Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Notice: Compiled catalog for 2ef393bcd580.localdomain in environment production in 0.67 seconds
Notice: /Stage[main]/Mariadb::Dependency/Package[pyyaml]/ensure: created
Notice: /Stage[main]/Mariadb::Dependency/File[/log]/ensure: created
Notice: /Stage[main]/Mariadb::Dependency/File[/log/database]/ensure: created
Notice: /Stage[main]/Mysql::Server::Install/Package[mysql-server]/ensure: created
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}77f15d6c87f9c136c4efcda072017f71' to '{md5}750b8ea845b2d8960680dad20f306b3e'
Notice: /Stage[main]/Mysql::Server::Service/File[/var/log/mysql/error.log]/group: group changed 'adm' to 'mysql'
Notice: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: ensure changed 'stopped' to 'running'
Notice: /Stage[main]/Mysql::Server::Root_password/Mysql_user[root@localhost]/password_hash: defined 'password_hash' as '*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19'
Notice: /Stage[main]/Mysql::Server::Root_password/File[/root/.my.cnf]/ensure: defined content as '{md5}5212916f2644f1c40d2601c69b0075dc'
Notice: /Stage[main]/Mysql::Server::Providers/Mysql_user[authenticated@%]/ensure: created
Notice: /Stage[main]/Mysql::Server::Providers/Mysql_user[provisioner@%]/ensure: created
Notice: /Stage[main]/Mysql::Server::Providers/Mysql_user[tester@%]/ensure: created
Notice: /Stage[main]/Mysql::Server::Providers/Mysql_grant[authenticated@%/db_machine_learning.*]/ensure: created
Notice: /Stage[main]/Mysql::Server::Providers/Mysql_grant[provisioner@%/db_machine_learning.*]/ensure: created
Notice: /Stage[main]/Mysql::Server::Providers/Mysql_grant[tester@%/db_machine_learning.*]/ensure: created
Notice: /Stage[main]/Mysql::Server::Providers/Mysql_database[db_machine_learning]/ensure: created
Notice: /Stage[main]/Mysql::Client::Install/Package[mysql_client]/ensure: created
Notice: /Stage[main]/Mysql::Bindings::Python/Package[python-mysqldb]/ensure: created
Notice: Applied catalog in 37.87 seconds
Removing intermediate container 2ef393bcd580
 ---> a7c6de1d1032
Step 6/8 : RUN printf "#!/bin/bash\n\ncd $MODULES/mariadb/scripts\nservice mysql start\npython setup_tables.py $ROOT_PUPPET/puppet\nservice mysql stop\nmysqld\n" > entrypoint
 ---> Running in 8f86b59d737a
Removing intermediate container 8f86b59d737a
 ---> c24ecfa3a325
Step 7/8 : RUN chmod 710 entrypoint
 ---> Running in daf129e4fcb3
Removing intermediate container daf129e4fcb3
 ---> 160dc6ff8341
Step 8/8 : ENTRYPOINT ["/bin/bash", "-c", "./entrypoint"]
 ---> Running in f8b568ba0840
Removing intermediate container f8b568ba0840
 ---> 7749faa477fd
Successfully built 7749faa477fd
Successfully tagged jeff1evesque/ml-mariadb:0.7
root@trusty64:/vagrant# docker run --hostname mariadb --name mariadb -it jeff1evesque/ml-mariadb:0.7   * Starting MariaDB database server mysqld                                                     [ OK ]
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
 * Stopping MariaDB database server mysqld                                                     [ OK ]
180421 19:05:29 [Note] mysqld (mysqld 5.5.59-MariaDB-1ubuntu0.14.04.1) starting as process 646 ...

Then, in a second terminal window, we notice that our database tables were provisioned:

root@trusty64:/vagrant# docker exec -it mariadb /bin/bash
root@mariadb:/# mysql -u root -p db_machine_learning
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1
Server version: 5.5.59-MariaDB-1ubuntu0.14.04.1 (Ubuntu)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [db_machine_learning]> show tables;
+-----------------------------------+
| Tables_in_db_machine_learning     |
+-----------------------------------+
| tbl_dataset_entity                |
| tbl_model_type                    |
| tbl_prediction_results            |
| tbl_svm_results_class             |
| tbl_svm_results_decision_function |
| tbl_svm_results_probability       |
| tbl_svr_results_r2                |
| tbl_user                          |
+-----------------------------------+
8 rows in set (0.00 sec)

Now, we need to further the testing into rancher, and ensure that our volumes are properly defined.

Our mariadb container also yields the same result, within rancher:

mariadb

Our mongodb container sufficiently provisions the database, with the corresponding authenticated user:

mongodb

The full traceback, can be reviewed as follows:

root@mongodb:/# mongo
MongoDB shell version: 3.2.19
connecting to: test
> use admin
switched to db admin
> db.auth('authenticated', 'password');
1
> db.getUsers();
[
        {
                "_id" : "admin.authenticated",
                "user" : "authenticated",
                "db" : "admin",
                "roles" : [
                        {
                                "role" : "readWrite",
                                "db" : "admin"
                        },
                        {
                                "role" : "userAdmin",
                                "db" : "admin"
                        },
                        {
                                "role" : "dbAdmin",
                                "db" : "admin"
                        },
                        {
                                "role" : "readWrite",
                                "db" : "dataset"
                        },
                        {
                                "role" : "userAdmin",
                                "db" : "dataset"
                        },
                        {
                                "role" : "dbAdmin",
                                "db" : "dataset"
                        }
                ]
        }
]
>

We may need to revisit our nginx-xxx containers, to confirm configurations are properly setup. However, the rancher GUI currently provides some syntax errors for our webserver-api container:

4/22/2018 4:46:37 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 578, in spawn_worker
4/22/2018 4:46:37 PM    worker.init_process()
4/22/2018 4:46:37 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 126, in init_process
4/22/2018 4:46:37 PM    self.load_wsgi()
4/22/2018 4:46:37 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 135, in load_wsgi
4/22/2018 4:46:37 PM    self.wsgi = self.app.wsgi()
4/22/2018 4:46:37 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
4/22/2018 4:46:37 PM    self.callable = self.load()
4/22/2018 4:46:37 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
4/22/2018 4:46:37 PM    return self.load_wsgiapp()
4/22/2018 4:46:37 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
4/22/2018 4:46:37 PM    return util.import_app(self.app_uri)
4/22/2018 4:46:37 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 352, in import_app
4/22/2018 4:46:37 PM    __import__(module)
4/22/2018 4:46:37 PM  File "/var/machine-learning/factory.py", line 21, in <module>
4/22/2018 4:46:37 PM    from interface.views_web import blueprint_web
4/22/2018 4:46:37 PMImportError: No module named views_web
[...REMAINING-TRACE-OMITTED...]

Similarly, our webserver-web container:

4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 231, in run
4/22/2018 4:47:24 PM    self.halt(reason=inst.reason, exit_status=inst.exit_status)
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 344, in halt
4/22/2018 4:47:24 PM    self.stop()
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 393, in stop
4/22/2018 4:47:24 PM    time.sleep(0.1)
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 244, in handle_chld
4/22/2018 4:47:24 PM    self.reap_workers()
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 524, in reap_workers
4/22/2018 4:47:24 PM    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
4/22/2018 4:47:24 PMgunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
4/22/2018 4:47:24 PM[2018-04-22 16:47:24 +0000] [11] [ERROR] Exception in worker process
4/22/2018 4:47:24 PMTraceback (most recent call last):
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 578, in spawn_worker
4/22/2018 4:47:24 PM    worker.init_process()
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 126, in init_process
4/22/2018 4:47:24 PM    self.load_wsgi()
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 135, in load_wsgi
4/22/2018 4:47:24 PM    self.wsgi = self.app.wsgi()
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
4/22/2018 4:47:24 PM    self.callable = self.load()
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
4/22/2018 4:47:24 PM    return self.load_wsgiapp()
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
4/22/2018 4:47:24 PM    return util.import_app(self.app_uri)
4/22/2018 4:47:24 PM  File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 352, in import_app
4/22/2018 4:47:24 PM    __import__(module)
4/22/2018 4:47:24 PM  File "/var/machine-learning/factory.py", line 20, in <module>
4/22/2018 4:47:24 PM    from interface.views_api import blueprint_api
4/22/2018 4:47:24 PMImportError: No module named views_api
[...REMAINING-TRACE-OMITTED...]

Our current rancher build, yields all our containers being Active:

rancher

Unfortunately, the current web application (likely the api instance too) does not render in the browser:

localhost

So, we'll need to take a closer look at the nginx-xxx integration, with respect to its corresponding webserver-xxx instance. Then, we'll need to adjust our unit-tests, which should partially verify our api implementation. Lastly, we may choose to test whether the following containers are persistent:

mariadb
mongodb
redis
nginx-xxx

Our unit tests are now able to execute via the travis ci, as well as locally:

[...BUILD-TRACE-OMITTED...]
TOTAL                                                                                  2576   1416    45%

=============================== warnings summary ===============================
3_programmatic_interface/results/pytest_1_svm_prediction.py::test_save_prediction
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'probability' at row 1
    self.cursor.execute(statement, sql_args)
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'decision_function' at row 1
    self.cursor.execute(statement, sql_args)

3_programmatic_interface/results/pytest_2_svr_prediction.py::test_save_prediction
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'r2' at row 1
    self.cursor.execute(statement, sql_args)

-- Docs: http://doc.pytest.org/en/latest/warnings.html
============== 32 failed, 16 passed, 3 warnings in 960.52 seconds ==============
Error: unit test exited.

Although, both unit test approach execute, each fail at varying margins. For example, the locally run unit tests have 32 failed tests, and 16 passed tests. However, the travis ci, reports 46 failed tests, and 2 passed tests. So, we'll need to dig a little deeper, and determine whether our docker implementation, puppet build, or python backend code needs to be adjusted.

Our travis ci results, are now consistent with earlier local results:

[...BUILD-TRACE-OMITTED...]
TOTAL                                                                                  2576   1416    45%
=============================== warnings summary ===============================
3_programmatic_interface/results/pytest_1_svm_prediction.py::test_save_prediction
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'probability' at row 1
    self.cursor.execute(statement, sql_args)
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'decision_function' at row 1
    self.cursor.execute(statement, sql_args)
3_programmatic_interface/results/pytest_2_svr_prediction.py::test_save_prediction
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'r2' at row 1
    self.cursor.execute(statement, sql_args)
-- Docs: http://doc.pytest.org/en/latest/warnings.html
============== 32 failed, 16 passed, 3 warnings in 916.23 seconds ==============
Error: unit test exited.

From a quick glance of the full error trace, we notice multiple failed attempts to connect to the mongodb database, which would certainly cascade to successive failures, since the corresponding dataset(s), used for analysis does not exist:

[...EARLIER-TRACE-OMITTED...]
            while not server_descriptions:
                # No suitable servers.
                if server_timeout == 0 or now > end_time:
                    raise ServerSelectionTimeoutError(
>                       self._error_message(selector))
E                   ServerSelectionTimeoutError: mongodb:27017: [Errno 111] Connection refused
[...SUCCESSIVE-TRACE-OMITTED...]

As a side note: our travis ci seems to require between 19-21 minutes to run a full verbose test. This means after our above refactored docker + puppet build, we have not lost performance during testing. If anything it has sped up by 1-2 minutes, compared to earlier travis ci builds. However, if our earlier mongodb problems are repaired, it is unlikely to add more than 1-2 minutes to the overall cycle time. Additionally, unit tests can be executed via the fast flag, which eliminates building each container manually, and instead relies on the downloading corresponding docker containers, fully provisioned, and ready for use.

2a87f26: our local verbose unit test passes all tests:

[...BUILD-TRACE-OMITTED...]
============================= test session starts ==============================
platform linux2 -- Python 2.7.6, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /var/machine-learning/test/live_server, inifile: pytest.ini
plugins: flask-0.10.0, cov-2.5.1
collected 48 items

test/live_server/1_authentication/pytest_1_crypto.py .                   [  2%]
test/live_server/1_authentication/pytest_2_account_registration.py .     [  4%]
test/live_server/1_authentication/pytest_3_validate_password.py .        [  6%]
test/live_server/1_authentication/pytest_4_user_login.py .               [  8%]
test/live_server/1_authentication/pytest_5_user_logout.py .              [ 10%]
test/live_server/2_database/pytest_svm_dbmax.py ..........               [ 31%]
test/live_server/2_database/pytest_svr_dbmax.py ..........               [ 52%]
test/live_server/3_programmatic_interface/dataset_url/pytest_svm_dataset_url.py . [ 54%]
...                                                                      [ 60%]
test/live_server/3_programmatic_interface/dataset_url/pytest_svr_dataset_url.py . [ 62%]
...                                                                      [ 68%]
test/live_server/3_programmatic_interface/file_upload/pytest_svm_file_upload.py . [ 70%]
...                                                                      [ 77%]
test/live_server/3_programmatic_interface/file_upload/pytest_svr_file_upload.py . [ 79%]
...                                                                      [ 85%]
test/live_server/3_programmatic_interface/results/pytest_1_svm_prediction.py . [ 87%]
..                                                                       [ 91%]
test/live_server/3_programmatic_interface/results/pytest_2_svr_prediction.py . [ 93%]
..                                                                       [ 97%]
test/live_server/3_programmatic_interface/results/pytest_3_all_prediction_titles.py . [100%]

---------- coverage: platform linux2, python 2.7.6-final-0 -----------
Name                                                                                  Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------
__init__.py                                                                               0      0   100%
app.py                                                                                    7      7     0%
brain/__init__.py                                                                         0      0   100%
brain/cache/__init__.py                                                                   0      0   100%
brain/cache/hset.py                                                                      22     14    36%
brain/cache/model.py                                                                     34     24    29%
brain/cache/query.py                                                                     87     77    11%
brain/cache/session.py                                                                   51     51     0%
brain/cache/settings.py                                                                  14     10    29%
brain/converter/__init__.py                                                               0      0   100%
brain/converter/crypto.py                                                                45      6    87%
brain/converter/format/__init__.py                                                        0      0   100%
brain/converter/format/csv2dict.py                                                       21     21     0%
brain/converter/format/xml2dict.py                                                       19     19     0%
brain/converter/md5.py                                                                    9      9     0%
brain/converter/model.py                                                                 12      7    42%
brain/converter/operation.py                                                             45     45     0%
brain/converter/settings.py                                                              50     50     0%
brain/database/__init__.py                                                                0      0   100%
brain/database/account.py                                                                55     16    71%
brain/database/dataset.py                                                                16      6    63%
brain/database/entity.py                                                                 60     29    52%
brain/database/model_type.py                                                             17      7    59%
brain/database/prediction.py                                                             89     19    79%
brain/database/query.py                                                                 125     59    53%
brain/database/session.py                                                                30     20    33%
brain/database/settings.py                                                               46     24    48%
brain/load_data.py                                                                       75     30    60%
brain/session/__init__.py                                                                 0      0   100%
brain/session/base.py                                                                    33     16    52%
brain/session/base_data.py                                                               59     27    54%
brain/session/data/__init__.py                                                            0      0   100%
brain/session/data/dataset.py                                                            61     37    39%
brain/session/data_append.py                                                             29     12    59%
brain/session/data_new.py                                                                27      9    67%
brain/session/model/__init__.py                                                           0      0   100%
brain/session/model/sv.py                                                                44      8    82%
brain/session/model_generate.py                                                          26     11    58%
brain/session/model_predict.py                                                           14      7    50%
brain/session/predict/__init__.py                                                         0      0   100%
brain/session/predict/sv.py                                                              22      5    77%
brain/validator/__init__.py                                                               0      0   100%
brain/validator/dataset.py                                                               40     25    38%
brain/validator/email.py                                                                  7      3    57%
brain/validator/password.py                                                              43      4    91%
brain/validator/settings.py                                                              28     14    50%
factory.py                                                                               57     19    67%
interface/__init__.py                                                                     0      0   100%
interface/views_api.py                                                                  199     94    53%
interface/views_web.py                                                                  165    165     0%
log/__init__.py                                                                           0      0   100%
log/logger.py                                                                            84     84     0%
test/live_server/1_authentication/pytest_1_crypto.py                                     15      2    87%
test/live_server/1_authentication/pytest_2_account_registration.py                       21      3    86%
test/live_server/1_authentication/pytest_3_validate_password.py                          15      2    87%
test/live_server/1_authentication/pytest_4_user_login.py                                 20      3    85%
test/live_server/1_authentication/pytest_5_user_logout.py                                16      1    94%
test/live_server/2_database/pytest_svm_dbmax.py                                         110      0   100%
test/live_server/2_database/pytest_svr_dbmax.py                                         110      0   100%
test/live_server/3_programmatic_interface/dataset_url/pytest_svm_dataset_url.py          58      0   100%
test/live_server/3_programmatic_interface/dataset_url/pytest_svr_dataset_url.py          48      0   100%
test/live_server/3_programmatic_interface/file_upload/pytest_svm_file_upload.py          58      0   100%
test/live_server/3_programmatic_interface/file_upload/pytest_svr_file_upload.py          48      0   100%
test/live_server/3_programmatic_interface/results/pytest_1_svm_prediction.py             68     17    75%
test/live_server/3_programmatic_interface/results/pytest_2_svr_prediction.py             67     17    75%
test/live_server/3_programmatic_interface/results/pytest_3_all_prediction_titles.py      36      7    81%
test/live_server/conftest.py                                                             19      2    89%
---------------------------------------------------------------------------------------------------------
TOTAL                                                                                  2576   1144    56%

=============================== warnings summary ===============================
3_programmatic_interface/results/pytest_1_svm_prediction.py::test_save_prediction
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'probability' at row 1
    self.cursor.execute(statement, sql_args)
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'decision_function' at row 1
    self.cursor.execute(statement, sql_args)

3_programmatic_interface/results/pytest_2_svr_prediction.py::test_save_prediction
  /var/machine-learning/brain/database/query.py:281: Warning: Data truncated for column 'r2' at row 1
    self.cursor.execute(statement, sql_args)

-- Docs: http://doc.pytest.org/en/latest/warnings.html
=================== 48 passed, 3 warnings in 358.24 seconds ====================
80de1ba426c5
3cba03a3f7df
37b0c6256c11
9d15b8864c0a
80de1ba426c5
3cba03a3f7df
37b0c6256c11
Deleted: sha256:ff8728ff9d5ba90a7fe3e0b439218e5dba6726fd4116a659b6278881a45fa71d
Deleted: sha256:10ef00c17387b4cfb31a13f3d057be0bd21ebf4f71f26987b6cf99026b3ea3b5
Deleted: sha256:51cac640ee1ff8299ff44b12b194d715e3a8870f28986b480a81177b037486a4
Deleted: sha256:6bbd67088bdf473d7f042407c63aec5c435d946796a90a480f96a9633d9dbbe7
Deleted: sha256:d908e9c67996a4c7994982dabc4bcf5ddc72a26a59677e7f7b87ca0612bd5d8b
Deleted: sha256:977d83b5b8bd241221b5bb98f5baffef6d1bec1ac4543386308e3166ec45cbfb
Deleted: sha256:0141c45efd509047bf6235daeefd0c0edbd1724ddb718b073881884b1c147182
b64bea46cff1
Error response from daemon: bridge is a pre-defined network and cannot be removed
Error response from daemon: host is a pre-defined network and cannot be removed
Error response from daemon: none is a pre-defined network and cannot be removed
root@trusty64:/vagrant#

Note: the corresponding travis ci build can also be reviewed. However, we'll need to make adjustments, to account for our linter results. Additionally, we'll need to push our corresponding webserver container to dockerhub, and verify whether are results successful, within a local rancher instance.

It's very encouraging to see that our verbose travis ci tests, returned identical results, to the above local verbose tests. Additionally, we've reduced the entire build-test cycle time, from our previous 21+ minutes, to just under 11 minutes.

In order for our current reverse_proxy puppet module to execute, we need to have the container's hostname defined during build time, or during the RUN directive, implementing the puppet apply. Having the hostname defined, would allow the puppet module to implement the hiera definition, contained within the copied nginx-xxx.yaml configuration. Otherwise, we'll need to refactor our nginx.dockerfile into nginx-web.dockerfile, and nginx-api.dockerfile. This would mean the corresponding COPY statements would need to be refactored, to copy only the required nginx-xxx.yaml, into the nginx-xxx container. Then, the hiera.yaml would need to become more verbose (not preferred), or we could dynamically copy the needed yaml configuration, as a fixed nginx.yaml, in each nginx-xxx container, while changing path: "hiera/webserver/%{::hostname}.yaml" to path: "hiera/webserver/nginx.yaml", in our hiera.yaml.

Trying to append to the /etc/hosts is not successful:

Step 16/16 : RUN echo $(head -1 /etc/hosts | cut -f1) 'nginx-web' >> /etc/hosts && echo "hostname: $(hostname)"
 ---> Running in 8519bccb1d13
hostname: 8519bccb1d13

Additionally, running the hostname command directly fails:

Step 16/16 : RUN echo $(head -1 /etc/hosts | cut -f1) nginx-web >> /etc/hosts && echo "hostname: $(hostname)" && hostname nginx-web && echo "hostname2: $(hostname)"
 ---> Running in 8873d524abba
hostname: you must be root to change the host name

So, we attempted to force docker to use the root user (should be the default behavior):

Step 16/17 : USER root
 ---> Running in 98460e35d31b
Removing intermediate container 98460e35d31b
 ---> 6356bba11f3e
Step 17/17 : RUN echo $(head -1 /etc/hosts | cut -f1) 'nginx-web' >> /etc/hosts && echo "hostname: $(hostname)" && hostname 'nginx-web' && echo "hostname2: $(hostname)"
 ---> Running in 212d6261fb7c
hostname: you must be root to change the host name

Note: another option we have, in addition to those discussed above, is to possibly run the puppet provisioning during an ENTRYPOINT. However, this seems a bit of a cumbersome step, since building should be during build run phase.

We are able to confirm that our current configurations, within our nginx-web container, could access the webserver-web container on port 5001:

port-open

Therefore, we need to verify if our nginx configurations, specifically the upstream is properly defined.

Our nginx-webserver seems to behave as expected, since we no longer experiencing an nginx 405 error, in the browser . However, it may be likely that our compiled content.js, is not properly implementing our anticipated nodejs presets, or modules, during the attempt to transform, and compile .jsx to .js:

nginx

jeff1evesque / machine-learning

Replace puppet 'docker' with containers #2935