Modify VagrantProvision script to support Jenkins

mattjdnv commented 8 years ago

As the title says.

mattjdnv commented 8 years ago

@brianhatchl Initial tweaks have been pushed into this branch.

brianhatchl commented 8 years ago

You know what I'd like to see? Removing the steps reverting postgres to 9.1 and postgis 1.5 and just go with the latest pg version available for unbuntu 14.

mattjdnv commented 8 years ago

I agree but the problem will be supporting Centos6.7 and Centos7. If they have pg9.5 then we are fine.

mattjdnv commented 8 years ago

Looks like pg9.5 exists for Centos6: https://download.postgresql.org/pub/repos/yum/9.5/redhat/rhel-6-x86_64/pgdg-centos95-9.5-2.noarch.rpm

brianhatchl commented 8 years ago

But this provision file won't be used for both ubuntu and centos will it? I think we'll need separate provision scripts and possibly define two different vms in the Vagrantfile, one ubuntu and one centos.

mattjdnv commented 8 years ago

I didn't think about that. I built my own Vagrantfile for Centos.

I agree we need two. Sigh.....

brianhatchl commented 8 years ago

You can define two separate vms in the Vagrantfile with separate provision scripts. Then the jenkins jobs will do vagrant up ubuntu14 vs. vagrant up centos67

Something like this

  config.vm.define "ubuntu14" do |ubuntu|
    ubuntu.vm.box = "ubuntu/trusty64"
    ubuntu.vm.box_url = "https://atlas.hashicorp.com/ubuntu/boxes/trusty64"
    ubuntu.vm.provision :shell, :path => "VagrantProvisionUbuntu14.sh"
  end

 config.vm.define "centos67" do |centos|
    centos.vm.box = "bento/centos-6.7"
    centos.vm.box_url = "https://atlas.hashicorp.com/bento/boxes/centos-6.7"
    centos.vm.provision :shell, :path => "VagrantProvisionCentos67.sh"
  end

mattjdnv commented 8 years ago

That is what I was planning to do to create the LoadTest VM. It will work for this as well.

I can move the stuff I have in the Centos Jenkins jobs into a "VagrantProvisionCentos67.sh" file

The only funkyness will be adding the pre-provisioning steps to install and configure NFS.

brianhatchl commented 8 years ago

This stuff is all going to live in the main hoot github repo right? And then just get the VagrantfileLocal for the jenkins job clones?

mattjdnv commented 8 years ago

That is the plan.... I just need to figure out how to merge the GitHub Vagrantfile with the Jenkins Vagrantfile & VagrantfileLocal so that we get:

Install NFS
Configure NFS
Run VagrantProvisionXXXX.sh

brianhatchl commented 8 years ago

I think you can have those first two steps invoked only for the vsphere provider (if that's what we want).

mattjdnv commented 8 years ago

VM appears to be working in Jenkins. Testing with a local copy of VirtualBox before asking for a merge.

The next issue is port forwarding from the VM.

bwitham commented 8 years ago

now messing around in this branch for #492

mattjdnv commented 8 years ago

After a fair amount of fiddling with the existing VM, I gave up and am starting with a clean Vagrant build on Selma.

Hopefully these errors will go away:

==> hoot: Deploying web application files...
==> hoot: rm: 
==> hoot: cannot remove ‘/usr/share/tomcat6/webapps/hoot-services/WEB-INF/lib/deegree-core-commons-3.3.6.jar’
==> hoot: : Permission denied
==> hoot: rm: 
==> hoot: cannot remove ‘/usr/share/tomcat6/webapps/hoot-services/WEB-INF/lib/commons-compress-1.0.jar’
==> hoot: : Permission denied
==> hoot: rm: 

etc

mattjdnv commented 8 years ago

If tonight's Jenkins nightly runs cleanly, I'm tempted to call this one done.

bwitham commented 8 years ago

If its running all the tests on vagrant, including hadoop and UI, then its done.

mattjdnv commented 8 years ago

It's close:

I think I can hide this error by piping the output to null and running it a couple of times. I think it is due to bindfs & nfs. When I ssh into the VM and run the "CopyWebAppsToTomcat.sh" script I get errors and then I don't, and then I do, and then I don't etc.

==> hoot: Deploying web application files...
==> hoot: rm: 
==> hoot: cannot remove ‘/var/lib/tomcat6/webapps/hoot-services/WEB-INF/lib/deegree-core-commons-3.3.6.jar’
==> hoot: : Permission denied
==> hoot: rm: 
==> hoot: cannot remove ‘/var/lib/tomcat6/webapps/hoot-services/WEB-INF/lib/commons-compress-1.0.jar’
==> hoot: : Permission denied
==> hoot: rm: 
==> hoot: cannot remove ‘/var/lib/tomcat6/webapps/hoot-services/WEB-INF/lib/stax2-api-3.1.1.jar’
==> hoot: : Permission denied
==> hoot: rm: 
etc

This error is related to the 'rm' one. If the rm fails, the web app doesn't get copied to Tomcat and the UI tests fail.

Checking Tomcat server for Hootenanny availability...
Tomcat not started or Hootenanny web application not deployed to http://localhost:8080.
make[4]: *** [test] Error 255
make[3]: *** [ui-test] Error 2
make[2]: *** [ui-test] Error 2
make[1]: *** [test-all-no-core] Error 2
make: *** [test-all] Error 2

Hoot-tests failure: This should be related to the previous errors.

# only report errors
mocha -R `pwd`/QuietReporter.js
fail: etl conflate should load and conflate Gaalkacyo roads -- error: Error: failed ingesting files
fail: etl load should load a couple of UTP shapefiles -- error: Error: failed ingesting files
fail: etl load should load a couple of UTP Shapefiles in a .zip -- error: Error: failed ingesting files
fail: etl load should load a MAAX .zip FGDB -- error: Error: failed ingesting files
end: 0/4
make[1]: Leaving directory `/home/vagrant/hoot/hoot-tests/services-system.child'
make[1]: *** [test] Error 4
make: *** [services-system.test] Error 2

bwitham commented 8 years ago

What permissions do those files have on nfs?

mattjdnv commented 8 years ago

I found that I didn't have the "sudo" in the command (sudo -u tomcat6 scripts/CopyWebAppsToTomcat.sh) but it didn't make a difference, I still got the reams of errors.

vagrant@vagrantubuntu:~/hoot$ ls -l /var/lib/tomcat6/
total 16
drwxr-xr-x 3 vagrant tomcat6 4096 Apr  6 16:23 common
lrwxrwxrwx 1 vagrant tomcat6   12 Feb 24  2014 conf -> /etc/tomcat6
lrwxrwxrwx 1 vagrant tomcat6   16 Apr 13 18:04 log -> /var/log/tomcat6
lrwxrwxrwx 1 vagrant tomcat6   17 Feb 24  2014 logs -> ../../log/tomcat6
drwxr-xr-x 3 vagrant tomcat6 4096 Apr  6 16:23 server
drwxr-xr-x 3 vagrant tomcat6 4096 Apr  6 16:23 shared
drwxrwxr-x 5 vagrant tomcat6 4096 Apr 14 11:24 webapps
lrwxrwxrwx 1 vagrant tomcat6   19 Feb 24  2014 work -> ../../cache/tomcat6
vagrant@vagrantubuntu:~/hoot$ ls -l /var/lib/tomcat6/webapps/
total 44292
drwxr-xr-x 10 vagrant vagrant     4096 Apr 13 18:16 hootenanny-id
drwxrwxr-x  4 tomcat6 tomcat6     4096 Apr 14 11:24 hoot-services
-rw-r--r--  1 tomcat6 tomcat6 45339322 Apr 14 11:24 hoot-services.war
drwxr-xr-x  3 vagrant tomcat6     4096 Apr  6 16:25 ROOT
lrwxrwxrwx  1 vagrant tomcat6       24 Apr 13 18:04 webapps -> /var/lib/tomcat6/webapps

mattjdnv commented 8 years ago

Weird. I just trashed the whole webapps directory and re-ran the copy script and it works...

mattjdnv commented 8 years ago

Getting closer. The "hoot-tests/services-system.child" is the only thing failing.

  1) etl conflate should load and conflate GK roads:
     Uncaught 
  AssertionError: Error: failed ingesting files
      at /home/vagrant/hoot/hoot-tests/services-system.child/ConflateTest.js:65:17
      at Request.<anonymous> (/home/vagrant/hoot/hoot-tests/services-system.child/TestUtils.js:315:21)
      at Request.EventEmitter.emit (events.js:98:17)
      at Request.mixin._fireSuccess (/home/vagrant/hoot/hoot-tests/services-system.child/node_modules/restler/lib/restler.js:223:12)
      at /home/vagrant/hoot/hoot-tests/services-system.child/node_modules/restler/lib/restler.js:161:20
      at IncomingMessage.parsers.auto (/home/vagrant/hoot/hoot-tests/services-system.child/node_modules/restler/lib/restler.js:402:7)
      at Request.mixin._encode (/home/vagrant/hoot/hoot-tests/services-system.child/node_modules/restler/lib/restler.js:198:29)
      at /home/vagrant/hoot/hoot-tests/services-system.child/node_modules/restler/lib/restler.js:157:16
      at Request.mixin._decode (/home/vagrant/hoot/hoot-tests/services-system.child/node_modules/restler/lib/restler.js:173:7)
      at IncomingMessage.<anonymous> (/home/vagrant/hoot/hoot-tests/services-system.child/node_modules/restler/lib/restler.js:150:14)
      at IncomingMessage.EventEmitter.emit (events.js:117:20)
      at _stream_readable.js:920:16
      at process._tickCallback (node.js:415:13)

bwitham commented 8 years ago

Change ~line 304 of TestUtils.js to look like:

        if (err) 
        {
          console.log("error: " + error);
          console.log("stderr: " + stderr);           
          done(err);
        }

Should give you a better error to work off of.

mattjdnv commented 8 years ago

No change to the error message.

bwitham commented 8 years ago

Tomcat's running?

mattjdnv commented 8 years ago

Update: In vagrant,

Make test-all runs cleanly
There is one error with hoot-tests:

In hoot-tests/services-system.child:

3 passing (13m)
1 failing
1) etl load should load a MAAX .zip FGDB:
Error: timeout of 600000ms exceeded
at null.<anonymous> (/usr/local/lib/node_modules/mocha/lib/runnable.js:139:19)
at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)

I think it is ready for review/merging. @bwitham , @brianhatchl , @jasonsurratt want to give it a go?

bwitham commented 8 years ago

I'll do it. Given this monumental task, that particular test failing isn't that big of deal. I'll try to fix it while reviewing.

bwitham commented 8 years ago

merged

mattjdnv commented 8 years ago

Happy Dance Time!

bwitham commented 8 years ago

mattjdnv commented 8 years ago

@bwitham As usual, you rock my world.

drew-bower commented 8 years ago

Freddie Mercury?

bwitham commented 8 years ago

matt is the champion.

ngageoint / hootenanny

Modify VagrantProvision script to support Jenkins #476