mbohun commented 6 years ago

CRUCIAL FIXES

[x] required linux packages
- jq is used by some of the jenkins jobs
- python (some distro-s like the ubuntu on aws do not have python installed)
- ntp or at least static TIMEZONE (to ensure the env, logs, etc. show ACT time)
- zip
- xmllint (for XML/XPath parsing; to install on ubuntu: sudo apt install libxml2-utils)
[x] after the installation the main page has broken CSS
- FIX: this is fixed by adjusting/editing /data/ag-bie/config/ag-bie-config.properties, hence this has to be done in the ansible template for ag-bie-config.properties (or through the inventory) to become permanent and to be done at the installation time
[x] create/setup private git / github repo for storing ansible inventory files (passwords, usernames, SSL certificates, etc.)
- install/setup "private"/"inhouse" git repo: https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server
- ideally with cgit OR gitlab frontend
[x] fix/fully automate nginx setup/configuration (this is to setup/configure nginx <---> tomcat)
[x] fix/fully automate SSL certificates setupjenkins jobs
[x] fix ansible bug ignoring webapps' context_path settings installing everything into /ROOT
- NOTE: the problem could have been introduced in ala-install AFTER ag-bie/ansible was used last time successfully
- FIX: this is an ala-install BUG; and it can be avoided/mitigated if you re-order your playbook to ensure the non-root apps (userdetails, cas, apikey, etc.) being installed BEFORE the root-app (bie_hub / ag-bie-0.3.war in our case)
[x] automate the installation of /data/taxxas (required by the daily jenkins/cron jobs) VERIFY
- Double-check with @charvolant where are we going to develop/store the java/grails sources/artifacts ?

[x] fix/verify all scripted SSH operations to use -oStrictHostKeyChecking=no to avoid:

    The authenticity of host 'sit-ag-bie.oztaxa.com (54.79.30.70)' can't be established.
    ECDSA key fingerprint is SHA256:xxsyyAakTItwgnNCljc9U6Z67hj7b9bLogRHTkluAHw.
    Are you sure you want to continue connecting (yes/no)?

[x] move jenkins jobs/templates to use Jenkinsfile (jenkins pipeline)
- PROBLEM / LIMITATION / REFACTORING: at the moment the jenkins setup REQUIRES manual/human interaction (see bellow for details)
- separate the BASH scripts from jenkins jobs (jenkins jobs' GUI configuration), so the BASH scripts can be run:
  - a) from CLI (manually and/or like a UNIX cronjob) without any dependencies on jenkins
  - b) from the jenkins GUI
[x] trigger the BASH scripts automatically at the end of the ag-bie env setup to generate data for solr/bie-index

ENHANCEMENTS / ADDITIONS

add/provide simple diagnostic / verification of the diff installed components (BASH/curl/jq test)
- starting with simple/trivial check-s / verification like:
```
curl -s 'https://uat-ag-bie.oztaxa.com/ws/search?q=Drosophila&rows=100' | jq '.searchResults.totalRecords'
=> 16
```
- ideally we should have a set of simple/minimalist curl check-s for each installed component
unify/sync nginx, tomcat (log4j) log message configuration/pattern
- use consistent date/time-stamp format allowing for easy processing/searching with grep, sed, etc.
remove all unused ALA components (for example: occurrences) CLEANUP
provide simple README/wiki page / cookbook demonstrating step-by-step installation of ag-bie environment

REFERENCES

Jenkins remote/REST API documentation

example: to trigger a jenkins job with curl:

curl -X POST \
    -u admin:${JENKINS_TOKEN} \
    http://dev-ag-bie.oztaxa.com:9192/job/test_ag-bie-solr_Jenkinsfile/build

TODO

mbohun commented 6 years ago

This is the expected tomcat layout:

ubuntu@ip-172-31-12-151:~$ find /var/lib/tomcat7/webapps* -maxdepth 1 -type d
/var/lib/tomcat7/webapps
/var/lib/tomcat7/webapps/ROOT
/var/lib/tomcat7/webapps/solr
/var/lib/tomcat7/webapps-uat-ag-bie.oztaxa.com
/var/lib/tomcat7/webapps-uat-ag-bie.oztaxa.com/ROOT
/var/lib/tomcat7/webapps-uat-ag-bie.oztaxa.com/userdetails
/var/lib/tomcat7/webapps-uat-ag-bie.oztaxa.com/apikey
/var/lib/tomcat7/webapps-uat-ag-bie.oztaxa.com/ws
/var/lib/tomcat7/webapps-uat-ag-bie.oztaxa.com/cas

...and this is the actual BUGGY layout:

ubuntu@ip-172-31-16-183:~$ find /var/lib/tomcat7/webapps* -maxdepth 1 -type d
/var/lib/tomcat7/webapps
/var/lib/tomcat7/webapps/ROOT
/var/lib/tomcat7/webapps-dev-ag-bie.oztaxa.com
/var/lib/tomcat7/webapps-dev-ag-bie.oztaxa.com/ROOT

caused by the ansible installer installing (overwriting one-by-one) all the diff wars into /var/lib/tomcat7/webapps-dev-ag-bie.oztaxa.com/ROOT

mbohun commented 6 years ago

SSL certificates not installed/setup properly:

Failed calling web service. class javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target URL= https://uat-ag-bie.oztaxa.com/ws/search?q=drosophila&start=0&rows=10&sort=&dir=desc&facets=idxtype,rank,speciesGroup,taxonomicStatus&q.op=OR.
2018-05-10 12:19:52.393 ERROR --- [io-8080-exec-14] g.a.c.au.org.ala.bie.SpeciesController   : Error requesting taxon concept object: Failed calling web service. class javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target URL= https://uat-ag-bie.oztaxa.com/ws/search?q=drosophila&start=0&rows=10&sort=&dir=desc&facets=idxtype,rank,speciesGroup,taxonomicStatus&q.op=OR.

The SSL certificates are required even "internally" for example by Import_Taxonomy.sh

jenkins@ip-172-31-23-2:/tmp/ag-bie-solr.git$ curl -L  -X GET -f --header 'Host: sit-ag-bie.oztaxa.com' http://localhost:8080/ws/admin/services/all
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

mbohun commented 6 years ago

This is the expected /etc/nginx/sites-available/dev-ag-bie.oztaxa.com.conf format:

#Ansible managed
server {
    listen 80;
    listen [::]:80;
    server_name uat-ag-bie.oztaxa.com;
    access_log  /var/log/nginx/access.log;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443;
    listen [::]:443;
    server_name uat-ag-bie.oztaxa.com;
    ssl on;
    ssl_certificate /home/ubuntu/certificate.pem;
    ssl_certificate_key /home/ubuntu/key.pem;
    # ssl_dhparam /etc/sslmate/dhparam.pem;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;
    root /srv/uat-ag-bie.oztaxa.com/www/;
    index index.html;
    client_max_body_size 20m;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_read_timeout 10m;
        proxy_pass http://127.0.0.1:8080/;
    }
    location /trove {
        rewrite ^/trove/(.*) /$1 break;
        proxy_pass http://api.trove.nla.gov.au;
    }
}

NOTE: this file above is from https://uat-ag-bie.oztaxa.com/ and it was manually edited/adjusted after the ansible isntallation.

...and this is the actual BUGGY format after installation:

#Ansible managed
server {
    listen 80;
    listen [::]:80;
    server_name dev-ag-bie.oztaxa.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name dev-ag-bie.oztaxa.com;
    ssl on;
    ssl_certificate /etc/nginx/ssl/dev-ag-bie.oztaxa.com.self.crt;
    ssl_certificate_key /etc/nginx/ssl/dev-ag-bie.oztaxa.com.self.key;
    ssl_protocols TLSv1.2;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;

    # Force HTTPS was configured, so setting STS value to 1 year (31536000 seconds)
    add_header Strict-Transport-Security "max-age=31536000" always;

    # Secure referrer policy to avoid leaking paths, which can include auth ticket information
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    root /srv/dev-ag-bie.oztaxa.com/www/;
    index index.html;
    client_max_body_size 20m;

    # If ssl is supported, then push this policy to all users who understand it to trigger HSTS
    add_header Content-Security-Policy upgrade-insecure-requests;
    location = /robots.txt {
        add_header  Content-Type  text/plain;
        return 200 
"
# Disallow particular paths for all user agents
User-agent: *
Disallow: 
Crawl-delay: 30

";

}
    location / {
        # proxy_set_header Host $host;
        proxy_set_header Host dev-ag-bie.oztaxa.com;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_read_timeout 10m;
        proxy_pass http://127.0.0.1:8080/;
    }
    location /apikey {
        # proxy_set_header Host $host;
        proxy_set_header Host dev-ag-bie.oztaxa.com;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_read_timeout 10m;
        proxy_pass http://127.0.0.1:8080/apikey;
    }
    location /cas {
        # proxy_set_header Host $host;
        proxy_set_header Host dev-ag-bie.oztaxa.com;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_read_timeout 10m;
        proxy_pass http://127.0.0.1:8080/cas;
    }
    location /trove {
        rewrite ^/trove/?(.*) /$1 break;
        proxy_pass http://api.trove.nla.gov.au;
    }
    location /userdetails {
        # proxy_set_header Host $host;
        proxy_set_header Host dev-ag-bie.oztaxa.com;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_read_timeout 10m;
        proxy_pass http://127.0.0.1:8080/userdetails;
    }
    location /ws {
        # proxy_set_header Host $host;
        proxy_set_header Host dev-ag-bie.oztaxa.com;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_read_timeout 10m;
        proxy_pass http://127.0.0.1:8080/ws;
    }
}

mbohun commented 6 years ago

THIS NEEDS DOUBLE_CHECKING

(maybe not required)

right after the ansible installation the solr is NOT working/installed properly, this is how to fix it:

ubuntu@ip-172-31-12-136:~$ du -hs /var/lib/tomcat7/webapps/ROOT
16K     /var/lib/tomcat7/webapps/ROOT
ubuntu@ip-172-31-12-136:~$ du -hs /var/lib/tomcat7/webapps/solr
33M     /var/lib/tomcat7/webapps/solr
ubuntu@ip-172-31-12-136:~$ sudo systemctl stop tomcat7
ubuntu@ip-172-31-12-136:~$ cd /var/lib/tomcat7/webapps/
ubuntu@ip-172-31-12-136:/var/lib/tomcat7/webapps$ ls -lahF
total 29M
drwxrwxr-x 4 tomcat7 tomcat7 4.0K May 25 11:22 ./
drwxr-xr-x 7 root    root    4.0K May 25 11:16 ../
drwxr-xr-x 3 root    root    4.0K May 25 11:13 ROOT/
drwxr-xr-x 8 tomcat7 tomcat7 4.0K May 25 11:22 solr/
-rw-r--r-- 1 root    root     29M May 25 11:21 solr.war
ubuntu@ip-172-31-12-136:/var/lib/tomcat7/webapps$ sudo cp solr.war ROOT.war
ubuntu@ip-172-31-12-136:/var/lib/tomcat7/webapps$ ls -lahF
total 57M
drwxrwxr-x 4 tomcat7 tomcat7 4.0K May 25 13:10 ./
drwxr-xr-x 7 root    root    4.0K May 25 11:16 ../
drwxr-xr-x 3 root    root    4.0K May 25 11:13 ROOT/
-rw-r--r-- 1 root    root     29M May 25 13:10 ROOT.war
drwxr-xr-x 8 tomcat7 tomcat7 4.0K May 25 11:22 solr/
-rw-r--r-- 1 root    root     29M May 25 11:21 solr.war
ubuntu@ip-172-31-12-136:/var/lib/tomcat7/webapps$ sudo systemctl start tomcat7

mbohun commented 6 years ago

REFACTORING

PROBLEM:

At the moment installed/configured jenkins is required to create/populate the SOLR data
However the jenkins ansible installation/configuration is NOT fully automated (after the ag-bie install finishes, the jenkins setup has to be completed manually)
Further: part of the functionality is currently "hard-coded" inside the jenkins jobs' GUI, in other words: it is not possible to run/execute the scripts without/outside jenkins

SOLUTION:

Extract/separate the jenkins jobs scripts into standalone/normal shell scripts jenkins jobs will simply call; This way the same script can be run a) separately from cli (without any dependencies on jenkins) AND b) launched from jenkins
Migrate all jenkins jobs to Jenkinsfile (Jenkins pipeline)

mbohun commented 6 years ago

ansible Jenkins jobs re-install verification

Today (2018-05-29) while re-installing https://dev-ag-bie.oztaxa.com with:

ansible-playbook -i /tmp/dev-ag-bie.oztaxa.com ag-bie.yml -u ubuntu -b --skip-tags=version_check

the installer/script failed with the following error:

 TASK [ag-bie-jenkins-config : add jenkins jobs to jenkins] *******************************************************************************************************************************************
failed: [dev-ag-bie.oztaxa.com] (item=Combine Taxonomies) => {"changed": true, "cmd": "./add-job.sh \"Combine Taxonomies\"", "delta": "0:00:00.609668", "end": "2018-05-29 14:34:19.770800", "item": "Combine Taxonomies", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:19.161132", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Import Taxonomy) => {"changed": true, "cmd": "./add-job.sh \"Import Taxonomy\"", "delta": "0:00:00.383706", "end": "2018-05-29 14:34:21.645245", "item": "Import Taxonomy", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:21.261539", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Load from SFTP) => {"changed": true, "cmd": "./add-job.sh \"Load from SFTP\"", "delta": "0:00:00.373527", "end": "2018-05-29 14:34:23.535805", "item": "Load from SFTP", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:23.162278", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process NSL Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process NSL Source Data\"", "delta": "0:00:00.369838", "end": "2018-05-29 14:34:25.283270", "item": "Process NSL Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:24.913432", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process TaxxaS Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process TaxxaS Source Data\"", "delta": "0:00:00.371659", "end": "2018-05-29 14:34:27.055252", "item": "Process TaxxaS Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:26.683593", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "sTASK [ag-bie-jenkins-config : add jenkins jobs to jenkins] *******************************************************************************************************************************************
failed: [dev-ag-bie.oztaxa.com] (item=Combine Taxonomies) => {"changed": true, "cmd": "./add-job.sh \"Combine Taxonomies\"", "delta": "0:00:00.609668", "end": "2018-05-29 14:34:19.770800", "item": "Combine Taxonomies", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:19.161132", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Import Taxonomy) => {"changed": true, "cmd": "./add-job.sh \"Import Taxonomy\"", "delta": "0:00:00.383706", "end": "2018-05-29 14:34:21.645245", "item": "Import Taxonomy", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:21.261539", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Load from SFTP) => {"changed": true, "cmd": "./add-job.sh \"Load from SFTP\"", "delta": "0:00:00.373527", "end": "2018-05-29 14:34:23.535805", "item": "Load from SFTP", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:23.162278", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HtTASK [ag-bie-jenkins-config : add jenkins jobs to jenkins] *******************************************************************************************************************************************
failed: [dev-ag-bie.oztaxa.com] (item=Combine Taxonomies) => {"changed": true, "cmd": "./add-job.sh \"Combine Taxonomies\"", "delta": "0:00:00.609668", "end": "2018-05-29 14:34:19.770800", "item": "Combine Taxonomies", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:19.161132", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Import Taxonomy) => {"changed": true, "cmd": "./add-job.sh \"Import Taxonomy\"", "delta": "0:00:00.383706", "end": "2018-05-29 14:34:21.645245", "item": "Import Taxonomy", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:21.261539", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Load from SFTP) => {"changed": true, "cmd": "./add-job.sh \"Load from SFTP\"", "delta": "0:00:00.373527", "end": "2018-05-29 14:34:23.535805", "item": "Load from SFTP", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:23.162278", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process NSL Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process NSL Source Data\"", "delta": "0:00:00.369838", "end": "2018-05-29 14:34:25.283270", "item": "Process NSL Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:24.913432", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process TaxxaS Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process TaxxaS Source Data\"", "delta": "0:00:00.371659", "end": "2018-05-29 14:34:27.055252", "item": "Process TaxxaS Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:26.683593", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat TASK [ag-bie-jenkins-config : add jenkins jobs to jenkins] *******************************************************************************************************************************************
failed: [dev-ag-bie.oztaxa.com] (item=Combine Taxonomies) => {"changed": true, "cmd": "./add-job.sh \"Combine Taxonomies\"", "delta": "0:00:00.609668", "end": "2018-05-29 14:34:19.770800", "item": "Combine Taxonomies", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:19.161132", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Import Taxonomy) => {"changed": true, "cmd": "./add-job.sh \"Import Taxonomy\"", "delta": "0:00:00.383706", "end": "2018-05-29 14:34:21.645245", "item": "Import Taxonomy", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:21.261539", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Load from SFTP) => {"changed": true, "cmd": "./add-job.sh \"Load from SFTP\"", "delta": "0:00:00.373527", "end": "2018-05-29 14:34:23.535805", "item": "Load from SFTP", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:23.162278", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process NSL Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process NSL Source Data\"", "delta": "0:00:00.369838", "end": "2018-05-29 14:34:25.283270", "item": "Process NSL Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:24.913432", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process TaxxaS Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process TaxxaS Source Data\"", "delta": "0:00:00.371659", "end": "2018-05-29 14:34:27.055252", "item": "Process TaxxaS Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:26.683593", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Swap Cores) => {"changed": true, "cmd": "./add-job.sh \"Swap Cores\"", "delta": "0:00:00.368916", "end": "2018-05-29 14:34:28.733633", "item": "Swap Cores", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:28.364717", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Swap Cores) => {"changed": true, "cmd": "./add-job.sh \"Swap Cores\"", "delta": "0:00:00.368916", "end": "2018-05-29 14:34:28.733633", "item": "Swap Cores", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:28.364717", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}tpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process NSL Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process NSL Source Data\"", "delta": "0:00:00.369838", "end": "2018-05-29 14:34:25.283270", "item": "Process NSL Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:24.913432", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process TaxxaS Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process TaxxaS Source Data\"", "delta": "0:00:00.371659", "end": "2018-05-29 14:34:27.055252", "item": "Process TaxxaS Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:26.683593", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Swap Cores) => {"changed": true, "cmd": "./add-job.sh \"Swap Cores\"", "delta": "0:00:00.368916", "end": "2018-05-29 14:34:28.733633", "item": "Swap Cores", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:28.364717", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}tdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Swap Cores) => {"changed": true, "cmd": "./add-job.sh \"Swap Cores\"", "delta": "0:00:00.368916", "end": "2018-05-29 14:34:28.733633", "item": "Swap Cores", "msg": "non-zero return code", "rc": 1, "start": "2018-05-29 14:34:28.364717", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}

Given the installer completed without errors on Friday (previous install session/run) it is assumed the installer works fine with a pristine ubuntu env, but fails if it is run against a previously installed ag-bie / jenkins jobs environment.

this is a sporadic error (we haven't seen in 20 days)

TASK [ag-bie-jenkins-config : add jenkins jobs to jenkins] ******************************************************************************************************************
failed: [dev-ag-bie.oztaxa.com] (item=Combine Taxonomies) => {"changed": true, "cmd": "./add-job.sh \"Combine Taxonomies\"", "delta": "0:00:00.603356", "end": "2018-06-18 15
:02:51.327179", "item": "Combine Taxonomies", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:02:50.723823", "stderr": "java.io.IOException: Server returned 
HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\ta
t sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat h
udson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Ser
ver returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.
java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStrea
m.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't
 get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Import Taxonomy) => {"changed": true, "cmd": "./add-job.sh \"Import Taxonomy\"", "delta": "0:00:00.374921", "end": "2018-06-18 15:02:53
.678446", "item": "Import Taxonomy", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:02:53.303525", "stderr": "java.io.IOException: Server returned HTTP resp
onse code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net
.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli
.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server retur
ned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894
)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:10
1)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job 
list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Load from SFTP) => {"changed": true, "cmd": "./add-job.sh \"Load from SFTP\"", "delta": "0:00:00.374995", "end": "2018-06-18 15:02:56.6
82699", "item": "Load from SFTP", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:02:56.307704", "stderr": "java.io.IOException: Server returned HTTP respons
e code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.ww
w.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CL
I.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned
 HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)",
 "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)"
, "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job lis
t", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process NSL Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process NSL Source Data\"", "delta": "0:00:00.372316", "end": "2018-06-18 15:02:58.731217", "item": "Process NSL Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:02:58.358901", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process TaxxaS Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process TaxxaS Source Data\"", "delta": "0:00:00.382723", "end": "2018-06-18 15:03:00.482116", "item": "Process TaxxaS Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:03:00.099393", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Swap Cores) => {"changed": true, "cmd": "./add-job.sh \"Swap Cores\"", "delta": "0:00:00.383625", "end": "2018-06-18 15:03:03.802335", "item": "Swap Cores", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:03:03.418710", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}

RUNNING HANDLER [common : restart tomcat] ***********************************************************************************************************************************
        to retry, use: --limit @/home/mbohun/src/ag-bie/ansible/ag-bie.retry

PLAY RECAP ******************************************************************************************************************************************************************
dev-ag-bie.oztaxa.com      : ok=466  changed=37   unreachable=0    failed=1   

real    13m49.676s
user    3m3.576s
sys     0m56.161s

mbohun@linux-khr1:~/src/ag-bie/ansible> date
Mon Jun 18 15:15:05 AEST 2018

today (2018-06-18) it happened twice, one install attempt after other:

TASK [name-processing : make the shell wrapper runnable] ********************************************************************************************************************
ok: [dev-ag-bie.oztaxa.com]

TASK [ag-bie-jenkins-config : ensure work paths exist] **********************************************************************************************************************
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/NSL)
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/NSL/DwCA)
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/NSL/Processed)
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/TaxxaS)
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/TaxxaS/DwCA)
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/TaxxaS/Processed)
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/combined)
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/combined/combined)
ok: [dev-ag-bie.oztaxa.com] => (item=/data/work/tmp)

TASK [ag-bie-jenkins-config : create jenkins jobs on server] ****************************************************************************************************************
ok: [dev-ag-bie.oztaxa.com] => (item=Combine Taxonomies)
ok: [dev-ag-bie.oztaxa.com] => (item=Import Taxonomy)
ok: [dev-ag-bie.oztaxa.com] => (item=Load from SFTP)
ok: [dev-ag-bie.oztaxa.com] => (item=Process NSL Source Data)
ok: [dev-ag-bie.oztaxa.com] => (item=Process TaxxaS Source Data)
ok: [dev-ag-bie.oztaxa.com] => (item=Swap Cores)

TASK [ag-bie-jenkins-config : ensure jenkins is running] ********************************************************************************************************************
changed: [dev-ag-bie.oztaxa.com]

TASK [ag-bie-jenkins-config : wait for jenkins server to get going] *********************************************************************************************************
Pausing for 30 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dev-ag-bie.oztaxa.com]

TASK [ag-bie-jenkins-config : get jenkins cli on server] ********************************************************************************************************************
ok: [dev-ag-bie.oztaxa.com]

TASK [ag-bie-jenkins-config : create job script] ****************************************************************************************************************************
ok: [dev-ag-bie.oztaxa.com]

TASK [ag-bie-jenkins-config : make the job script runnable] *****************************************************************************************************************
ok: [dev-ag-bie.oztaxa.com]

TASK [ag-bie-jenkins-config : add jenkins jobs to jenkins] ******************************************************************************************************************
failed: [dev-ag-bie.oztaxa.com] (item=Combine Taxonomies) => {"changed": true, "cmd": "./add-job.sh \"Combine Taxonomies\"", "delta": "0:00:00.555406", "end": "2018-06-18 15
:27:38.922725", "item": "Combine Taxonomies", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:27:38.367319", "stderr": "java.io.IOException: Server returned 
HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\ta
t sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat h
udson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Ser
ver returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.
java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStrea
m.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't
 get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Import Taxonomy) => {"changed": true, "cmd": "./add-job.sh \"Import Taxonomy\"", "delta": "0:00:00.367484", "end": "2018-06-18 15:27:40
.635036", "item": "Import Taxonomy", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:27:40.267552", "stderr": "java.io.IOException: Server returned HTTP resp
onse code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net
.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli
.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server retur
ned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894
)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:10
1)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job 
list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Load from SFTP) => {"changed": true, "cmd": "./add-job.sh \"Load from SFTP\"", "delta": "0:00:00.360904", "end": "2018-06-18 15:27:42.3
09698", "item": "Load from SFTP", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:27:41.948794", "stderr": "java.io.IOException: Server returned HTTP respons
e code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.ww
w.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CL
I.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned
 HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)",
 "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)"
, "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job lis
t", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process NSL Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process NSL Source Data\"", "delta": "0:00:00.373246", "end": "2018-06-18 15:27:44.010576", "item": "Process NSL Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:27:43.637330", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Process TaxxaS Source Data) => {"changed": true, "cmd": "./add-job.sh \"Process TaxxaS Source Data\"", "delta": "0:00:00.362147", "end": "2018-06-18 15:27:45.703883", "item": "Process TaxxaS Source Data", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:27:45.341736", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}
failed: [dev-ag-bie.oztaxa.com] (item=Swap Cores) => {"changed": true, "cmd": "./add-job.sh \"Swap Cores\"", "delta": "0:00:00.371123", "end": "2018-06-18 15:27:47.357873", "item": "Swap Cores", "msg": "non-zero return code", "rc": 1, "start": "2018-06-18 15:27:46.986750", "stderr": "java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)\n\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)\n\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)\n\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)\n\tat hudson.cli.CLI._main(CLI.java:612)\n\tat hudson.cli.CLI.main(CLI.java:426)", "stderr_lines": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://localhost:9192/cli?remoting=false", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)", "\tat sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)", "\tat hudson.cli.FullDuplexHttpStream.<init>(FullDuplexHttpStream.java:101)", "\tat hudson.cli.CLI.plainHttpConnection(CLI.java:652)", "\tat hudson.cli.CLI._main(CLI.java:612)", "\tat hudson.cli.CLI.main(CLI.java:426)"], "stdout": "Can't get job list", "stdout_lines": ["Can't get job list"]}

RUNNING HANDLER [common : restart tomcat] ***********************************************************************************************************************************
        to retry, use: --limit @/home/mbohun/src/ag-bie/ansible/ag-bie.retry

PLAY RECAP ******************************************************************************************************************************************************************
dev-ag-bie.oztaxa.com      : ok=466  changed=36   unreachable=0    failed=1   

real    11m24.204s
user    2m47.700s
sys     0m54.880s
mbohun@linux-khr1:~/src/ag-bie/ansible> date
Mon Jun 18 15:34:00 AEST 2018

mbohun commented 6 years ago

TASK [nameindex : set name index to use (Custom)] ***************************************************************************************************************************
skipping: [dev-ag-bie.oztaxa.com]

TASK [nameindex : name index URL is] ****************************************************************************************************************************************
ok: [dev-ag-bie.oztaxa.com] => {
    "msg": "Downloading ala index from http://ala-starr.it.csiro.au/archives/nameindexes/20170927/namematching-20170927.tgz"
}

TASK [nameindex : ensure lucene base directory exists] **********************************************************************************************************************
ok: [dev-ag-bie.oztaxa.com]

TASK [nameindex : Download lucene index (ALA)] ******************************************************************************************************************************
fatal: [dev-ag-bie.oztaxa.com]: FAILED! => {"changed": false, "dest": "/data/lucene", "gid": 0, "group": "root", "mode": "0755", "msg": "Request failed: <urlopen error [Errno 111] Connection refused>", "owner": "root", "size": 4096, "state": "directory", "uid": 0, "url": "http://ala-starr.it.csiro.au/archives/nameindexes/20170927/namematching-20170927.tgz"}

RUNNING HANDLER [common : restart tomcat] ***********************************************************************************************************************************
        to retry, use: --limit @/home/mbohun/src/ag-bie/ansible/ag-bie.retry

PLAY RECAP ******************************************************************************************************************************************************************
dev-ag-bie.oztaxa.com      : ok=383  changed=22   unreachable=0    failed=1   

real    8m52.645s
user    2m11.553s
sys     0m41.653s

mbohun commented 6 years ago

VERIFY the jvm/tomcat options

our playbook-s use:

tomcat_java_opts = -Xmx2g -Xms2g -XX:MaxPermSize=1g -Xss512k -Djava.awt.headless=true

However:

Area: HotSpot / gc 
Synopsis
The command line flags PermSize and MaxPermSize have been removed and are ignored. If used on the command line a warning will be emitted for each.

Java HotSpot(TM) Server VM warning: ignoring option PermSize=32m; support 
was removed in 8.0
Java HotSpot(TM) Server VM warning: ignoring option MaxPermSize=128m; support 
was removed in 8.0
Nature of Incompatibility
source

mbohun commented 6 years ago

linux / unix user & group setup

after a successful installation:

ubuntu@ip-172-31-23-2:~$ sudo su tomcat7 -s /bin/bash
tomcat7@ip-172-31-23-2:/home/ubuntu$ id -a
uid=113(tomcat7) gid=117(tomcat7) groups=117(tomcat7)

ubuntu@ip-172-31-23-2:~$ sudo su jenkins -s /bin/bash
jenkins@ip-172-31-23-2:/home/ubuntu$ id -a
uid=115(jenkins) gid=119(jenkins) groups=119(jenkins)

HOWEVER some of the "combine taxonomies" scripts (run from jenkins jobs) assume/require:

ubuntu@ip-172-31-23-2:~$ sudo su tomcat7 -s /bin/bash
tomcat7@ip-172-31-23-2:/home/ubuntu$ id -a
uid=113(tomcat7) gid=117(tomcat7) groups=117(tomcat7),119(jenkins)

ubuntu@ip-172-31-23-2:~$ sudo su jenkins -s /bin/bash
jenkins@ip-172-31-23-2:/home/ubuntu$ id -a
uid=115(jenkins) gid=119(jenkins) groups=119(jenkins),117(tomcat7)

meaning someone/something has to do:

ubuntu@ip-172-31-23-2:~$ sudo usermod -aG jenkins tomcat7
ubuntu@ip-172-31-23-2:~$ sudo usermod -aG tomcat7 jenkins

mbohun commented 6 years ago

No such file or directory ERRORS/PROBLEMS

ubuntu@ip-172-31-23-2:/tmp/ag-bie-solr.git$ sudo su jenkins
jenkins@ip-172-31-23-2:/tmp/ag-bie-solr.git$ ./test_all.sh 
Load_from_SFTP.sh
+ echo 

+ date +%Y%m%d
+ datestamp=20180622
+ echo 20180622
20180622
+ username=dawr
+ configDir=/data/taxxas/config
+ workDir=/data/work/taxxas
+ sourceDir=/data/work/taxxas/20180622
+ credentials=/data/taxxas/config/.ssh/dawr.pem
+ sftpServer=ag-bie.oztaxa.com
+ processDir=/data/taxxas/process/TaxxaS_Package_DwCA
+ [ -d /data/work/taxxas/20180622 ]
+ mkdir -p /data/work/taxxas/20180622
mkdir: cannot create directory ‘/data/work/taxxas’: Permission denied
+ bail Unable to create source directory /data/work/taxxas/20180622
+ echo Unable to create source directory /data/work/taxxas/20180622
Unable to create source directory /data/work/taxxas/20180622
+ exit 1

jenkins@ip-172-31-23-2:/tmp/ag-bie-solr.git$ find /data/taxxas
find: ‘/data/taxxas’: No such file or directory

mbohun commented 6 years ago

./Import_Taxonomy.sh

curl -X GET \
       -f \
       --header 'Host: sit-ag-bie.oztaxa.com' \
       'http://localhost:8080/ws/admin/services/all'

was failing - redirecting to /cas:

because /admin/services.* was missing from /data/bie-index/config/bie-index-config.properties security.cas.uriExclusionFilterPattern property
```
9c9
< security.cas.uriExclusionFilterPattern=/admin/indexFields
---
> security.cas.uriExclusionFilterPattern=/admin/indexFields,/admin/services.*
```
This is supposed to be set in the inventory:
```
bie_index_uri_exclusion_filter_pattern=/admin/indexFields,/admin/services.*
```
The default value, if not set is: /admin/indexFields
TODO

mbohun commented 6 years ago

see: https://gist.github.com/mbohun/d6a21c6d31c66d7a517565c8caa8d454

for more detail-s
additions, inspiration, etc.

moziauddin commented 4 years ago

The installation is fully automated using Jenkins, ansible and bash scripts. Code in this repo.

ess-acppo / ag-bie

Fully automate ag-bie installation / env setup #2

CRUCIAL FIXES

ENHANCEMENTS / ADDITIONS

REFERENCES

THIS NEEDS DOUBLE_CHECKING

REFACTORING

ansible Jenkins jobs re-install verification

VERIFY the jvm/tomcat options

linux / unix user & group setup

No such file or directory ERRORS/PROBLEMS