merlijn-sebrechts commented 7 years ago

I get the error: bundle URL has invalid form: "/tmp/builds/docker".

How I added the charm:

Note: I'm not sure why I need to specify a "reference bundle". The Charm tests deploy the needed charms themselves.

juju run-action cwr/0 build-on-commit \
  repo=https://github.com/IBCNServices/layer-docker \
  charm-name=docker \
  reference-bundle=cs:~tengu-team/limeds-core \
  push-to-channel=edge \
  lp-id=tengu-team \
  controller=msebrechts-amazon

Relevant lines in log

    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html>
        <head>
            <title>Error: 406 Not Acceptable</title>
            <style type="text/css">
              html {background-color: #eee; font-family: sans;}
              body {background-color: #fff; border: 1px solid #ddd;
                    padding: 15px; margin: 15px;}
              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
            </style>
        </head>
        <body>
            <h1>Error: 406 Not Acceptable</h1>
            <p>Sorry, the requested URL <tt>&#039;http://svg.juju.solutions/&#039;</tt>
               caused an error:</p>
            <pre>/usr/local/bin/python-jujusvg /tmp/tmpxSV5x9: 2017/01/19 16:07:46 FAIL: Error generating canvas: cannot parse charm &quot;/tmp/builds/docker&quot;: charm or bundle URL has invalid form: &quot;/tmp/builds/docker&quot;

</pre>
        </body>
    </html>

bundle.yaml:
machines:
  '0': {constraints: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740 root-disk=8192,
    series: xenial}
relations:
- ['limeds:dockerhost', 'docker:dockerhost']
series: xenial
services:
  docker:
    annotations: {gui-x: 700, gui-y: 300}
    charm: /tmp/builds/docker
    num_units: 1
    to: ['0']
  limeds:
    annotations: {gui-x: 400, gui-y: 300}
    charm: cs:~tengu-team/limeds-0
    num_units: 1
    to: ['0']

Complete log

Started by user admin
Building in workspace /var/lib/jenkins/jobs/charm-docker/workspace
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/IBCNServices/layer-docker # timeout=10
Fetching upstream changes from https://github.com/IBCNServices/layer-docker
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/IBCNServices/layer-docker +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision dadcbcad4d68ad7a353ad2928fb3ace157cb035d (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f dadcbcad4d68ad7a353ad2928fb3ace157cb035d
 > git rev-list dadcbcad4d68ad7a353ad2928fb3ace157cb035d # timeout=10
[workspace] $ /bin/bash /tmp/hudson4606823355749867228.sh
+ '[' '!' -f first-run.lock ']'
+ export JUJU_REPOSITORY=/tmp/
+ JUJU_REPOSITORY=/tmp/
+ OUTPUT_DIR=builds
+ SERIES_FLAG=
+ '[' '!' -z '' ']'
+ charm build ''
build: Composing into /tmp/
build: Destination charm directory: /tmp/builds/docker
build: Processing layer: layer:basic
build: Processing layer: layer:apt
build: Processing layer: docker
build: Processing interface: docker-image-host
proof: OK!
+ CONTROLLERS=
+ MODELS_TO_TEST=
+ '[' -z msebrechts-amazon ']'
+ CONTROLLERS=msebrechts-amazon
+ for controller in '$CONTROLLERS'
+ juju switch msebrechts-amazon
msebrechts-amazon (controller) (no change)
++ juju add-model wont-be-added --credential invalid-credential
++ sed -e 's/.*cloud "\?\([^ "]*\)"\?.*/\1/'
+ cloud=aws
+ credential_arg=
+ [[ aws != localhost* ]]
+ [[ aws != lxd ]]
+ juju credentials --format=json
+ grep -q aws
++ juju credentials --format=json
++ jq -r '.credentials.aws."cloud-credentials" | keys[0]'
+ credential=amazon-merlijn
+ credential_arg=--credential=amazon-merlijn
+ juju add-model test-docker-2 --credential=amazon-merlijn
Uploading credential 'aws/ciuser/amazon-merlijn' to controller
Added 'test-docker-2' model on aws/us-east-1 with credential 'amazon-merlijn' for user 'ciuser'
+ MODELS_TO_TEST+='msebrechts-amazon:test-docker-2 '
+ eval 'function cleanup_msebrechts-amazon_model() { juju destroy-model msebrechts-amazon:test-docker-2 -y; }'
++ trap -p EXIT
++ awk '-F'\''' '{print $2}'
+ old_trap=
+ new_trap='cleanup_msebrechts-amazon_model;'
+ trap 'cleanup_msebrechts-amazon_model;' EXIT
+ '[' -e /tmp/bundles ']'
+ mkdir /tmp/bundles
+ rm -rf /tmp/bundles/cs__tengu_team_limeds_core
+ charm pull 'cs:~tengu-team/limeds-core' /tmp/bundles/cs__tengu_team_limeds_core
cs:~tengu-team/bundle/limeds-core-0
++ update_bundle
++ cat
+ python3 -c 'import yaml
with open('\''/tmp/bundles/cs__tengu_team_limeds_core/bundle.yaml'\'', '\''r'\'') as fp:
    bundle = yaml.safe_load(fp)
app = bundle.get('\''applications'\'', bundle.get('\''services'\'', {}))['\''docker'\'']
app['\''charm'\''] = '\''/tmp/builds/docker'\''
with open('\''/tmp/bundles/cs__tengu_team_limeds_core/bundle.yaml'\'', '\''w'\'') as fp:
    yaml.dump(bundle, fp)'
+ rm -f totest.yaml
+ echo 'bundle: /tmp/bundles/cs__tengu_team_limeds_core'
+ echo 'bundle_name: docker_in_cs__tengu_team_limeds_core'
+ echo 'bundle_file: bundle.yaml'
+ eval 'function transform_results() { python3 /var/lib/juju/agents/unit-cwr-1/charm/scripts/json2junit.py /srv/artifacts/ hadoop_client_in_cs_hadoop_processing 2  > /var/lib/jenkins/jobs/charm-docker/workspace/report.xml; }'
++ trap -p EXIT
++ awk '-F'\''' '{print $2}'
+ old_trap='cleanup_msebrechts-amazon_model;'
+ new_trap='transform_results; cleanup_msebrechts-amazon_model;'
+ trap 'transform_results; cleanup_msebrechts-amazon_model;' EXIT
+ ARTIFACTS_DIR=/srv/artifacts/docker_in_cs__tengu_team_limeds_core/2
+ mkdir -p /srv/artifacts/docker_in_cs__tengu_team_limeds_core/2
+ ln -s /srv/artifacts/docker_in_cs__tengu_team_limeds_core/2 /var/lib/jenkins/jobs/charm-docker/builds/2/archive
+ env MATRIX_OUTPUT_DIR=/srv/artifacts/docker_in_cs__tengu_team_limeds_core/2 cwr -F msebrechts-amazon:test-docker-2 totest.yaml --results-dir /srv/artifacts --test-id 2
2017-01-19 15:57:27 INFO Running test on AWS.
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin: 
Selecting previously unselected package python-backports.ssl-match-hostname.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 70131 files and directories currently installed.)
Preparing to unpack .../python-backports.ssl-match-hostname_3.4.0.2-1_all.deb ...
Unpacking python-backports.ssl-match-hostname (3.4.0.2-1) ...
Selecting previously unselected package python-websocket.
Preparing to unpack .../python-websocket_0.18.0-2_all.deb ...
Unpacking python-websocket (0.18.0-2) ...
Selecting previously unselected package python-jujuclient.
Preparing to unpack .../python-jujuclient_0.50.5-0ubuntu1_all.deb ...
Unpacking python-jujuclient (0.50.5-0ubuntu1) ...
Selecting previously unselected package juju-deployer.
Preparing to unpack .../juju-deployer_0.6.4-0ubuntu1_all.deb ...
Unpacking juju-deployer (0.6.4-0ubuntu1) ...
Selecting previously unselected package libpython3-all-dev:amd64.
Preparing to unpack .../libpython3-all-dev_3.5.1-3_amd64.deb ...
Unpacking libpython3-all-dev:amd64 (3.5.1-3) ...
Selecting previously unselected package python3-all.
Preparing to unpack .../python3-all_3.5.1-3_amd64.deb ...
Unpacking python3-all (3.5.1-3) ...
Selecting previously unselected package python3-all-dev.
Preparing to unpack .../python3-all-dev_3.5.1-3_amd64.deb ...
Unpacking python3-all-dev (3.5.1-3) ...
Selecting previously unselected package python3-theblues.
Preparing to unpack .../python3-theblues_0.3.7-0ubuntu1~ubuntu16.04.1~ppa2_all.deb ...
Unpacking python3-theblues (0.3.7-0ubuntu1~ubuntu16.04.1~ppa2) ...
Selecting previously unselected package python3-libcharmstore.
Preparing to unpack .../python3-libcharmstore_0.0.3-1ubuntu4~ubuntu16.04.1~ppa1_all.deb ...
Unpacking python3-libcharmstore (0.0.3-1ubuntu4~ubuntu16.04.1~ppa1) ...
Selecting previously unselected package python3-path.
Preparing to unpack .../python3-path_8.1.2+dfsg-1ubuntu1~ubuntu16.04.1~ppa1_all.deb ...
Unpacking python3-path (8.1.2+dfsg-1ubuntu1~ubuntu16.04.1~ppa1) ...
Selecting previously unselected package python3-amulet.
Preparing to unpack .../python3-amulet_1.18.1-0ubuntu1~ubuntu16.04.1~ppa2_all.deb ...
Unpacking python3-amulet (1.18.1-0ubuntu1~ubuntu16.04.1~ppa2) ...
Selecting previously unselected package python-amulet.
Preparing to unpack .../python-amulet_1.18.1-0ubuntu1~ubuntu16.04.1~ppa2_all.deb ...
Unpacking python-amulet (1.18.1-0ubuntu1~ubuntu16.04.1~ppa2) ...
Selecting previously unselected package amulet.
Preparing to unpack .../amulet_1.18.1-0ubuntu1~ubuntu16.04.1~ppa2_all.deb ...
Unpacking amulet (1.18.1-0ubuntu1~ubuntu16.04.1~ppa2) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up python-backports.ssl-match-hostname (3.4.0.2-1) ...
Setting up python-websocket (0.18.0-2) ...
Setting up python-jujuclient (0.50.5-0ubuntu1) ...
Setting up juju-deployer (0.6.4-0ubuntu1) ...
Setting up libpython3-all-dev:amd64 (3.5.1-3) ...
Setting up python3-all (3.5.1-3) ...
Setting up python3-all-dev (3.5.1-3) ...
Setting up python3-theblues (0.3.7-0ubuntu1~ubuntu16.04.1~ppa2) ...
Setting up python3-libcharmstore (0.0.3-1ubuntu4~ubuntu16.04.1~ppa1) ...
Setting up python3-path (8.1.2+dfsg-1ubuntu1~ubuntu16.04.1~ppa1) ...
Setting up python3-amulet (1.18.1-0ubuntu1~ubuntu16.04.1~ppa2) ...
Setting up python-amulet (1.18.1-0ubuntu1~ubuntu16.04.1~ppa2) ...
Setting up amulet (1.18.1-0ubuntu1~ubuntu16.04.1~ppa2) ...
2017-01-19 16:07:46 INFO Starting new HTTP connection (1): svg.juju.solutions
2017-01-19 16:07:47 WARNING Could not generate svg. Response from svg.juju.solutions: 
Status code:406 
Content: 
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html>
        <head>
            <title>Error: 406 Not Acceptable</title>
            <style type="text/css">
              html {background-color: #eee; font-family: sans;}
              body {background-color: #fff; border: 1px solid #ddd;
                    padding: 15px; margin: 15px;}
              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
            </style>
        </head>
        <body>
            <h1>Error: 406 Not Acceptable</h1>
            <p>Sorry, the requested URL <tt>&#039;http://svg.juju.solutions/&#039;</tt>
               caused an error:</p>
            <pre>/usr/local/bin/python-jujusvg /tmp/tmpxSV5x9: 2017/01/19 16:07:46 FAIL: Error generating canvas: cannot parse charm &quot;/tmp/builds/docker&quot;: charm or bundle URL has invalid form: &quot;/tmp/builds/docker&quot;

</pre>
        </body>
    </html>

bundle.yaml:
machines:
  '0': {constraints: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740 root-disk=8192,
    series: xenial}
relations:
- ['limeds:dockerhost', 'docker:dockerhost']
series: xenial
services:
  docker:
    annotations: {gui-x: 700, gui-y: 300}
    charm: /tmp/builds/docker
    num_units: 1
    to: ['0']
  limeds:
    annotations: {gui-x: 400, gui-y: 300}
    charm: cs:~tengu-team/limeds-0
    num_units: 1
    to: ['0']

+ transform_results
+ python3 /var/lib/juju/agents/unit-cwr-1/charm/scripts/json2junit.py /srv/artifacts/ hadoop_client_in_cs_hadoop_processing 2
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cwr-1/charm/scripts/json2junit.py", line 22, in <module>
    with open(json_filename, 'r') as json_fp:
FileNotFoundError: [Errno 2] No such file or directory: '/srv/artifacts//hadoop_client_in_cs_hadoop_processing/2/report.json'
Build step 'Execute shell' marked build as failure
Recording test results
Finished: FAILURE

pengale commented 7 years ago

We are working on this bug right now, and it should be fixed in the near future. Basically, some of the things that we do to make the bundle available locally break bundletester. @ktsakalozos pointed out that we probably just need to pass -b <local bundle path> to fix this.

To answer your question about the reference bundle: we need it because these tests run matrix, which should a) eliminate the need for you to write basic deploy tests in amulet, but b) needs to be pointed at a bundle.

merlijn-sebrechts commented 7 years ago

Another way to fix this is to push the charms before testing, but don't release them. Then specify those exact revisions in the bundle. Charms that are pushed but not released can be accessed by specifying the exact revision but they won't be visible in the store. If the tests succeed, you release the charms. If not, you don't release them.

That's what I did in my ci pipeline...

kwmonroe commented 7 years ago

Hey @galgalesh, couple things to note here...

The bundle URL has invalid form message really isn't a problem. That's just telling you that we were unable to generate a bundle svg icon. While it makes the log a bit hairy, it's not actually causing a failure. We do plan to fix this at some point (https://github.com/juju-solutions/layer-cwr/issues/32).
The real error is:

FileNotFoundError: [Errno 2] No such file or directory: '/srv/artifacts//hadoop_client_in_cs_hadoop_processing/2/report.json'

That happened because @ktsakalozos fell asleep on his keyboard. It was fixed a few weeks ago with https://github.com/juju-solutions/layer-cwr/commit/6372e3089223a2eff5b6b106fa57e3af7ac72eca.

I'd also like to mention that you can omit the reference-bundle argument if your charm specifies one in its ./tests/tests.yaml. For example, the docker charm in your case could add a reference-bundle: cs:~tengu-team/limeds-core in ./tests/tests.yaml and you could omit it from your action invocation.

However, because limeds-core may not be a suitable reference bundle for all users of your docker charm, we allow you to specify one with the action param. This also helps when charm authors write poor tests -- i know, gasp!. By requiring a reference bundle, we're forcing users to test charms in a larger context.

On a side note, we've heard some users lamenting the reference bundle because it takes too long to run (esp if the bundle includes charms with extensive tests like cassandra/postgres). I've opened https://github.com/juju-solutions/layer-cwr/issues/60 to see what we can do about limiting tests in some scenarios.

Ok, so back to the original point of this comment... I'm closing this issue because Kostas fixed the real bug a while back :)

juju-solutions / bundle-cwr-ci

bundle URL has invalid form: "/tmp/builds/docker" #5

Relevant lines in log

Complete log