search

typesafehub / conductr-cli

CLI for Lightbend ConductR
Other
16 stars 21 forks source link

Bundle validation #469

Closed markusjura closed 7 years ago

markusjura commented 7 years ago

The bndl command is now validating the content of the bundle.conf in case one is created or updated given the bndl arguments and a possible bundle.conf as input.

The following bundle.conf are performed:

Also, the bndl implementation has changed to support all use cases described in https://docs.google.com/document/d/11aEKF8SUv-7f7HrOypAG7CICJLXIhqA-6M14kdzt_dM/edit. In particular the following use cases are now supported as well:

Note that when creating a bundle configuration, the format need to be explicitly set to --format configuration. The format auto-detection of the bndl command cannot distinguish between a bundle and a configuration. Because the default use case of the bndl command is to create or modify a bundle, the format auto-detection will fallback to bundle.

markusjura commented 7 years ago

Manual tests

Producing bundles

Creating bundle configuration without input

Creating archive with bundle.conf and runtime-config.sh

$ bndl --format configuration --env 'KEY=VALUE' --tag 1.0.0 -o /tmp/my-bundle-conf.zip

$ zipinfo /tmp/my-bundle-conf.zip
Archive:  /tmp/my-bundle-conf.zip
Zip file size: 369 bytes, number of entries: 2
?rw-r--r--  2.0 unx       20 b- stor 80-Jan-03 00:59 bundle/bundle.conf
-rw-r--r--  2.0 unx       18 b- stor 80-Jan-03 00:59 bundle/runtime-config.sh
2 files, 38 bytes uncompressed, 38 bytes compressed:  0.0%

$ unzip -c /tmp/my-bundle-conf.zip bundle/bundle.conf | cat
Archive:  /tmp/my-bundle-conf.zip
 extracting: bundle/bundle.conf
tags = [
  "1.0.0"
]

$ unzip -c /tmp/my-bundle-conf.zip bundle/runtime-config.sh | cat
Archive:  /tmp/my-bundle-conf.zip
 extracting: bundle/runtime-config.sh
export 'KEY=VALUE'

conduct load a new bundle configuration

This test case failed due to a current bug in the master branch. This issue will be solved within another PR.

$ bndl --format configuration --env 'KEY=VALUE' --tag 1.0.0 | conduct load visualizer -
Retrieving bundle..
Loading bundle from cache typesafe/bundle/visualizer
Bintray credentials loaded from /Users/mj/.lightbend/commercial.credentials
Retrieving from cache /Users/mj/.conductr/cache/bundle/visualizer-v2-6cc7dbcf3b655b1942edf50a5574d62f5bbb12ef37db5b3990dc3fa80078b827.zip
Retrieving configuration..
Loading bundle configuration from cache typesafe/bundle-configuration/-
Bintray credentials loaded from /Users/mj/.lightbend/commercial.credentials
Retrieving file:///Users/mj/workspace/conductr-cli/-
Resolving bundle configuration typesafe/bundle-configuration/-
Bintray credentials loaded from /Users/mj/.lightbend/commercial.credentials
Error: Bundle not found: Unable to resolve bundle using -

Creating bundle or bundle configuration from input

From bundle directory

$ bndl ~/.conductr/cache/bundle/visualizer --env 'KEY=VALUE' --tag 1.0.0 -o /tmp/my-bundle.zip

$ unzip -c /tmp/my-bundle.zip visualizer/bundle.conf | cat
Archive:  /tmp/my-bundle.zip
 extracting: visualizer/bundle.conf
version = "1"
name = "visualizer"
compatibilityVersion = "2"
system = "visualizer"
systemVersion = "2"
nrOfCpus = 0.1
memory = 402653184
diskSpace = 200000000
roles = [
  "web"
]
components {
  visualizer {
    description = "visualizer"
    file-system-type = "universal"
    start-command = [
      "visualizer/bin/visualizer"
      "-J-Xms134217728"
      "-J-Xmx134217728"
      "-Dhttp.address=$VISUALIZER_BIND_IP"
      "-Dhttp.port=$VISUALIZER_BIND_PORT"
    ]
    endpoints {
      visualizer {
        bind-protocol = "http"
        bind-port = 0
        service-name = "visualizer"
        acls = [
          {
            http {
              requests = [
                {
                  path-beg = "/"
                }
              ]
            }
          }
        ]
      }
    }
  }
}
tags = [
  "1.0.0"
]

$ unzip -c /tmp/my-bundle.zip visualizer/runtime-config.sh | cat
Archive:  /tmp/my-bundle.zip
 extracting: visualizer/runtime-config.sh
export 'KEY=VALUE'

From app directory

$ cd ~/workspace/conductr
$ sbt visualizer/stage
$ cd ~/workspace/conductr-cli

$ bndl ~/workspace/conductr/visualizer/target/universal/stage --format configuration --env 'KEY=VALUE' --tag 1.0.0 -o /tmp/my-bundle-conf.zip

$ unzip -c /tmp/my-bundle-conf.zip bundle/bundle.conf | cat
Archive:  /tmp/my-bundle-conf.zip
 extracting: bundle/bundle.conf
tags = [
  "1.0.0"
]

$ unzip -c /tmp/my-bundle-conf.zip bundle/runtime-config.sh | cat
Archive:  /tmp/my-bundle-conf.zip
 extracting: bundle/runtime-config.sh
export 'KEY=VALUE'

From bundle.conf

$ bndl ~/.conductr/cache/bundle/visualizer/bundle.conf --format configuration --env 'KEY=VALUE' --tag 1.0.0 -o /tmp/my-bundle-conf.zip

$ unzip -c /tmp/my-bundle-conf.zip visualizer/bundle.conf | cat
Archive:  /tmp/my-bundle-conf.zip
 extracting: visualizer/bundle.conf
version = "1"
name = "visualizer"
compatibilityVersion = "2"
system = "visualizer"
systemVersion = "2"
nrOfCpus = 0.1
memory = 402653184
diskSpace = 200000000
roles = [
  "web"
]
components {
  visualizer {
    description = "visualizer"
    file-system-type = "universal"
    start-command = [
      "visualizer/bin/visualizer"
      "-J-Xms134217728"
      "-J-Xmx134217728"
      "-Dhttp.address=$VISUALIZER_BIND_IP"
      "-Dhttp.port=$VISUALIZER_BIND_PORT"
    ]
    endpoints {
      visualizer {
        bind-protocol = "http"
        bind-port = 0
        service-name = "visualizer"
        acls = [
          {
            http {
              requests = [
                {
                  path-beg = "/"
                }
              ]
            }
          }
        ]
      }
    }
  }
}
tags = [
  "1.0.0"
]

$ unzip -c /tmp/my-bundle-conf.zip visualizer/runtime-config.sh | cat
Archive:  /tmp/my-bundle-conf.zip
 extracting: visualizer/runtime-config.sh
export 'KEY=VALUE'

From input stream (docker)

$ docker save hello-world | bndl --env 'KEY=VALUE' --tag 1.0.0 -o /tmp/my-bundle.zip

$ unzip -c /tmp/my-bundle.zip hello-world/bundle.conf | cat
Archive:  /tmp/my-bundle.zip
 extracting: hello-world/bundle.conf
annotations {
  com {
    lightbend {
      conductr {
        oci-image-tags {
          hello-world = "latest"
        }
      }
    }
  }
}
compatibilityVersion = "0"
diskSpace = 1073741824
memory = 402653184
name = "hello-world"
nrOfCpus = 0.1
roles = [
  "web"
]
system = "hello-world"
systemVersion = "1"
tags = [
  "1.0.0"
]
version = "1"
components {
  hello-world {
    description = ""
    file-system-type = "oci-image"
    start-command = []
    endpoints {}
  }
}

$ unzip -c /tmp/my-bundle.zip hello-world/runtime-config.sh | cat
Archive:  /tmp/my-bundle.zip
 extracting: hello-world/runtime-config.sh
export 'KEY=VALUE'

Validation errors

Required check

The --format is bundle so we check if all required fields are part of the bundle.conf.

$ bndl --format bundle --tag 1.0.0 -o /tmp/my-bundle.zip
Error: bndl: bundle.conf validation errors:
  The following required properties are not declared: compatibilityVersion, components, diskSpace, memory, name, nrOfCpus, roles, system, systemVersion, version

Valid property names check

Given the following bundle.conf as an input:

$ cat ~/.conductr/cache/bundle/visualizer/bundle.conf
invalid-property = "some-data"
version = "1"
name = "visualizer"
compatibilityVersion = "2"
system = "visualizer"
systemVersion = "2"
nrOfCpus = 0.1
memory = 402653184
diskSpace = 200000000
roles = [
  "web"
]
components {
  visualizer {
    description = "visualizer"
    file-system-type = "universal"
    start-commands = [
      "visualizer/bin/visualizer"
      "-J-Xms134217728"
      "-J-Xmx134217728"
      "-Dhttp.address=$VISUALIZER_BIND_IP"
      "-Dhttp.port=$VISUALIZER_BIND_PORT"
    ]
    endpoints {
      visualizer {
        bind-protocol = "http"
        bind-port = 0
        service-name = "visualizer"
        acls = [
          {
            http {
              requests = [
                {
                  path-beg = "/"
                }
              ]
            }
          }
        ]
      }
    }
  }
}

.. the property name validation will fail:

$ bndl ~/.conductr/cache/bundle/visualizer/bundle.conf -o /tmp/my-bundle.zip
Error: bndl: bundle.conf validation errors:
  The following property names are invalid: components.visualizer.start-commands, invalid-property
  The following required properties are not declared: components.visualizer.start-command

Validations can be excluded if necessary:

$ bndl ~/.conductr/cache/bundle/visualizer/bundle.conf -o /tmp/my-bundle.zip --validation-exclude required --validation-exclude property-names
> 0

sandbox run from ~/workspace/conductr

~/workspace/conductr$ sandbox run 2.0.7
|------------------------------------------------|
| Starting ConductR                              |
|------------------------------------------------|
Extracting ConductR core to /Users/mj/.conductr/images/core
Extracting ConductR agent to /Users/mj/.conductr/images/agent
Starting ConductR core instance on 192.168.10.1..
Waiting for ConductR to start..
Starting ConductR agent instance on 192.168.10.1..
|------------------------------------------------|
| Starting HAProxy                               |
|------------------------------------------------|
Exposing the following ports [80, 443, 3000, 5601, 8999, 9000, 9200, 9999]
53735884865ddf7a870a381792b43a75d0c822fe43fe00086bf6f1731c614764
Deploying bundle conductr-haproxy with configuration conductr-haproxy-dev-mode
Retrieving bundle..
Loading bundle from cache typesafe/bundle/conductr-haproxy
Bintray credentials loaded from /Users/mj/.lightbend/commercial.credentials
Retrieving file:///Users/mj/workspace/conductr/conductr-haproxy
Resolving bundle typesafe/bundle/conductr-haproxy
Bintray credentials loaded from /Users/mj/.lightbend/commercial.credentials
Retrieving https://dl.bintray.com/typesafe/bundle/typesafe/conductr-haproxy/v2-a10e59a4fc021ebfb4bf729ddc66608d7ffcd3e91dbf823d736276b6d2005c2b/conductr-haproxy-v2-a10e59a4fc021ebfb4bf729ddc66608d7ffcd3e91dbf823d736276b6d2005c2b.zip
[#################################################] 100%
Retrieving configuration..
Loading bundle configuration from cache typesafe/bundle-configuration/conductr-haproxy-dev-mode
Bintray credentials loaded from /Users/mj/.lightbend/commercial.credentials
Retrieving from cache /Users/mj/.conductr/cache/configuration/conductr-haproxy-dev-mode-v2-e5f35046d59c27f4cfeebcc026f3e3b76e027b15223a8961b59d728e559402e2.zip
Loading bundle to ConductR..
[#################################################] 100%
Bundle a10e59a4fc021ebfb4bf729ddc66608d-e5f35046d59c27f4cfeebcc026f3e3b7 is installed
Bundle loaded.
Bundle run request sent.
Bundle a10e59a4fc021ebfb4bf729ddc66608d-e5f35046d59c27f4cfeebcc026f3e3b7 waiting to reach expected scale 1
Bundle a10e59a4fc021ebfb4bf729ddc66608d-e5f35046d59c27f4cfeebcc026f3e3b7 has scale 0, expected 1.
Bundle a10e59a4fc021ebfb4bf729ddc66608d-e5f35046d59c27f4cfeebcc026f3e3b7 expected scale 1 is met
|------------------------------------------------|
| Starting logging feature based on eslite       |
|------------------------------------------------|
Deploying bundle eslite..
Retrieving bundle..
Loading bundle from cache typesafe/bundle/eslite
Bintray credentials loaded from /Users/mj/.lightbend/commercial.credentials
Retrieving from cache /Users/mj/.conductr/cache/bundle/eslite-v1-7dc78f8c80474dc7ff4cabbdf5b5ce386cdebc3bd52693601bd2383dfb4e86be.zip
Loading bundle to ConductR..
[#################################################] 100%
Bundle 7dc78f8c80474dc7ff4cabbdf5b5ce38 is installed
Bundle loaded.
Bundle run request sent.
Bundle 7dc78f8c80474dc7ff4cabbdf5b5ce38 waiting to reach expected scale 1
Bundle 7dc78f8c80474dc7ff4cabbdf5b5ce38 has scale 0, expected 1.
Bundle 7dc78f8c80474dc7ff4cabbdf5b5ce38 expected scale 1 is met
|------------------------------------------------|
| Summary                                        |
|------------------------------------------------|
|- - - - - - - - - - - - - - - - - - - - - - - - |
| ConductR                                       |
|- - - - - - - - - - - - - - - - - - - - - - - - |
ConductR has been started:
  core instance on 192.168.10.1
  agent instance on 192.168.10.1
ConductR service locator has been started on:
  192.168.10.1:9008
|- - - - - - - - - - - - - - - - - - - - - - - - |
| Proxy                                          |
|- - - - - - - - - - - - - - - - - - - - - - - - |
HAProxy has been started
By default, your bundles are accessible on:
  192.168.10.1:9000
|- - - - - - - - - - - - - - - - - - - - - - - - |
| Bundles                                        |
|- - - - - - - - - - - - - - - - - - - - - - - - |
Check latest bundle status with:
  conduct info
Current bundle status:
ID               NAME              VER  #REP  #STR  #RUN  ROLES
a10e59a-e5f3504  conductr-haproxy   v2     1     0     1  haproxy
7dc78f8          eslite             v1     1     0     1  elasticsearch