unable to create-release (can gen manifest, d/l binaries & create blobs fine)

[brybinary]

Hi

following the guide but it fails on creating the release <- I am using bosh v2. binaries/files are where they are supposed to be (afaik).

bosh version: version 2.0.29-1f8ec43-2017-07-14T16:10:31Z go version: go version go1.9 darwin/amd64 spiff version: spiff version 1.0.7 virtualbox version: 5.1.22r115126 OS: mac osx El Capitan 10.11.6

Error: user@machine:~/deployments/vbox/splunk-firehose-nozzle-release$ ./scripts/build-release.sh

Building splunk-firehose-nozzle-release

Cleaning up blobs Downloading binaries Go package already exist, skipping download Splunk package already exist, skipping download Adding blobs Added blob 'golang'

Succeeded Added blob 'golang'

Succeeded Added blob 'splunk'

Succeeded Added blob 'splunk'

Succeeded Creating release Building a release from directory '/Users/USER/deployments/vbox/splunk-firehose-nozzle-release':

• Constructing packages from directory:
  • Reading package from '/Users/USER/deployments/vbox/splunk-firehose-nozzle-release/packages/common': Collecting package files: Missing files for pattern 'common/*/'
  • Reading package from '/Users/USER/deployments/vbox/splunk-firehose-nozzle-release/packages/golang': Collecting package files: Missing files for pattern 'golang/go-linux-amd64.tar.gz'
  • Reading package from '/Users/USER/deployments/vbox/splunk-firehose-nozzle-release/packages/splunk': Collecting package files: Missing files for pattern 'splunk/splunk-linux-x86_64.tgz'
  • Expected to find package 'golang' since it's a dependency of package 'splunk-nozzle'
• Constructing jobs from manifest:
  • Expected to find package 'splunk' since it's a dependency of job 'splunk-forwarder'
  • Expected to find package 'splunk' since it's a dependency of job 'splunk-full'
  • Expected to find package 'common' since it's a dependency of job 'splunk-nozzle'

Exit code 1

Any pointers appreciated.

Thx

[chenziliang]

could you please go to Splunk-firehose-nozzle repo and follow the instructions in readme to deploy nozzle as a PCF app ?

[brybinary]

Hi chenziliang

I will give that a try and report back.

Thx

[brybinary]

update:

followed deploying the nozzle into cf (https://github.com/cloudfoundry-community/splunk-firehose-nozzle) and it seemed to work ok.

I do still require to deploy splunk via bosh, so will continue looking into why it failed when creating the release. As always, any pointers greatly appreciated!

Here's an output showing the nozzle being deployed into CF (#2), as suggested.

Thx

output Starting app splunk-firehose-nozzle in org cloudfoundry / space development as admin... Downloading go_buildpack... Downloaded go_buildpack (4.6M) Creating container Successfully created container Downloading app package... Downloaded app package (5.5M) Staging... -----> Go Buildpack version 1.8.6 -----> Installing godep v79 -----> Running: go install -tags cloudfoundry -buildmode pie . Exit status 0 Staging complete Uploading droplet, build artifacts cache... Uploading build artifacts cache... Uploading droplet... Uploaded build artifacts cache (215B) Uploaded droplet (8M) Uploading complete Stopping instance 7975897a-3cd0-4049-b3a5-bf7075810d88 Destroying container Successfully destroyed container

2 of 2 instances running

App started OK App splunk-firehose-nozzle was started using this command splunk-firehose-nozzle Showing health and status for app splunk-firehose-nozzle in org cloudfoundry / space development as admin... OK

requested state: started instances: 2/2 usage: 512M x 2 instances urls: splunk-firehose-nozzle-northern-bedspring.bosh-lite.com last uploaded: Fri Sep 1 17:06:31 UTC 2017 stack: cflinuxfs2 buildpack: go_buildpack

 state     since                    cpu    memory         disk         details

0 running 2017-09-01 06:07:43 PM 0.0% 564K of 512M 1.3M of 1G

1 running 2017-09-01 06:07:43 PM 0.0% 0 of 512M 0 of 1G

user@machine:~/deployments/splunk_nozzle$

[brybinary]

update 2:

updated the ./scripts/build-release to the following, which then seemed to allow the release to be created. Just doing some further testing on the deployment. Happy to receive feedback, as to why putting the splunk-linux-x86_64.tgz as a blob for common made this progress through creating release...

my ./scripts/build-release.sh `#!/usr/bin/env bash

set -e

if [ "$0" != "./scripts/build-release.sh" ]; then echo "'build-release.sh' should be run from repository root" exit 1 fi

function usage(){

&2 echo " Usage: $0 [version]

Ex: $0 0+dev.1 " exit 1 }

if [ "$1" == "-h" ] || [ "$1" == "--help" ] || [ "$1" == "help" ]; then usage fi

if [ "$#" -gt 0 ]; then if [ -e "$1" ]; then export version=cat $1 else export version=$1 fi fi

echo "Building splunk-firehose-nozzle-release ${version}" echo ""

echo "Cleaning up blobs" rm -rf .blobs/ blobs/ echo "--- {} " > config/blobs.yml

echo "Downloading binaries"

mkdir -p tmp

export go_pkg_path=./tmp/go-linux-amd64.tar.gz export go_version_path=./tmp/go-version.txt export go_pkg_remote=https://storage.googleapis.com/golang/go1.7.4.linux-amd64.tar.gz

export splunk_pkg_path=./tmp/splunk-linux-x86_64.tgz export splunk_version_path=./tmp/splunk-version.txt export splunk_pkg_remote=https://download.splunk.com/products/splunk/releases/6.5.1/linux/splunk-6.5.1-f74036626f0c-Linux-x86_64.tgz

export common_pkg_path=./packages/common/splunk-linux-x86_64.tgz

if [ -a "${go_pkg_path}" ]; then echo "Go package already exist, skipping download" else echo "Go package doesn't exist, downloading" wget "${go_pkg_remote}" -O "${go_pkg_path}" fi echo "${go_pkg_remote}" > "${go_version_path}"

if [ -a "${splunk_pkg_path}" ]; then echo "Splunk package already exist, skipping download" else echo "Splunk pagage doesn't exist, downloading" wget "${splunk_pkg_remote}" -O "${splunk_pkg_path}" fi echo "${splunk_pkg_remote}" > "${splunk_version_path}"

if [ -a "${common_pkg_path}" ]; then echo "Common package path already exists, nothing to see here" fi echo "${common_pkg_path}" done

echo "Adding blobs" bosh2 -e vbox add-blob "${go_pkg_path}" golang/go-linux-amd64.tar.gz bosh2 -e vbox add-blob "${go_version_path}" golang/go-version.txt bosh2 -e vbox add-blob "${splunk_pkg_path}" splunk/splunk-linux-x86_64.tgz bosh2 -e vbox add-blob "${splunk_version_path}" splunk/splunk-version.txt bosh2 -e vbox add-blob "${common_pkg_path}" common/splunk-linux-x86_64.tgz

echo "Creating release" create_cmd="bosh2 -e vbox create-release --name cf-splunk --tarball="./release.tgz" --force" if [ "$version" != "" ]; then create_cmd+=" --version "${version}"" fi

eval ${create_cmd}`

releases check on director {8:49}~/splunk-firehose-nozzle-release:master ✗ ➭ bosh2 -e vbox releases Using environment '192.168.50.6' as client 'admin'

Name Version Commit Hash cf-splunk 0+dev.1 18f3418+

(*) Currently deployed (+) Uncommitted changes

1 releases

Succeeded {8:55}~/splunk-firehose-nozzle-release:master ✗ ➭

Thx Bry

[chenziliang]

May I ask why in your case, deployed via bosh is requied?

[brybinary]

hi chenziliang

sure - the current build process for the clustered splunk estate I work with (index clusters, search head clusters etc) is all via ansible code. As a move to BOSH is currently underway, I want to be able to deploy a clustered splunk env using bosh only (possibly with terraform). I am also in engagement with splunk (vendor) as this is still fairly new (via bosh) afaik.

thx Bry