Documentation for getting off the ground is missing, leading to multiple dead ends

[apurtell]

I would like to use the apache_hbase topology checked in on HBase master branch. There is no documentation for going from those files to something that will work.

I did try

• Copying the apache_hbase topology files into clusterdock/clusterdock/topologies/. This does not work "build_cluster: error: argument topology: invalid choice: 'apache_hbase' (choose from 'nodebase', 'cdh')"
• Setting CLUSTERDOCK_TOPOLOGY_IMAGE to the path clusterdock/clusterdock/topologies/apache_hbase. Yeah, this wasn't going to work because it expects a Docker repository ID, but it seemed the only other obvious thing to try.

No docs available for making a local topology image, so that's the end of the road presently.

[dimaspivak]

Hey Andy,

If you check out the clusterdock repo and copy the apache_hbase folder into the topologies folder and then do docker build from the repo root, it will build that image. Then, you can specify the image name of your custom-rolled clusterdock image with CLUSTERDOCK_IMAGE when invoking clusterdock_run. If you name your own version cloudera/clusterdock, it may will be overwritten since we default to pulling the image normally (can be disabled with CLUSTERDOCK_PULL=false). I agree I need to get the docs away from their currently dismal state.

[apurtell]

Tried this:

1. docker build -t clusterdock:apache_hbase_topology apache_hbase_topology
2. Add 'CLUSTERDOCK_TOPOLOGY_IMAGE=clusterdock:apache_hbase_topology' to command invocations.

build_cluster: error: argument topology: invalid choice: 'apache_hbase' (choose from 'nodebase', 'cdh')

And this:

1. cd apache_hbase_topology
2. docker build -t clusterdock:apache_hbase_topology .
3. Add 'CLUSTERDOCK_TOPOLOGY_IMAGE=clusterdock:apache_hbase_topology' to command invocations.

build_cluster: error: argument topology: invalid choice: 'apache_hbase' (choose from 'nodebase', 'cdh')

[apurtell]

Here's what worked for me

1. cp -a apache_hbase_topology clusterdock/clusterdock/topologies/apache_hbase
2. docker build -t clusterdock:latest ./clusterdock

Proceed with new local clusterdock image 'clusterdock:latest'

Now I'm stuck with

"build_cluster: error: unrecognized arguments: --nodebase-image=centos6.6_nodebase"

[apurtell]

Dropping the old '--nodebase-image' argument to build_cluster gets me further along, now

INFO:clusterdock.topologies.apache_hbase.actions:Building image docker.io/apache/clusterdock:centos6.6_java-8u91_hadoop-2.7.3-sfdc-1.0.0_hbase-0.98.22-sfdc-1.0.0... Sending build context to Docker daemon 449.2 MB Step 1 : FROM docker.io/apache/clusterdock:centos6.6_nodebase Pulling repository docker.io/apache/clusterdock Error: image apache/clusterdock not found

Fatal error: local() encountered an error (return code 1) while executing 'docker build -t docker.io/apache/clusterdock:centos6.6_java-8u91_hadoop-2.7.3-sfdc-1.0.0_hbase-0.98.22-sfdc-1.0.0 --no-cache /tmp/clusterdock/afc72cf9-0fbd-482f-ad91-88ceb396e551'

Aborting.

My invocation was

CLUSTERDOCK_IMAGE=clusterdock:latest CLUSTERDOCK_PULL=false clusterdock_run \
./bin/build_cluster apache_hbase \
  --hbase-version=foo \
  --hbase-tarball bar \
  --hadoop-version=baz \
  --hadoop-tarball blah

[apurtell]

I have tried

CLUSTERDOCK_IMAGE=clusterdock:latest CLUSTERDOCK_PULL=false clusterdock_run \ ./bin/build_image centos6.6_nodebase

followed by

CLUSTERDOCK_IMAGE=clusterdock:latest CLUSTERDOCK_PULL=false clusterdock_run \ ./bin/build_cluster -o centos6.6_nodebase apache_hbase ...

CLUSTERDOCK_IMAGE=clusterdock:latest CLUSTERDOCK_PULL=false clusterdock_run \ ./bin/build_cluster -o centos6.6 apache_hbase ...

Step 1 : FROM docker.io/apache/clusterdock:centos6.6_nodebase Pulling repository docker.io/apache/clusterdock Error: image apache/clusterdock not found

Fatal error: local() encountered an error (return code 1) while executing 'docker build -t docker.io/apache/clusterdock:centos6.6_java-8u91_hadoop-2.7.3-sfdc-1.0.0_hbase-0.98.22-sfdc-1.0.0 --no-cache /tmp/clusterdock/384dc3e4-09d7-4b4b-8c19-048b4aa9ef66'

Aborting.

How do I get this to stop trying to pull a nonexistent OS image from docker.io/apache/... ?

[apurtell]

Well, for now I can move forward by changing apache_namespace to "cloudera" in constants.cfg :-)

[dimaspivak]

Sorry, wasn't getting notifications on this issue. So the two parts of naming an image (and deciding where to get base images) are the --registry-url/-r and --namespace arguments to the ./bin/build_cluster (or ./bin/start_cluster) scripts. As it stands, the Apache HBase topology will default to using the apache_namespace if no --namespace is specified. Try this workflow:

1. From the hbase/dev-support/apache_hbase_topology folder, run docker build -t dev/clusterdock:apache_hbase_topology ..
2. Assuming you've sourced the clusterdock.sh script (we should get a copy of this with defaults set for the Apache Bintray account checked in...), build the centos6.6_nodebase image: clusterdock_run ./bin/build_image --namespace=dev centos6.6_nodebase.
3. Build an HBase image: CLUSTERDOCK_TOPOLOGY_IMAGE=dev/clusterdock:apache_hbase_topology clusterdock_run ./bin/build_cluster --namespace=dev apache_hbase ....

I'll try to get on updating the HBase ref guide (and also pushing images of the centos6.6_nodebase and apache_hbase_topology itself into our apache/hbase Bintray) in a few days when I have some more time.