cloudera.exe - Runlevel Management and Utilities for Cloudera Data Platform (CDP)

cloudera.exe is an Ansible collection that offers runlevel management of your Cloudera Data Platform (CDP) Public Cloud and Private Cloud deployments. The collection contains a number of utilities for common scenarios encountered when managing a CDP deployment, including:

• Set up and management of external dependencies, e.g. database, Kerberos, LDAP
• Execution of Common deployment sequences (via importable playbooks)

The collection is unabashedly an opinionated approach of managing your CDP resources - it's resources can be used to set up your CDP infrastructure, configure the host machines, install and configure CDP and its services, and more. The collection interacts across several control planes from CDP Public Cloud and cloud provider endpoints to Cloudera Manager for CDP Private Cloud and Public Cloud Data Hubs. In short, it has opinions about how to get things done. If you are looking for automation resources that only interact with CDP resources - that is, assets that are focused solely on Cloudera software - please look at cloudera.cloud for Public Cloud and cloudera.cluster for Private Cloud and Cloudera Manager.

Core to the collection is the configuration file which many of the collection's roles use as a central "switchboard" for their functions. The collection works hand-in-hand with the cloudera-deploy application to execute definitions which include variations on this configuration; many of the functions in cloudera-deploy have relocated to this collection to streamline its use.

The collection provides playbooks, roles, and plugins for working with CDP deployments. Notably, the playbooks encapsulate typical set up and tear down deployment operations, aka runlevels:

cloudera.exe-powered applications, like cloudera-deploy, import these playbooks to enable these runlevel operations.

The other collection assets - the roles and plugins - are detailed in the API documentation. While these resource can be used separately, most expect the common configuration noted above and a sequence of execution defined within the noted playbooks.

Quickstart

1. Install the collection
2. Install the requirements
3. Use the collection

API

See the API documentation for details for each plugin and role within the collection.

Roadmap

If you want to see what we are working on or have pending, check out:

• the Milestones and active issues to see our current activity,
• the issue backlog to see what work is pending or under consideration, and
• read up on the Ideas we have in mind.

Are we missing something? Let us know by creating a new issue or posting a new idea!

Contribute

For more information on how to get involved with the cloudera.exe Ansible collection, head over to CONTRIBUTING.md.

Installation

To install the cloudera.exe collection, you have several options. Please note that to date, we have not yet published this collection to the public Ansible Galaxy server, so you cannot install it via direct namespace declaration, rather you must specify a Git project and (optionally) branch.

Option #1: Install from GitHub

Create or edit the requirements.yml file in your project with the following:

collections:
  - name: https://github.com/cloudera-labs/cloudera.exe.git
    type: git
    version: main

And then run in your project:

ansible-galaxy collection install -r requirements.yml

You can also install the collection directly:

ansible-galaxy collection install git+https://github.com/cloudera-labs/cloudera.exe.git@main

Option #2: Install the tarball

Periodically, the collection is packaged into a distribution which you can install directly:

ansible-galaxy collection install <collection-tarball>

See Building the Collection for details on creating a local tarball.

Requirements

The cloudera.exe expects ansible-core>=2.10,<2.13.

[!WARNING] The current functionality of the cloudera.cluster dependency does not yet work with Ansible version 2.13 and later.

The collection has the following required dependencies:

You will need to add the following, depending on your target deployment, but all are collectively optional dependencies:

Private Cloud

See the requirements for cloudera-labs/cloudera.cluster for details.

Terraform

If you intend to use Terraform as your infrastructure engine within the cloudera.exe.infra role, then install the following:

AWS

See the AWS Execution Environment configuration in cloudera-labs/cldr-runner for details on setting up the Python and system requirements.

Azure

See the Azure Execution Environment configuration in cloudera-labs/cldr-runner for details on setting up the Python and system requirements.

GCP

See the GCP Execution Environment configuration in cloudera-labs/cldr-runner for details on setting up the Python and system requirements.

The collection also requires the following Python libraries to operate its modules and tasks:

• netaddr

The collection's Python dependencies alone, not the required Python libraries of its collection dependencies, are in requirements.txt.

All collection dependencies, required and optional, can be found in requirements.yml; only the required non-Cloudera dependencies are in galaxy.yml. ansible-galaxy will install only the required non-Cloudera collection dependencies; you will need to add cloudera.cloud, cloudera.cluster, and the optional collection dependencies as needed (see above).

ansible-builder can discover and install all Python dependencies - current collection and dependencies - if you wish to use that application to construct your environment. Otherwise, you will need to read each collection and role dependency and follow its installation instructions.

See the Collection Metadata section for further details on how to install (and manage) collection dependencies.

You may wish to use a virtual environment to manage the Python dependencies.

Using the Collection

This collection is designed to work hand-in-hand with the cloudera-deploy application, which uses the reference playbooks in the playbooks directory to drive the operations of its example definitions.

Once installed, reference the collection in your playbooks and roles.

For example, here we use the cloudera.exe.init_deployment role to read the configuration details and then import the Public Cloud playbooks to set up and provision an Environment and Datalake:

- name: Marshal the variables
  hosts: localhost
  connection: local
  gather_facts: yes
  tasks:
    - name: Read definition variables
      ansible.builtin.include_role:
        name: cloudera.exe.init_deployment
        public: yes
      when: init__completed is undefined
  tags:
    - always

- name: Set up CDP Public Cloud infrastructure (Ansible-based)
  ansible.builtin.import_playbook: cloudera.exe.pbc_infra_setup.yml

- name: Set up CDP Public Cloud (Env and DL example)
  ansible.builtin.import_playbook: cloudera.exe.pbc_setup.yml

[!IMPORTANT] You must run cloudera.exe.init_deployment before calling any of the collection's playbooks. This call must occur within the source project, otherwise Ansible's playbook_dir will change to the collection's installation directory and variable lookups might not work as expected.

Legacy Execution Modes

[!WARNING] These documents and their modes of operation are deprecated in version 2.x. For example, the use of Ansible tags to trigger coarse runlevels have been replaced by explicit playbook execution. However, the "inner" tag structures still remain and might be relevant to some execution modes.

See the execution examples in the Deployment Runlevels document.

For more information on the collection, check out:

• Configuration Guide
• Runlevels Guide
• Architecture and Design Guide

Building the Collection

To create a local collection tarball, run:

ansible-galaxy collection build 

Building the API Documentation

To create a local copy of the API documentation, first make sure the collection is in your ANSIBLE_COLLECTIONS_PATHS. Then run the following:

# change into the /docsbuild directory
cd docsbuild

# install the build requirements (antsibull-docs); you may want to set up a
# dedicated virtual environment
pip install ansible-core https://github.com/cloudera-labs/antsibull-docs/archive/cldr-docsite.tar.gz

# Install the collection's build dependencies
pip install requirements.txt

# Then run the build script
./build.sh

License and Copyright

Copyright 2023, Cloudera, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.