search

upcmd / up

UP - Ultimate Provisioner CLI
https://upcmd.netlify.app/
MIT License
50 stars 4 forks source link
ansible ant automation automation-framework build-tool cicd cli cloud-environments configuration-management devops drone-ci flexible gocd golang gradle jenkins make shell task terraform

readme

UPcmd Docs Telegram Channel

The Ultimate Provisioner: the modern configuration management, build and automation tool

No worries of replacing anything you already setup, UPcmd does not dictate and work exclusively with other tools, rather it incorporates and collaborates with others, but it is capable to be a framework in case you do need it

How does it look like?

You could use a quick terminal GUI to select the task to execute Interactive GUI in terminal

All the project build, test, regression tests, the documentation site generation, publishing the tagged release and latest rolling release are using Up tasks

This is how publishing latest bleeding edge release look like when you run UP task

up ngo Publish_latest

UPcmd - The Ultimate Provisioner

UP is designed and implemented to shine as a modern tool for below:

It is a build tool like Ansible, Make, Rake, Ant, Gradle, Puppet, Taskfile etc, but it is little smarter to try to make things a little easier

The goal of UP is to provide a quick (I'd say the quickest) solution to enable continuous integration and continuous deployment (CI/CD). It is simple to use and yet powerful to address many common challenges nowadays devop teams facing in the Cloud environment.

It is designed with mindful consideration of collaboration with automation in Kubernetes, helm charts, api call.

It follows best practices integrating with common CI/CD tools, such as GOCD, Jenkins, Drone, Gitlab CI and so on and is a good company of all types of CLI tools.

It brings a fun DSL programming interface, a way of modeling and engineering into CLI. It enables OO design and rapid Test Driven Development (TDD) and shorter software delivery cycle.

tasks:
  -
    name: task
    task:
      -
        func: shell
        do:
          - echo "hello, world"

Below shows:

To deploy, simply specify a instanceid to be associated with an environment, eg: dev, staging or prod

up ngo -i staging

The config: 


scopes:
- name: global
  vars:
    db_driver: postgres
    port: 5432

- name: nonprod
  members:
  - dev
  - staging
  vars:
    db_host: nonpord_database.test.host
    db_user: test_db_user
    db_password: could_be_encrypted_using_upcmd_too
  dvars:
    - name: db_password
      value: '6HmsmiJIW1PfIXcF4WwOKOMDiL7PstgfKs2aRFajrwY='

- name: prod
  members: [prod]
  vars:
    host_alias: prod

- name: dev
  vars:
    host_alias: dev

- name: staging
  vars:
    host_alias: staging

- name: prod
  vars:
    host_alias: prod
    db_host: pord_database.proddb.host
    db_user: prod_db_user
  dvars:
    - name: db_password
      value: 'prod_encrypte_aes'

dvars:
  - name: db_hostname
    value: '{{.host_alias}}.myapp.com'
  - name: db_url
    value: 'jdbc:{{.db_driver}}://{{.db_hostname}}:{{.db_port}}/test?user={{.db_user}}&password={{.db_password}}&ssl=true'

tasks:
  -
    name: Main
    desc: deploy my web app stack
    task:
      -
        func: call
        do:
          - deploy_database
          - deploy_web

      -
        func: shell
        do:
          - systemctl start my_database 
          - systemctl start my_web_server

  -
    name: deploy_web
    task:
      -
        func: shell
        do:
          - deploy myweb_server

  -
    name: deploy_database
    task:
      -
        func: shell
        do:
          - deploy mydatabase

With the evolving of the up.yml file, you could externalize the configuration to individual files or make them as module to be reused or shared. Please check out the doc for more details.

High level design

At high level, UPcmd process flows like below:

high level design

Possible applications

UPcmd is a generic automation tool, given your automation solution being backed by Unix Shell. You do not need Shell executable though, as it has default GOSH builtin just in case you will need one to fall back to.

There could be application as below, but not limit to:

Installation

There are 32 different distro for different combination of OS and Arch type, check them out: release

Generic Installation

  1. Download the binary for your platform from the
  2. Rename it to up, or up.exe in windows
  3. Move it to be under your one of your env PATH

Auto Latest Installation (recommended)

Always try to use the latest unless you have CI/CD pipeline to progressively to promote to production, then use tagged version

install_latest(){
if [ "$1" == "" ];then
    echo "syntax exaple: install_latest darwin | linux | windows"
else
    os=$1
    curl -s https://api.github.com/repos/upcmd/up/releases \
        |grep ${os}_amd64_latest \
        |grep download \
        |head -n 1 \
        |awk '{print $2}' \
        |xargs -I % curl -L % -o up \
        && chmod +x up
fi
}
  1. install for mac: 
install_latest darwin
  1. install for linux: 
install_latest linux
  1. install for windows: 
install_latest windows

Install from source

Ensure you use go 1.14 (prefered)

go get -v github.com/upcmd/up/app/up

The up CLI command will be installed to: $HOME/go/bin, make sure you have this in your PATH

Use up cli command in docker

This will map your current working directory as /workspace directory inside of docker container:

docker run -it --rm  -v `pwd`:/workspace docker.pkg.github.com/upcmd/up/upcli:latest /bin/sh

Or you can source this from the funcs.rc

. ./funcs.rc
run_upcli_docker

In the container:

cd /workspace
up ngo

A little taste of UPcmd

Below is a simple greeting example, and a list, inspect and execution view of the task.

This shows:


Vars:
  greet_to: Tom
  weather: sunny

tasks:
  -
    name: task
    desc: main task of hello world demo of UPcmd
    task:
      -
        func: call
        desc: greet to Tom
        do:
          - greet

      -
        func: call
        desc: greet to Grace
        vars:
          greet_to: Grace
        do:
          - greet

      -
        func: cmd
        desc: do  you get the idea?
        do:
          - name: print
            cmd: |
              Have you got a little taste of using the UPcmd?

      -
        func: call
        desc: greet to a team
        vars:
          team:
            - Jason
            - Connie
          weather: stormy
        loop: team
        do:
          - sayhi

  -
    name: greet
    desc: greet to some one
    task:
      -
        func: shell
        desc: say hello
        do:
          - echo "Hello, {{.greet_to}}"

      -
        func: cmd
        desc: talk about weather
        do:
          - name: print
            cmd: 'It is {{.weather}}'

      -
        func: cmd
        desc: ice break
        do:
          - name: print
            cmd: 'What a great day!'
        if: '{{eq .weather "sunny"}}'
        else:
          -
            func: cmd
            do:
              - name: print
                cmd: 'What a bad day!!'

  -
    name: sayhi
    desc: say hi to some one
    task:
      -
        func: cmd
        desc: say hi to someone
        do:
          - name: print
            cmd: 'Hi {{.loopitem}}, how are you?'

      -
        func: call
        desc: greet to the team member
        dvars:
          - name: greet_to
            value: '{{.loopitem}}'
        do:
          - greet

A little taste

Demo

It demos:

Check it out yourself: source and try to have fun to run though the examples by yourself

demo

Why yet another build tool

With all above considerations, UP is designed to be a generic, tiny footprint (zero depedency), effective automation tool in a cloud environment.

Features

  1. Drop in replacement for Makefile, but way more powerful. It uses a composition model rather than dependency model for flexibility/composibility
  2. Implemented in golang, so no dependency hell, no maintenance of runtime and ensure the version consistency across multiple/many execution contexts
  3. Use scopes to manage group of execution context, the variables associated with the scope. Fine grained scoping model to support variable auto overriding/merging. Similar to Ansible global/group vars, host vars, but more powerful to support leaf level objects auto merging
  4. Use dvar - dynamic var, a special design to achieve many incredible features, for example:
    • manage security: encrypt/decrypt, like ansible-vault, builtin
    • dynamics on dynamics: it allows you to specify how many layers of expansion you'd like to dynamically render a variable
    • builtin templating capability
    • use golang template, supporting all(220+) (builtin|sprig|gtf) funcs/pipeline so that your configuration could be well controlled in template using objects
    • auto message transformation between yaml|json result to object used internally
    • conform the hierarchical scoping model for var merging to leaf level
    • manage setup/read env vars in the same scoping model so that you could have seamless integration with minimal exposed demanding ENV vars from CD/CI tool
    • auto validation for mandatory vars
  5. Color print and adjustable verbose level
  6. Flexible programming model to allow you to separate implementation with interface so that common code could be reused via call func Allow empty skeleton to be laid for testing driving process or guide as seudo code, but fill in the details and implementation gradually
  7. Flow control:
    • ignoreError
    • dry run
    • if condition support
    • loop support to iterate through a list of items
    • mult-layered loop, break and until condition
    • block and embedded block of code for execution
    • finally/rescue to ensure cleanup
  8. Flexible configuration style to load dvar, scope, flow from external yaml so that the programming code will be a little cleaner and organised. Your code could evolve starting from simple, then externalize detailed implementation to files.
  9. Support the unlimited yml object, so yml config in var is text and it is also object.It could be merged in scopes automatically, it could be processed using go template
  10. Battery included for common builtin commands: print, reg, deReg, template, readFile, writeFile
  11. Builtin yml liter and object query, modification
  12. Call func is really shining powerful design to be used:
    • Compose the sequential execution of block of code
    • Use the stack design, so it segregates all its local vars so that the vars used in its implementation will not pollute the caller's vars
    • It serves like a interface to separates the goal and implementation and makes the code is reusable
  13. The shell execution binary is configurable, builtin support for GOSH (mvdan.cc/sh). This means that you do not need native shell/bash/zsh installed in order for task execution, you can run task from windows machine.
  14. It provides a module mechanism to encourage community to share modular code so that you do not need to reinvent the wheel to develop the same function again
  15. Use execution profile to simplify your Ci/CD pipeline integration with zero arguments in your command line but all managed in a nice configurable way.
  16. Use virtualEnv cmd to snapshot an env context, unset all env vars to create a pure clean execution env context, or restore to a point of time of the execution

Real Examples

Both UPcmd project build and the docs entire site build use the UPcmd itself

Project release for UPcmd source
up ngo publish
Documentation doc site

build of the entire doc site using one build task:

A web scripting example how?

Testing

There are over 230+ test cases, every release come with a full passed regression test of all cases defined, source

These test cases are not only about the tests, they are the usage examples with documentation self explaned

License

This project is under MPL-2.0 License