Add Playbook Templates

[janithcooray]

Pretty Surprised that this feature is missing and not even considered, i think it would be extremely useful.

We have task templates, that can run a single playbook with multiple variables. Pretty Good. But what about Playbook templates. It's pretty common to have multiple playbooks with the same roles but with different variables.

For example, we can have the playbook template similar to task template. We can define the playbook yml like this..

# predefined arrays and nested vars
array_templates:
   - example_array:
       field_1: @required @string
       feild_2: @required @string
       nested_field: 
         some_more: @required @string

roles:
 - role-1
 - role-2
vars:
    service_name: @required @string
    service_port: @required @number
    image: @required @string
    some_unneeded_var:  @string
    some_array: @required @array example_array
# .... your required variables on how you made your playbook

Playbook instance 1 from template

#populated
roles:
 - role-1
 - role-2
vars:
    service_name: "test"
    service_port: 31000
    image: "some-registry/something/image"
    some_unneeded_var:  "valid-string"
    some_array: 
       - sample:
              field_1: "test"
              field_2: "test"
              nested_field: 
                 some_more: "test2"
# .... your required variables on how you made your playbook

Then we should make it register as a task template.

WHY?

While a single playbooks can run with multiple vars, some would often have multiple playbooks for the same roles. For example, say we have a role(s) that can provision a new k8 deployments and its requirements such as Volume claims , repository variables , service accounts etc. we would generally prefer to have deployment_1.yml ,deployment_2.yml deployment_n.yml ...

I think such a feature would extend the audience of semaphore.

While I prefer CLI all the way for doing this, my colleagues prefer something visual/UI as they aren't very well versed with CLI. And i'm pretty sure most of semaphore's audience out there would prefer UI.

Thoughts?

[samerbahri98]

this is already implemented, with one playbook, and different environments, and you copy the task (there is a copy button). The environment here is not just environment variables but ansible variables as well.

[janithcooray]

i know that's possible, i've mentioned that and its not covering some use cases. maybe it wasn't clear. I will create a PR for this

[tboerger]

Maybe you should describe in note detail why environments won't work before you sirens the time into an pr.

[janithcooray]

What im trying to say is, the way Semphore is designed to run playbooks is different to the way we run it. having a single playbook means i can no longer see it as structured files in the repository.

what i have is somewhat similar to https://gitlab.archlinux.org/archlinux/infrastructure/

The existing setup is good for running a single playbook in many environments, not many playbooks in single or multiple environments. most of our playbooks only require a single run per environment. The way it is , our playbooks need to be built in a way that it supports semaphore's way of running playbooks. i'm suggesting a method that does not require that.

Semphore to be used as a secondary method of running playbooks, not primary as my team (Devs) is not versed with ansible as i (DevOps/Admin) am. So while i can continue using ansible the traditional way and my team can use Semphore

current director structure (looks like infrastructure as code)

playbooks/
 - provision/  (contains specific roles meant for provisioning )
   - asia/
     - cluster-a/
       - node-1 
       - node-2
       - node-N  
     - cluster-b/
   - hongkong/
   - onsite/
 - production/  (contains specific roles meant for prduction )
   - service_a
 - staging/ (contains specific roles meant for staging )
   - service_a 
 - staging/
 - testing/ (contains specific roles meant for testing )
   -  service_a_1  
   -  service_b_1  
 - tools/ (contains several roles )
   - some_automation

Semphore's adjustments - (now its only manageable by Semphore, very difficult to manage manually )

playbooks/
 cluster-provision
 service_a

my requirement is for the dev team to be able to provision services (new playbooks) using semphore. while i continue traditionally. with having the same playbook in many different environments does not cover that requirement.

[samerbahri98]

If I am not mistaken, you want to run a sequence of playbooks (or maybe the same playbook) on multiple inventories, or environments sequentially?

[janithcooray]

no that is not at all whats needed. it's also not just related to running the playbook, issue is there is no option to generate playbooks.

i cannot run the same playbook in many environments, nor can i use the same playbook to make new services. Each playbook type has its own roles. and i need it to generate a playbook yaml

for example , i have these categories of playbooks

provision_production
 - provisions a set of servers on the same zone ready for production and communication with other clusters
provision_staging
 - provisions a set of servers on the same zone excludes communication with other clusters and production optimization is disabled 

production _service
-  only to be run on production, 
    it contains roles that deploy to over 12 servers across 3 clusters / regions, adds DNS records, adds load_balancer entries, adds health check configs, adds database entries to SQL proxies, adds variables to repository.

staging _service
-  only to be run on staging, 
    it contains roles that deploy to a single cluster, adds DNS records to our internal network, adds database entries to SQL proxies, adds variables to repository.

a copy of production playbook needs to be added according to "production_service" requirements. there are each playbook for each service, some playbooks are cluster specific (for example network configs are different)

production/
 - service1.yml 
 - service2.yml

current solution in Semphore is to have a task template (from 1 playbook) and add variables to go across multiple environments with varibles. with that solution i have to run (taking to account that i have to create many task templates per env )

ansible playbook/path.yml -extra-vars  -i production "nodes=webgroup” -extra-vars "some.otherthing='xyz'" ... # a couple of these

^ this is very difficult, and i can no longer see configuration in files. and will not be able to create a new service from Semphore UI

my current method

# edit /playbook/env/service.yml with what i need to edit
# all configs are in the file
ansible playbook/env/service.yml

^ simple It's pretty much impossible at this point to get it working properly on Semphore. i can run existing playbooks but that is very rarely used.

I want let my devs provision new services on staging and testing environment. so they can create a new ones without waiting for me

I'm certain im not the only 1 facing this issue. and my tower is also pretty basic/

This suggestion will help Semphore expand its audience.

[tboerger]

I think the only way to really get the whole picture would be some proof of concept.