search

StackStorm / ansible-st2

Ansible Roles and Playbooks to deploy StackStorm
https://galaxy.ansible.com/StackStorm/stackstorm/
Apache License 2.0
100 stars 77 forks source link

Split into `st2`, `st2web`, `st2chatops` #45

Open arm4b opened 8 years ago

arm4b commented 8 years ago

For simplicity reasons, at the moment we keep all the required roles (12) to install/configure StackStorm in one single repository ansible-st2, available under StackStorm.stackstorm Ansible Galaxy, see meta/main.yml workaround.

Alternative is to add more flexibility, distribute everything as different Ansible Galaxy roles (?)

Some arguments:

Disadvantages:

cognifloyd commented 6 years ago

How about we do both? One main repo for development and unified use with read-only subsplit repos for each of the roles? We could use something like: https://github.com/dflydev/git-subsplit

cognifloyd commented 6 years ago

I learned about git-subsplit from the Neos community. They use Neos/slicer (which uses git-subsplit) to manage their split repos. I just asked, and they'd like to move to from git-subsplit to splitsh/lite, but the tools create different commit hashes, so they couldn't complete the transition. We should probably start with slitsh/lite to begin with as it is supposed to be much faster.

cognifloyd commented 6 years ago

Right now, I'm mixing the stackstorm roles in a custom playbook that intersperses some of my own roles. I installed StackStorm.stackstorm from galaxy. To use the stackstorm roles individually, however, I have to refer to them as StackStorm.stackstorm/roles/<role>, and if I want to replace one of the stackstorm provided roles, it might still run the stackstorm one, because of meta dependencies. ie: StackStorm.stackstorm/roles/mongodb is not mongodb. Publishing them as individual roles makes it easier to reuse, or override roles selectively.

Current meta dependencies:

Thus, if I override any of the above roles, it will still use the stackstorm one in addition to my role. Moving to galaxy installed roles solves that because roles can always be referenced individually by name instead of using non-standard paths.

cognifloyd commented 6 years ago

Here is how an initial split can be done with splitsh/lite (with newlines for legibility):

time (git clone git@github.com:cognifloydtest/ansible-st2.git ansible-st2.git);
cd ansible-st2.git;
time (
    for role in roles/*; do
        echo; echo $role;
        git checkout master;
        splitsh-lite --progress --prefix $role --target=refs/heads/ansible-role-${role##*/};
        git checkout ansible-role-${role##*/};
        hub create cognifloydtest/ansible-role-${role##*/};
        git remote add ansible-role-${role##*/} git@github.com:cognifloydtest/ansible-role-${role##*/}.git -t master;
        git push ansible-role-${role##*/} -u ansible-role-${role##*/}:master;
    done;
    git checkout master
)

Check out my example repos at https://github.com/cognifloydtest Check out the output from this command in https://gist.github.com/cognifloyd/4f4fbd0251ddb7115c3d9475f7eab507

If a copy of the repository is kept (to keep the cached split info) in some CI or ST2 instance somewhere, the split and push could be repeated to quickly update the split repositories when new commits are made on the master branch. That way, the ansible-st2 repo is the main development repo that includes all the CI checking, PRs, etc. But, read-only split copies of the roles can also be published on galaxy.

This does not deal with updating tags on the read-only repositories.

Also: Note that I used hub to create the repos on the fly.

arm4b commented 6 years ago

@cognifloyd The solution you're suggesting with git split is definitely interesting and worth trying/considering. Thanks for showing 👍 We'll research this approach more when we'll reach the migration point.

As @ytjohn noted https://github.com/StackStorm/ansible-st2/issues/12#issuecomment-313487075, it might be a good idea to rename the roles/paths to StackStorm.st2, StackStorm.st2web etc soon to make the further transition smoother.

cognifloyd commented 6 years ago

I would take a two step approach:

  1. rename the role folders: st2 => StackStorm.st2
  2. split the roles off into their own repos.

I think you want them to be named appropriately in this repo first so that the meta dependencies, the example playbook, and all of the CI infra uses roles named StackStorm.*.

cognifloyd commented 5 years ago

It's not supported in any released version of ansible yet, but ansible/mazer (like ansible-galaxy, used to install galaxy roles) will allow installing and addressing multi-role repos. (see also https://github.com/ansible/ansible/pull/42867)

So, that would mean I could include the roles as

peschmae commented 5 years ago

Is there any work happening in this regard right now?

I'd be happy to help split up this repo into a role per service, to make it easier to use in an existing ansible environment.

arm4b commented 5 years ago

@peschmae Sorry, we're not working on this change and have no immediate plans about it. There are some good points, but also significant disadvantages in a splitting approach which are stopping us to do this breaking change.

As a workaround , you can install the "main" StackStorm.stackstorm role from Galaxy (https://galaxy.ansible.com/StackStorm/stackstorm/) and use specific sub-roles in your plays. Here is how to help Ansible find it: https://github.com/StackStorm/ansible-st2/blob/master/ansible.cfg.galaxy via custom ansible.cfg.

peschmae commented 5 years ago

Sadly, this approach won't work for me, since we already use roles_path differently in our setup, and it can only be set once. Additionally the nginx/mongodb/rabbitmq/postgres roles might interfere with the roles we already use.

What are the disadvantages, beside needing to set some variable more than once (or cross reference them) in group/host vars, that you are struggling with? I'll happily help you approach and solves these.

In my experience, it should be enough to set a proper requirements.yml which installs the other roles. In this case here, it might make sense to have dependencies between st2* and the st2 roles, and leverage the ansible-galaxy for dependency management.

arm4b commented 5 years ago

See https://github.com/StackStorm/ansible-st2/issues/45#issue-153459530 for the list of pros/cons. The major concern is support & maintenance & visibility while splitting it to 12 or so separated role-repos.

Yes, adding a namespace like StackStorm.mongodb, StackStorm.st2chatops would be a good next step as a transition which will also resolve name interference issues.

That we can do. And if we decide to split in future it would be a good migration point.

cognifloyd commented 5 years ago

If namespacing is ok (desirable? helpful?), then I can resurrect this PR: https://github.com/StackStorm/ansible-st2/pull/173 But I'll only do that if someone at StackStorm (@armab? @LindsayHill?) wants to take the time and review it soon before it gets out of sync with master again.

arm4b commented 5 years ago

@cognifloyd Let's do it :+1: I think it's a good timing.

peschmae commented 5 years ago

Prefixing the roles, would already be a good step, since it would allow to install them locally using mazer, but until ansible/ansible#42867 is merged, there still wont be a way to install the roles in Ansible Tower/AWX, which is our go to.

If you need help with testing the prefixed roles, I'm happy to help.

arm4b commented 5 years ago

@cognifloyd @peschmae Thanks everyone, just released v1.1.0 role version with StackStorm galaxy org namespace/prefix (https://github.com/StackStorm/ansible-st2/pull/173).

gardar commented 2 years ago

Now that Ansible supports collections (for a while), perhaps it would be a good idea to publish the roles as a ansible collection? Rather than a role with sub-roles.

gardar commented 2 years ago

I just came across https://github.com/cognifloyd/st2_installer.ansible which is the approach I was suggesting in my previous comment.
@cognifloyd Do you at some point plan to replace this role with your collection, or is it going to stay as separate projects?

cognifloyd commented 2 years ago

My goals for that collection are somewhat incompatible with the views of other maintainers here, which is why I forked the ansible-based installer.

One of my goals was to use external collections for 3rd party deps like mongo and rabbitmq. To that end I was contributing to community.mongodb and community.rabbitmq. I'm now one of the maintainers for each of those collections, so I can more directly facilitate improvements we might need.

I have been spending time on several other things in the StackStorm community and I've had other priorities at work, so I haven't been able to do as much as I'd like on the ansible collection. I'd be happy for any help. If enough people use and prefer to use that collection over the roles in this repo, then maybe we can discuss making it official in some future TSC meeting. I'm happy to provide commit access if you'd like to contribute to that collection (I would move it from cognifloyd/ to st2sandbox/ in that case - st2sandbox is also not an official part of the StackStorm project).

gardar commented 2 years ago

Thanks for clarifying!

I certainly understand both views, to depend on 3rd party roles and also to have it included. Although in my opinion it doesn't make much sense to have stackstorm specific mongodb and rabbitmq roles, it would make more sense to have the bare minimum rabbitmq/mongodb/etc tasks included in the st2 roles where needed (st2repo, st2, st2web). Then you could have a variable where you can disable those tasks if you plan to use 3rd party roles for more fine grained control over rabbitmq/mongodb/etc.

I'm giving both your collection and the roles in this repo a spin and I'll contribute if I see some opportunities to improvise.