production-ize bdas by separating roles out

[kbroughton]

The current playbook is very easy to set up with only master and worker roles. This means all services - zookeeper, tachyon, mesos, etc are co-located on hosts.

We would like to support microservices / docker by separating roles. We could maintain the simple master/worker facade by wrapping the differentiated hosts file as follows:

[master] zookeeper tachyon mesos_master hdfs_namenode hadoop_job_tracker etc

[worker] zookeeper hadoop_task_traker mesos_worker etc

---------- and have differentiated playbook here-----------------

[zookeeper] host1 host2

or with dynamic tagging

[tag_role_zookeeper]

filled by dynamic inventory ec2.py

[zookeeper:children]

and so on for every differentiated role.

tag_role_worker

---------------------------------------------- end hosts file ---------------------

To do this you would have to

1. change site.yml to use microservice role names in stead of master/worker
2. change variable names like
   • tachyon_master_host: "{{ master_hostname }}"
   • tachyon_master_host: "{{ groups['tachyon'][0] }}" *
   • there may be issues to [0] due to dict order not being stable. Could use jinja | sort possibly.

Would you be interested in a PR for this?

[irifed]

Yes, I agree that current master-workers concept makes this playbook is very limited and I planned to implement something along these lines. Thanks a lot for this effort! Please go ahead and open PR.

[kbroughton]

should have a pr in later today. Part I - allow non-root user, move vars/ to defaults/ to allow ansible over-rides structure Part II - split master/owners into differentiated roles.