Update the playbooks to v8.12.2

Jimmy-Newtron commented 8 months ago

Buongiorno Lorenzo,

I am trying to use this wonderful project to install a new bare metal Elastic cluster. I am a little crazy and I tried to install the version 8.12.2

At first I encountered some issues I could solve with some vars.yml tuning. Now I see that some setups are getting misaligned with the Elastic recommendations.

Examples:

Kibana is not installable with SSL and https
the Kibana user is now deprecated in favor of kybana_system
Beats is not trivial to install and make it work on my setup
the passwords and users for the additional modules are not on specific names, but always use the same elasticsearch_username variable

Jimmy-Newtron commented 8 months ago

And I forgot

It would be super if I can run a cleanup role that removes the installed files/folders and services

SpirosKBetsson commented 7 months ago

To add another issue with 8.12.2: The "Beats" (all 3 of them) for ARM processors, have their last part renamed to "arm64" from "aarch64" (it remains aarch64 in Kibana and Elastic, though).

Locally, I fixed it easily by hardcoding it in the collection's roles, so I could run it and they installed just fine.

garutilorenzo commented 7 months ago

Hello @Jimmy-Newtron thank you for reporting this. Can you give me some more information about this two points?

Kibana is not installable with SSL and https
the Kibana user is now deprecated in favor of kybana_system

For the second point I think I'm currently using kybana_system user https://github.com/garutilorenzo/ansible-collection-elk/blob/3c13ed73d8513296e9da64399dfade65210d81f0/roles/elasticsearch/defaults/main.yml#L43

Can you also give me some more information about the third point? (Beats is not trivial to install and make it work on my setup)

garutilorenzo commented 7 months ago

Hello @Jimmy-Newtron, I've just tested the collection with the version 8.12.2, no problem found.

kibana-8 12 2

Can you please add more information (points 1,2)?

garutilorenzo commented 7 months ago

@Jimmy-Newtron can you please also add more information about the third point? (Beats is not trivial to install and make it work on my setup)

bm-fj commented 7 months ago

@garutilorenzo I would like to thank you for the great effort, the playbook is highly customizable and works like a charm.

I have just one question related to this issue:

I did enable the flag kibana_https_enabled but it did not affect the final deployment. Kibana is only reachable under the HTTP protocol

Off-Current Topic Question (let me know if I should open a separate issue):

Could you please tell me how I can adapt the playbook in a way to have: 1 node with the role master only and 2 other nodes with master and data roles? Enabling the flag elasticsearch_master_is_also_data_node would make the master node a data node. Disabling it would allow me to have a master node as desired but the rest would have only data roles.

Many thanks!

garutilorenzo commented 7 months ago

Hello @bm-fj, for the question 1, you can achieve this by playing with Ansible inventory and host_vars.

Example inventory:

[elasticsearch_master]
elk-node-0 ansible_host=192.168.x.x
elk-node-1 ansible_host=192.168.x.x
elk-node-2 ansible_host=192.168.x.x

then in host_varsdiectory you will have two yml files named with the hostname of your servers (eg. elk-node-1.yml) ad inside this files set the variable elasticsearch_master_is_also_data_node to yes. I think this will do the trick.

For the question 2, yes in this collection Kibana is never exposed and configured via https. In general and for best practise Kibana (and also elasticsearch) are never exposed directly to the internet (or to a local network). Usually a web server (nginx/apache etc) is put in front of this services. Is possible to integrate this funcion on this collection and get two options:

expose kibana directlry to the internet (or LAN) via https
setup a web server in front of all the services

bm-fj commented 7 months ago

Hi @garutilorenzo,

Thank you for your prompt response.

I believe playing with the variable elasticsearch_master_is_also_data_node does not fit my use case.

I have the following setup:

[elasticsearch_master]
elk-ubuntu-0 ansible_host=192.168.25.x

[elasticsearch_data]
elk-ubuntu-1 ansible_host=192.168.25.x
elk-ubuntu-2 ansible_host=192.168.25.x
elk-ubuntu-3 ansible_host=192.168.25.x

In my case, I would like to have the data nodes: elk-ubuntu-1, elk-ubuntu-2, elk-ubuntu-3 to have the role master as well

and the node master elk-ubuntu-0 to be master only.

Regarding the second question, exposing Kibana directly via HTTPS, would be very helpful. I guess this will be implemented in the future as a feature request. Please provide any guidance on how to do it.

And could you please then clarify more about what the flag: kibana_https_enabled does? Which traffic does it encrypt?

garutilorenzo commented 7 months ago

Hello @bm-fj, I confirm my previous message. With the _hostvars trick you can have 3 master nodes and only two data nodes. This the setup I've tested:

[elasticsearch_master]
elk-node-0 ansible_host=192.168.25.110
elk-node-1 ansible_host=192.168.25.111
elk-node-2 ansible_host=192.168.25.112

[elasticsearch_data]
# elk-node-3 ansible_host=192.168.25.113
# elk-node-4 ansible_host=192.168.25.114
# elk-node-5 ansible_host=192.168.25.115

[elasticsearch:children]
elasticsearch_master
elasticsearch_data

[elasticsearch_ca]
elk-node-0 ansible_host=192.168.25.110

[kibana]
kibana-node-0 ansible_host=192.168.25.110
kibana-node-1 ansible_host=192.168.25.111

my folder structure:

-rw-rw-r--  1 lorenzo lorenzo   65 Mar 13 11:33 ansible.cfg
drwxrwxr-x  2 lorenzo lorenzo 4096 Mar 13 12:47 assets
drwxrwxr-x  2 lorenzo lorenzo 4096 Apr  3 09:45 certs
drwxrwxr-x  3 lorenzo lorenzo 4096 Mar 13 11:32 collections
-rw-rw-r--  1 lorenzo lorenzo  849 Apr  3 14:18 hosts.ini
drwxrwxr-x  2 lorenzo lorenzo 4096 Apr  3 14:17 host_vars
-rw-rw-r--  1 lorenzo lorenzo  139 Mar 13 11:11 Pipfile
-rw-rw-r--  1 lorenzo lorenzo  819 Apr  3 14:29 site.yml
-rw-rw-r--  1 lorenzo lorenzo  647 Apr  3 14:28 vars.yml

indide the _hostvars folder I have two files:

lorenzo@ubuntu-virtual:~/workspaces-local/ansible-test$ ll host_vars/
total 16
drwxrwxr-x 2 lorenzo lorenzo 4096 Apr  3 14:17 ./
drwxrwxr-x 6 lorenzo lorenzo 4096 Apr  3 14:17 ../
-rw-rw-r-- 1 lorenzo lorenzo   43 Apr  3 14:18 elk-node-1.yml
-rw-rw-r-- 1 lorenzo lorenzo   43 Apr  3 14:18 elk-node-2.yml

the content of the file is:

elasticsearch_master_is_also_data_node: yes

If i query elasticsearch on _GET /nodes/data:true api this is the result:

{
  "_nodes": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
}

and If i query elasticsearch on _GET /nodes/master:true api this is the result:

{
  "_nodes": {
    "total": 3,
    "successful": 3,
    "failed": 0
  },
}

garutilorenzo commented 7 months ago

I've extended on #10 the function of _kibana_httpsenabled Now when _kibana_httpsenabled is set to yes kibana is exposed via https and not http (Eg. https://192.168.25.101:5061).

garutilorenzo commented 7 months ago

Closing this issue with #10 merged.

garutilorenzo / ansible-collection-elk

Update the playbooks to v8.12.2 #9