garutilorenzo / ansible-collection-elk

The Elastic stack (ELK) powered by Ansible
https://garutilorenzo.github.io/ansible-collection-elk/
GNU General Public License v3.0
50 stars 23 forks source link

Update the playbooks to v8.12.2 #9

Closed Jimmy-Newtron closed 7 months ago

Jimmy-Newtron commented 8 months ago

Buongiorno Lorenzo,

I am trying to use this wonderful project to install a new bare metal Elastic cluster. I am a little crazy and I tried to install the version 8.12.2

At first I encountered some issues I could solve with some vars.yml tuning. Now I see that some setups are getting misaligned with the Elastic recommendations.

Examples:

Jimmy-Newtron commented 8 months ago

And I forgot

It would be super if I can run a cleanup role that removes the installed files/folders and services

SpirosKBetsson commented 7 months ago

To add another issue with 8.12.2: The "Beats" (all 3 of them) for ARM processors, have their last part renamed to "arm64" from "aarch64" (it remains aarch64 in Kibana and Elastic, though).

Locally, I fixed it easily by hardcoding it in the collection's roles, so I could run it and they installed just fine.

garutilorenzo commented 7 months ago

Hello @Jimmy-Newtron thank you for reporting this. Can you give me some more information about this two points?

For the second point I think I'm currently using kybana_system user https://github.com/garutilorenzo/ansible-collection-elk/blob/3c13ed73d8513296e9da64399dfade65210d81f0/roles/elasticsearch/defaults/main.yml#L43

Can you also give me some more information about the third point? (Beats is not trivial to install and make it work on my setup)

garutilorenzo commented 7 months ago

Hello @Jimmy-Newtron, I've just tested the collection with the version 8.12.2, no problem found.

kibana-8 12 2

Can you please add more information (points 1,2)?

garutilorenzo commented 7 months ago

@Jimmy-Newtron can you please also add more information about the third point? (Beats is not trivial to install and make it work on my setup)

bm-fj commented 7 months ago

@garutilorenzo I would like to thank you for the great effort, the playbook is highly customizable and works like a charm.

I have just one question related to this issue:

Off-Current Topic Question (let me know if I should open a separate issue):

Many thanks!

garutilorenzo commented 7 months ago

Hello @bm-fj, for the question 1, you can achieve this by playing with Ansible inventory and host_vars.

Example inventory:

[elasticsearch_master]
elk-node-0 ansible_host=192.168.x.x
elk-node-1 ansible_host=192.168.x.x
elk-node-2 ansible_host=192.168.x.x

then in host_varsdiectory you will have two yml files named with the hostname of your servers (eg. elk-node-1.yml) ad inside this files set the variable elasticsearch_master_is_also_data_node to yes. I think this will do the trick.

For the question 2, yes in this collection Kibana is never exposed and configured via https. In general and for best practise Kibana (and also elasticsearch) are never exposed directly to the internet (or to a local network). Usually a web server (nginx/apache etc) is put in front of this services. Is possible to integrate this funcion on this collection and get two options:

bm-fj commented 7 months ago

Hi @garutilorenzo,

Thank you for your prompt response.

I believe playing with the variable elasticsearch_master_is_also_data_node does not fit my use case.

I have the following setup:

[elasticsearch_master]
elk-ubuntu-0 ansible_host=192.168.25.x

[elasticsearch_data]
elk-ubuntu-1 ansible_host=192.168.25.x
elk-ubuntu-2 ansible_host=192.168.25.x
elk-ubuntu-3 ansible_host=192.168.25.x

In my case, I would like to have the data nodes: elk-ubuntu-1, elk-ubuntu-2, elk-ubuntu-3 to have the role master as well

and the node master elk-ubuntu-0 to be master only.

Regarding the second question, exposing Kibana directly via HTTPS, would be very helpful. I guess this will be implemented in the future as a feature request. Please provide any guidance on how to do it.

And could you please then clarify more about what the flag: kibana_https_enabled does? Which traffic does it encrypt?

garutilorenzo commented 7 months ago

Hello @bm-fj, I confirm my previous message. With the _hostvars trick you can have 3 master nodes and only two data nodes. This the setup I've tested:

[elasticsearch_master]
elk-node-0 ansible_host=192.168.25.110
elk-node-1 ansible_host=192.168.25.111
elk-node-2 ansible_host=192.168.25.112

[elasticsearch_data]
# elk-node-3 ansible_host=192.168.25.113
# elk-node-4 ansible_host=192.168.25.114
# elk-node-5 ansible_host=192.168.25.115

[elasticsearch:children]
elasticsearch_master
elasticsearch_data

[elasticsearch_ca]
elk-node-0 ansible_host=192.168.25.110

[kibana]
kibana-node-0 ansible_host=192.168.25.110
kibana-node-1 ansible_host=192.168.25.111

my folder structure:

-rw-rw-r--  1 lorenzo lorenzo   65 Mar 13 11:33 ansible.cfg
drwxrwxr-x  2 lorenzo lorenzo 4096 Mar 13 12:47 assets
drwxrwxr-x  2 lorenzo lorenzo 4096 Apr  3 09:45 certs
drwxrwxr-x  3 lorenzo lorenzo 4096 Mar 13 11:32 collections
-rw-rw-r--  1 lorenzo lorenzo  849 Apr  3 14:18 hosts.ini
drwxrwxr-x  2 lorenzo lorenzo 4096 Apr  3 14:17 host_vars
-rw-rw-r--  1 lorenzo lorenzo  139 Mar 13 11:11 Pipfile
-rw-rw-r--  1 lorenzo lorenzo  819 Apr  3 14:29 site.yml
-rw-rw-r--  1 lorenzo lorenzo  647 Apr  3 14:28 vars.yml

indide the _hostvars folder I have two files:

lorenzo@ubuntu-virtual:~/workspaces-local/ansible-test$ ll host_vars/
total 16
drwxrwxr-x 2 lorenzo lorenzo 4096 Apr  3 14:17 ./
drwxrwxr-x 6 lorenzo lorenzo 4096 Apr  3 14:17 ../
-rw-rw-r-- 1 lorenzo lorenzo   43 Apr  3 14:18 elk-node-1.yml
-rw-rw-r-- 1 lorenzo lorenzo   43 Apr  3 14:18 elk-node-2.yml

the content of the file is:

elasticsearch_master_is_also_data_node: yes

If i query elasticsearch on _GET /nodes/data:true api this is the result:

{
  "_nodes": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
}

and If i query elasticsearch on _GET /nodes/master:true api this is the result:

{
  "_nodes": {
    "total": 3,
    "successful": 3,
    "failed": 0
  },
}
garutilorenzo commented 7 months ago

I've extended on #10 the function of _kibana_httpsenabled Now when _kibana_httpsenabled is set to yes kibana is exposed via https and not http (Eg. https://192.168.25.101:5061).

garutilorenzo commented 7 months ago

Closing this issue with #10 merged.