Missing variables for service port and firewall configuration

TOSIT-IO / tdp-collection-extras

Ansible roles to deploy the extra components of TDP

Apache License 2.0

5 stars 13 forks source link

Missing variables for service port and firewall configuration #145

Open KaidoKopli opened 10 months ago

KaidoKopli commented 10 months ago

Installing TDP lib on freshly imaged machine we noticed that all firewall ports were not properly open by the firewalld playbook (ansible_collection/tosit/tdp_extra/playbooks/meta/firewall.yml).

Comparing with the pre TDP lib deployment method we noted the following, ports and port ranges were missing from tdp-cluster.yml

hdfs_nn: [9820] dfs.namenode.rpc-address hdfs_dn: [1004] dfs.datanode.address hive_s2: [10002] hive. server2. webui.port

We suggest that variables are added to the playbook for services listening to these ports as well as variables for the missing port ranges.

New variables for services should be used in the corresponding playbooks instead of hardcoded port values and the firewalld playbook should make sure network flows are open for the services and port ranges.

Is there some plan to add livy variables in the tdp-cluster.yml?

GuillaumeHold commented 9 months ago

This is probably a configuration issue.The properties are defined in tdp-collection as follow :

"dfs.namenode.rpc-address.{{ cluster_name }}.nn1": "{{ groups['hdfs_nn'][0] | tosit.tdp.access_fqdn(hostvars) }}:{{ hdfs_nn_rpc_port }}"
dfs.datanode.address: "0.0.0.0:{{ hdfs_dn_data_port }}"
hive.server2.webui.port: "{{ hive_hiveserver2_webui_port }}"

The configured port are:

hdfs_nn_rpc_port : 8020
hdfs_dn_data_port : 9866
hive_hiveserver2_webui_port : 0