search

vitabaks / postgresql_cluster

PostgreSQL High-Availability Cluster (based on "Patroni" and DCS "etcd" or "consul"). Automating with Ansible.
MIT License
1.29k stars 352 forks source link

Extra + in the {% endfor +%} block causing playbook to fail. #488

Closed guillaumeldc closed 6 months ago

guillaumeldc commented 7 months ago

https://github.com/vitabaks/postgresql_cluster/blob/c030eea4c0b724830c9712a84f3e3d351db9bc78/roles/patroni/templates/patroni.yml.j2#L38C135-L38C136

bboy8012 commented 7 months ago

Oh good, it's not just me. I was just starting to debug it to see where the error was.

vitabaks commented 7 months ago

@guillaumeldc Thanks!

here is a recent commit that added '+' to 'endfor':

https://github.com/vitabaks/postgresql_cluster/commit/de4d4005d7cb6ecd50ae7ab2a4a1c4bfddc41968#diff-6cd5a640c38bf41a44e6115d2ab9d552e4cc4f41fbd78ac2a3ff9d2faee706a4

vitabaks commented 7 months ago

Until I can reproduce the problem with dcs_exists: true, what errors are you getting?

vitabaks commented 7 months ago

On the contrary, without using '+' ({% endfor %}) I got incorrect formatting and hence the Patroni error

etcd3:
  hosts: 10.172.0.20:2379,10.172.0.21:2379,10.172.0.22:2379  username: etcd
  password: etcd-pass

journalctl -u patroni -n 100

Oct 18 11:04:49 pgnode01 systemd[1]: Started Runners to orchestrate a high-availability PostgreSQL - Patroni.
Oct 18 11:04:49 pgnode01 patroni[17608]: Traceback (most recent call last):
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/local/bin/patroni", line 8, in <module>
Oct 18 11:04:49 pgnode01 patroni[17608]:     sys.exit(main())
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/local/lib/python3.10/dist-packages/patroni/__main__.py", line 191, in main
Oct 18 11:04:49 pgnode01 patroni[17608]:     return patroni_main(args.configfile)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/local/lib/python3.10/dist-packages/patroni/__main__.py", line 162, in patroni_main
Oct 18 11:04:49 pgnode01 patroni[17608]:     abstract_main(Patroni, configfile)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/local/lib/python3.10/dist-packages/patroni/daemon.py", line 168, in abstract_main
Oct 18 11:04:49 pgnode01 patroni[17608]:     config = Config(configfile)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 214, in __init__
Oct 18 11:04:49 pgnode01 patroni[17608]:     self._local_configuration = self._load_config_file()
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 263, in _load_config_file
Oct 18 11:04:49 pgnode01 patroni[17608]:     config = self._load_config_path(self._config_file)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/local/lib/python3.10/dist-packages/patroni/config.py", line 255, in _load_config_path
Oct 18 11:04:49 pgnode01 patroni[17608]:     config = yaml.safe_load(f)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 162, in safe_load
Oct 18 11:04:49 pgnode01 patroni[17608]:     return load(stream, SafeLoader)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 114, in load
Oct 18 11:04:49 pgnode01 patroni[17608]:     return loader.get_single_data()
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 49, in get_single_data
Oct 18 11:04:49 pgnode01 patroni[17608]:     node = self.get_single_node()
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/composer.py", line 36, in get_single_node
Oct 18 11:04:49 pgnode01 patroni[17608]:     document = self.compose_document()
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/composer.py", line 55, in compose_document
Oct 18 11:04:49 pgnode01 patroni[17608]:     node = self.compose_node(None, None)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/composer.py", line 84, in compose_node
Oct 18 11:04:49 pgnode01 patroni[17608]:     node = self.compose_mapping_node(anchor)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
Oct 18 11:04:49 pgnode01 patroni[17608]:     item_value = self.compose_node(node, item_key)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/composer.py", line 84, in compose_node
Oct 18 11:04:49 pgnode01 patroni[17608]:     node = self.compose_mapping_node(anchor)
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/composer.py", line 127, in compose_mapping_node
Oct 18 11:04:49 pgnode01 patroni[17608]:     while not self.check_event(MappingEndEvent):
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/parser.py", line 98, in check_event
Oct 18 11:04:49 pgnode01 patroni[17608]:     self.current_event = self.state()
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/parser.py", line 428, in parse_block_mapping_key
Oct 18 11:04:49 pgnode01 patroni[17608]:     if self.check_token(KeyToken):
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 116, in check_token
Oct 18 11:04:49 pgnode01 patroni[17608]:     self.fetch_more_tokens()
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 223, in fetch_more_tokens
Oct 18 11:04:49 pgnode01 patroni[17608]:     return self.fetch_value()
Oct 18 11:04:49 pgnode01 patroni[17608]:   File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 577, in fetch_value
Oct 18 11:04:49 pgnode01 patroni[17608]:     raise ScannerError(None, None,
Oct 18 11:04:49 pgnode01 patroni[17608]: yaml.scanner.ScannerError: mapping values are not allowed here
Oct 18 11:04:49 pgnode01 patroni[17608]:   in "/etc/patroni/patroni.yml", line 18, column 70
Oct 18 11:04:49 pgnode01 systemd[1]: patroni.service: Main process exited, code=exited, status=1/FAILURE
vitabaks commented 7 months ago

And current code ({% endfor +%})

etcd3:
  hosts: 10.172.0.20:2379,10.172.0.21:2379,10.172.0.22:2379
  username: etcd
  password: etcd-pass
root@pgnode01:/# journalctl -u patroni -n 10
Oct 18 11:22:10 pgnode01 patroni[26397]: 2023-10-18 11:22:10,443 INFO: no action. I am (pgnode01), the leader with the lock
Oct 18 11:22:20 pgnode01 patroni[26397]: INFO:patroni.ha:Lock owner: pgnode01; I am pgnode01
Oct 18 11:22:20 pgnode01 patroni[26397]: INFO:patroni.__main__:no action. I am (pgnode01), the leader with the lock
Oct 18 11:22:20 pgnode01 patroni[26397]: 2023-10-18 11:22:20,443 INFO: no action. I am (pgnode01), the leader with the lock
Oct 18 11:22:30 pgnode01 patroni[26397]: INFO:patroni.ha:Lock owner: pgnode01; I am pgnode01
Oct 18 11:22:30 pgnode01 patroni[26397]: INFO:patroni.__main__:no action. I am (pgnode01), the leader with the lock
Oct 18 11:22:30 pgnode01 patroni[26397]: 2023-10-18 11:22:30,447 INFO: no action. I am (pgnode01), the leader with the lock
Oct 18 11:22:40 pgnode01 patroni[26397]: INFO:patroni.ha:Lock owner: pgnode01; I am pgnode01
Oct 18 11:22:40 pgnode01 patroni[26397]: INFO:patroni.__main__:no action. I am (pgnode01), the leader with the lock
Oct 18 11:22:40 pgnode01 patroni[26397]: 2023-10-18 11:22:40,447 INFO: no action. I am (pgnode01), the leader with the lock
postgres@pgnode01:~$ patronictl list
+ Cluster: postgres-cluster -------+-----------+----+-----------+
| Member   | Host        | Role    | State     | TL | Lag in MB |
+----------+-------------+---------+-----------+----+-----------+
| pgnode01 | 10.172.0.20 | Leader  | running   |  2 |           |
| pgnode02 | 10.172.0.21 | Replica | streaming |  2 |         0 |
| pgnode03 | 10.172.0.22 | Replica | streaming |  2 |         0 |
+----------+-------------+---------+-----------+----+-----------+
vitabaks commented 7 months ago

So I'm not sure if this part of the code needs fixing.

Please give more information about your error.

guillaumeldc commented 7 months ago

Hi @vitabaks sorry for the short (no) description in filing the error. Not sure if we are talking about the same thing. In the jinja template, in the patroni role there is definitely a typo line 38 resulting in the following error when executing the playbook:

AnsibleError: template error while templating string: expected token 'end of statement block', got '+'

Unless my knowledges in Jinja templating are already outdated, the "+" doesn't belong there.

This is the code I'm talking about:

{% if not dcs_exists|bool and dcs_type == 'etcd' %}
etcd3:
  hosts: {% for host in groups['etcd_cluster'] %}{{ hostvars[host]['inventory_hostname'] }}:2379{% if not loop.last %},{% endif %}{% endfor %}
{% endif %}
{% if dcs_exists|bool and dcs_type == 'etcd' %}
etcd3:
  hosts: {% for etcd_hosts in patroni_etcd_hosts %}{{etcd_hosts.host}}:{{etcd_hosts.port}}{% if not loop.last %},{% endif %}
### this is not correct below
 {% endfor +%}
guillaumeldc commented 7 months ago

Removing the "+" corrected my issue and the playbook ran smoothly.

vitabaks commented 7 months ago

@guillaumeldc Ok, I removed the '+' and added an empty string before the condition for the variable 'patroni_etcd_username' to avoid formatting issues.

Could you test this?

PR: https://github.com/vitabaks/postgresql_cluster/pull/490