search

ansible-collections / community.dns

Ansible modules and plugins for working with DNS
https://galaxy.ansible.com/ui/repo/published/community/dns/
GNU General Public License v3.0
25 stars 17 forks source link

txt_transformation doesn't work with Hetzner API #190

Closed chuegel closed 4 months ago

chuegel commented 4 months ago
SUMMARY

When using txt_transformation: quoted the TXT record cannot be pushed to the Hetzner API

- name: "Update DNS TXT record"                                                                                       
  community.dns.hetzner_dns_record:
    state: present                                                                                                                                                                                                                           
    zone: example.tech                                                                                                                                                                                                                 
    record: text.example.tech                                                                       
    type: TXT                                                                                                                                                                                                                                
    txt_transformation: quoted
    ttl: 7200          
    value: "{{ lookup('file', '{{ playbook_dir }}/../templates/text.json') }}"                                                                                                                                                           
    hetzner_token: "{{ lookup('ansible.builtin.env', 'HETZNER_TOKEN') }}"
ISSUE TYPE
COMPONENT NAME

txt_transformation

ANSIBLE VERSION
ansible [core 2.14.3]
COLLECTION VERSION
Collection    Version
------------- -------
community.dns 2.8.1
CONFIGURATION
ANSIBLE_NOCOWS(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = True
CALLBACKS_ENABLED(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = ['timer']
COLLECTIONS_PATHS(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = ['/home/user/Documents/gitlab/ansible-network-automation/collections']
CONFIG_FILE() = /home/user/Documents/gitlab/ansible-network-automation/ansible.cfg
DEFAULT_GATHERING(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = explicit
DEFAULT_LOAD_CALLBACK_PLUGINS(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = True
DEFAULT_ROLES_PATH(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = ['/home/user/Documents/gitlab/ansible-network-automation/roles']
DEFAULT_STDOUT_CALLBACK(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = yaml
DEPRECATION_WARNINGS(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = False
DISPLAY_SKIPPED_HOSTS(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = True
HOST_KEY_CHECKING(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = False
INTERPRETER_PYTHON(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = /usr/bin/python3
INVENTORY_ENABLED(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = ['host_list', 'script', 'auto', 'yaml', 'ini', 'toml', 'netbox.netbox.nb_inventory']
PERSISTENT_COMMAND_TIMEOUT(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = 180
SYSTEM_WARNINGS(/home/user/Documents/gitlab/ansible-network-automation/ansible.cfg) = True
OS / ENVIRONMENT

Ubuntu 22.04

STEPS TO REPRODUCE

Works:

- name: "Update DNS TXT record"                                                                                       
  community.dns.hetzner_dns_record:
    state: present                                                                                                                                                                                                                           
    zone: example.tech                                                                                                                                                                                                                 
    record: text.example.tech                                                                       
    type: TXT                                                                                                                                                                                                                                
    ttl: 7200          
    value: "{{ lookup('file', '{{ playbook_dir }}/../templates/text.json') }}"                                                                                                                                                           
    hetzner_token: "{{ lookup('ansible.builtin.env', 'HETZNER_TOKEN') }}"

Doesn't work:

- name: "Update DNS TXT record"                                                                                       
  community.dns.hetzner_dns_record:
    state: present                                                                                                                                                                                                                           
    zone: example.tech                                                                                                                                                                                                                 
    record: text.example.tech                                                                       
    type: TXT                                                                                                                                                                                                                                
    txt_transformation: quoted
    ttl: 7200          
    value: "{{ lookup('file', '{{ playbook_dir }}/../templates/text.json') }}"                                                                                                                                                           
    hetzner_token: "{{ lookup('ansible.builtin.env', 'HETZNER_TOKEN') }}"      
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.community.dns.plugins.module_utils.zone_record_api.DNSAPIError: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server"
fatal: [localhost]: FAILED! => changed=false 
  error: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server"
  msg: 'Error: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server"'
EXPECTED RESULTS
ACTUAL RESULTS

see above

felixfontein commented 4 months ago

Could you please provide a value for the record that exhibits this problem? The following task (with another domain/record) works fine for me:

    - community.dns.hetzner_dns_record:
        zone: example.com
        record: text.example.com
        type: TXT
        txt_transformation: quoted
        hetzner_token: "{{ hetzner_token }}"
        value: >-
          "foobar"
        state: present
chuegel commented 4 months ago

Well, the content of the file above looks like this:

{"prefixes": [          
"2.51.168.20/32",
"5.93.200.152/30",
"5.93.210.152/30"
 ]
}

When using without txt_transformation: quoted the TXT record is added but the quotes change to single quotes:

{'prefixes': [          
'2.51.168.20/32',
'5.93.200.152/30',
'5.93.210.152/30'
 ]
}

which brakes json.

felixfontein commented 4 months ago

I'm not sure what you are trying to achieve. If you want to set the contents of the JSON file as a TXT entry, why do you use txt_transformation: quoted? That setting tells the module that you provide a correctly quoted TXT entry, which you apparently do not pass. A JSON file is not a correctly quoted TXT entry. Something like "foo" "bar baz" is for example correctly quoted and results in an entry of value foobar baz. So this is very likely not what you want.

If you use the module without txt_transformation: quoted, note that you fall into the Ansible trap that Ansible sometimes parses JSON when processing it in templates. So you basically pass a dictionary to the module, where AnsibleModule in turn converts that dictionary back to a string by calling Python's str() - which makes it end up with single quotes instead of double quotes.

That is a problem with Ansible that's notoriously hard to handle. The best way is to convert the result of the lookup to string:

    value: "{{ lookup('file', '{{ playbook_dir }}/../templates/text.json') | string }}"
chuegel commented 4 months ago

Yes, my goal is to have a JSON as TXT record. As I mentioned the JSON file is valid but the double quotes get converted to single quotes in the Hetzner API TXT record which ends with a invalid JSON.

I noticed though that this happens in the DNS Console as well when pasting the JSON directly into the record. Also there is weird input when pasting just a simple text string without quotes and spaces:

2.150.168.20/32,5.197.200.152/30,37.206.241.212/29,41.133.102.32/28,41.133.114.144/29,41.133.117.32/29,41.133.124.16/28,41.33.245.240/29,41.33.256.64/29,41.133.195.104/3330,51.177.80.0/24,57.133.168.112/28,58.137.128.96/29,62.174.44.254/31,62.174.245.96/30,74.234.244.64/32,78.133.212.40/29

the record ends with spaces which get quoted: "2.150.168.20/32,5.197.200.152/30,37.206.241.212/29,41.133.102.32/28,41.133.114.144/29,41.133.117.32/29,41.133.124.16/28,41.33.245.240/29,41.33.256.64/29,41.133.195.104/3330,51.177.80.0/24,57.133.168.112/28,58.137.128.96/29,62.174.44.254/31,62.174.245.96/3" "0,74.234.244.64/32,78.133.212.40/29"

value: "{{ lookup('file', '{{ playbook_dir }}/../templates/text.json') | string }}"

resulted in this error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.community.dns.plugins.module_utils.zone_record_api.DNSAPIError: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server" 
fatal: [localhost]: FAILED! => changed=false 
error: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server"
  msg: 'Error: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server"'
felixfontein commented 4 months ago

Did you remove txt_transformation: quoted?

chuegel commented 4 months ago

Did you remove txt_transformation: quoted?

yes

This is the pb:

- name: "Lookup internet prefixes"
  ansible.builtin.set_fact:
    int_prefixes: "{{ query('netbox.netbox.nb_lookup', 'prefixes', api_filter='role=internet') }}"

- name: "Create json file with internet prefixes"
  ansible.builtin.template:
    src: "{{ playbook_dir }}/../templates/text.j2"
    dest: "{{ playbook_dir }}/../templates/text.json"

- name: "Update DNS TXT record"                                                                                       
  community.dns.hetzner_dns_record:
    state: present                                                                                                                                                                                                                           
    zone: example.tech                                                                                                                                                                                                                 
    record: text.example.tech                                                                       
    type: TXT                                                                                                                                                                                                                                
    ttl: 7200          
    value: "{{ lookup('file', '{{ playbook_dir }}/../templates/text.json') }}"                                                                                                                                                           
    hetzner_token: "{{ lookup('ansible.builtin.env', 'HETZNER_TOKEN') }}"

- name: "Update DNS TXT record"                                                                                       
  community.dns.hetzner_dns_record:
    state: present                                                                                                                                                                                                                           
    zone: example.tech                                                                                                                                                                                                                 
    record: text.example.tech                                                                       
    type: TXT                                                                                                                                                                                                                                
    # txt_transformation: quoted
    ttl: 7200          
    value: "{{ lookup('file', '{{ playbook_dir }}/../templates/text.json') }}"                                                                                                                                                           
    hetzner_token: "{{ lookup('ansible.builtin.env', 'HETZNER_TOKEN') }}"

sample text.json

{
    "prefixes": [
        "2.50.168.20/32",
        "5.97.200.152/30",
        "37.206.241.112/29",
        "41.33.102.32/28",
        "41.33.114.144/29",
        "41.33.117.32/29",
        "41.33.124.16/28",
        "41.33.145.240/29",
        "41.33.156.64/29",
        "41.33.195.104/30",
        "51.77.80.0/24",
        "57.133.68.112/28",
        "58.137.228.96/29",
        "62.74.44.254/31",
        "62.74.245.96/30",
        "74.234.144.64/32",
        "78.133.112.40/29",
        "78.133.122.96/29",
        "80.28.106.117/32",
        "80.106.4.128/30",
        "81.4.137.4/30",
        "81.27.173.192/26",
        "81.192.177.170/32",
        "81.192.225.64/30",
        "81.192.226.68/30",
        "81.192.226.72/30",
        "82.185.206.192/29",
        "83.56.10.168/32",
        "83.164.178.152/30",
        "85.45.230.144/29",
        "85.74.255.48/30",
        "85.114.48.208/30",
        "85.114.55.92/30",
        "85.114.62.176/29",
        "85.114.63.0/24",
        "86.98.144.86/32",
        "87.202.120.116/30",
        "87.202.121.132/30",
        "87.202.121.200/30",
        "87.202.121.200/30",
        "87.213.91.56/30",
        "87.215.26.116/30",
        "88.2.183.94/32",
        "88.255.131.144/29",
        "88.255.153.92/30",
        "88.255.153.140/30",
        "88.255.249.224/29",
        "88.255.249.232/29",
        "91.25.126.128/28",
        "91.73.172.148/30",
        "95.0.52.32/28",
        "95.0.75.152/30",
        "95.0.75.160/30",
        "95.0.103.188/30",
        "95.0.170.128/29",
        "144.129.84.80/28",
        "145.255.67.201/32",
        "152.206.108.32/29",
        "156.200.127.64/29",
        "190.166.237.248/29",
        "194.224.125.168/29",
        "194.224.156.104/29",
        "194.224.194.64/29",
        "194.224.194.88/29",
        "194.224.194.104/29",
        "194.224.195.248/29",
        "195.29.106.96/28",
        "195.53.118.40/29",
        "195.53.240.32/29",
        "195.53.240.240/29",
        "195.53.241.208/29",
        "195.53.245.0/29",
        "195.53.245.8/29",
        "195.53.245.24/29",
        "195.53.245.40/29",
        "195.53.245.56/29",
        "195.53.245.64/29",
        "195.55.244.184/29",
        "195.57.146.200/29",
        "195.76.192.152/29",
        "195.76.202.248/29",
        "195.77.58.136/29",
        "195.77.170.32/29",
        "195.158.92.140/32",
        "195.158.111.187/32",
        "195.175.18.80/30",
        "195.175.18.92/30",
        "195.175.44.236/30",
        "195.175.65.96/30",
        "195.175.65.204/30",
        "195.175.107.0/30",
        "195.235.161.240/29",
        "197.230.77.176/30",
        "197.230.174.72/29",
        "212.156.123.32/29",
        "212.156.247.164/30",
        "212.170.220.224/29",
        "212.174.0.48/29",
        "212.174.187.104/29",
        "212.203.87.56/29",
        "213.4.210.56/29",
        "213.61.58.24/29",
        "213.61.78.192/26",
        "213.61.111.240/29",
        "213.61.135.76/30",
        "213.99.25.176/29",
        "213.150.169.160/30",
        "213.249.13.44/30",
        "213.249.36.4/30",
        "217.110.4.160/29",
        "217.192.100.80/29"
    ]
}
felixfontein commented 4 months ago

You don't use | string after the lookup (https://github.com/ansible-collections/community.dns/issues/190#issuecomment-1998245863). Did you try that?

chuegel commented 4 months ago

You don't use | string after the lookup (#190 (comment)). Did you try that?

Yes, it gives me the following error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.community.dns.plugins.module_utils.zone_record_api.DNSAPIError: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server"
fatal: [localhost]: FAILED! => changed=false 
  error: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server"
  msg: 'Error: Expected HTTP status 200, 422 for POST https://dns.hetzner.com/api/v1/records, but got HTTP status 502 (Unknown Error) with message "An invalid response was received from the upstream server"'
felixfontein commented 4 months ago

Ok, I think I found the problem. The TXT encoder has a bug so that it sometimes inserts TXT string breaks between \ and the following character (\ or "). In case of \", this ends up as \" "", which decoded translates to \" (note the extra space). This happens with your text.json file, resulting in Hetzner's API having trouble with the wrongly quoted string the module is sending.

191 fixes this, with that PR your example works fine for me. Without the fix, I can set the TXT record, but the module isn't idempotent (and in fact it sets something else) - when trying to set it again I get a similar error as you get.

chuegel commented 3 months ago

Hi @felixfontein,

thanks for the fix. Now it looks better but the quotes are now escaped :) like:

"{\"prefixes\": [
\"2.50.168.20/32\",\"5.97.200.152/30\",\"37.206.241.112/29\",\"41.33.102.32/28\",\"41.33.114.144/29\",\"41.33.117.32/29\",\"217.192.100.80/29\"]}"
felixfontein commented 3 months ago

Yes, but that's proper DNS TXT record escaping. The value of that record is:

{"prefixes": [
"2.50.168.20/32","5.97.200.152/30","37.206.241.112/29","41.33.102.32/28","41.33.114.144/29","41.33.117.32/29","217.192.100.80/29"]}
chuegel commented 3 months ago

Thanks, I understand

However I noticed some weird entries in the TXT record:

"{\"prefixes\": [\"2.50.168.20/32\",\"5.97.200.152/30\",\"37.206.241.112/29\",\"41.33.102.32/28\",\"41.33.114.144/29\",\"41.33.117.32/29\",\"41.33.124.16/28\",\"41.33.145.240/29\",\"41.33.156.64/29\",\"41.33.195.104/30\",\"51.77.80.0/24\",\"57.133.68.112/2" "8\",\"58.137.228.96/29\",\"62.74.44.254/31\",\"62.74.245.96/30\",\"74.234.144.64/32\",\"78.133.112.40/29\",\"78.133.122.96/29\",\"80.28.106.117/32\",\"80.106.4.128/30\",\"81.4.137.4/30\",\"81.27.173.192/26\",\"81.192.177.170/32\",\"81.192.225.64/30\",\"81" ".192.226.68/30\",\"81.192.226.72/30\",\"82.185.206.192/29\",\"83.56.10.168/32\",\"83.164.178.152/30\",\"85.45.230.144/29\",\"85.74.255.48/30\",\"85.114.48.208/30\",\"85.114.55.92/30\",\"85.114.62.176/29\",\"85.114.63.0/24\",\"86.98.144.86/32\",\"87.202.12" "0.116/30\",\"87.202.121.132/30\",\"87.202.121.200/30\",\"87.202.121.200/30\",\"87.213.91.56/30\",\"87.215.26.116/30\",\"88.2.183.94/32\",\"88.255.131.144/29\",\"88.255.153.92/30\",\"88.255.153.140/30\",\"88.255.249.224/29\",\"88.255.249.232/29\",\"91.25.1" "26.128/28\",\"91.73.172.148/30\",\"95.0.52.32/28\",\"95.0.75.152/30\",\"95.0.75.160/30\",\"95.0.103.188/30\",\"95.0.170.128/29\",\"144.129.84.80/28\",\"145.255.67.201/32\",\"152.206.108.32/29\",\"156.200.127.64/29\",\"190.166.237.248/29\",\"194.224.125.16" "8/29\",\"194.224.156.104/29\",\"194.224.194.64/29\",\"194.224.194.88/29\",\"194.224.194.104/29\",\"194.224.195.248/29\",\"195.29.106.96/28\",\"195.53.118.40/29\",\"195.53.240.32/29\",\"195.53.240.240/29\",\"195.53.241.208/29\",\"195.53.245.0/29\",\"195.53" ".245.8/29\",\"195.53.245.24/29\",\"195.53.245.40/29\",\"195.53.245.56/29\",\"195.53.245.64/29\",\"195.55.244.184/29\",\"195.57.146.200/29\",\"195.76.192.152/29\",\"195.76.202.248/29\",\"195.77.58.136/29\",\"195.77.170.32/29\",\"195.158.92.140/32\",\"195.1" "58.111.187/32\",\"195.175.18.80/30\",\"195.175.18.92/30\",\"195.175.44.236/30\",\"195.175.65.96/30\",\"195.175.65.204/30\",\"195.175.107.0/30\",\"195.235.161.240/29\",\"197.230.77.176/30\",\"197.230.174.72/29\",\"212.156.123.32/29\",\"212.156.247.164/30\"" ",\"212.170.220.224/29\",\"212.174.0.48/29\",\"212.174.187.104/29\",\"212.203.87.56/29\",\"213.4.210.56/29\",\"213.61.58.24/29\",\"213.61.78.192/26\",\"213.61.111.240/29\",\"213.61.135.76/30\",\"213.99.25.176/29\",\"213.150.169.160/30\",\"213.249.13.44/30" "\",\"213.249.36.4/30\",\"217.110.4.160/29\",\"217.192.100.80/29\"]}"

like \"57.133.68.112/2" "8\"

The generated text.json from the template seems to be fine though, no spaces

felixfontein commented 3 months ago

That's also normal DNS TXT quoting. Individual strings must not be longer than 255 bytes, you have to insert " " from time to time. Since spaces between strings are ignored, the resulting value does not have the space.

chuegel commented 3 months ago

I got this jinga template to generate the json: {"prefixes": [{% for item in int_prefixes -%}"{{ item.value.prefix }}"{{ "," if not loop.last else ""}}{%- endfor%}]}

Where should " "go to have a well formated TXT JSON? Thanks

felixfontein commented 3 months ago

What do you mean with "well formatted TXT JSON"? There is the DNS representation of a TXT entry, and there is it's value. If you want its value to be a JSON, then the module will just do it for you. You apparently want something else. Please define what you actually want to do since it doesn't seem to be something standard.

chuegel commented 3 months ago

I mean to have the above JSON into a TXT record but due to the 255 character limitation this is not possible (the API adds this whitespace and brakes the JSON) Another way would be to split the string into multiple JSON files but this is to much hassle.