search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.11k stars 5.47k forks source link

[BUG] boto3_route53.rr_present `Route 53 public hosted zone None not found` #65817

Open japtain-cack opened 8 months ago

japtain-cack commented 8 months ago

Description

I believe we've encountered a bug with boto3_route53.rr_present and related states/modules. It seems the function that returns the hosted zone data is potentially not parsing it properly; from what I can tell. Pillars are being passed in properly and we double and triple checked the zoneID is correct.

Also, we still have to have boto installed even though this is a boto3 module/state. This state was broken until we included boto as a dependency.

[DEBUG   ] Starting new HTTPS connection (1): route53.amazonaws.com:443
[DEBUG   ] https://route53.amazonaws.com:443 "GET /2013-04-01/hostedzone/ABS10239576HTOEWS HTTP/1.1" 200 623
[INFO    ] Route 53 public hosted zone None not found
[INFO    ] Route 53 public hosted zone None not found

Setup

Please be as specific as possible and give set-up details.

Steps to Reproduce the behavior state:

{%- set records  = salt['pillar.get']('aws:route53:records', []) %}
{%- set region = salt['pillar.get']('aws:route53:region') %}

include:
  - apps.boto
  - apps.boto3

{%- for record in records %}
  {%- set host_name = record.get('name') or grains['id'].split('.')[0] %}

  {%- if record['enabled'] %}
    # Sometimes primary_ipv4 isn't set. Assuming order places primary in the first item of the list of IPs
    {%- set value = record.get('value') or salt['grains.get']('primary_ipv4') or salt['grains.get']('ipv4')[0] %}

r53_rr_update_{{ host_name }}_{{ value }}:
  boto3_route53.rr_present:
    - name: {{ host_name }}.{{ record['zone'] }}
    - HostedZoneId: {{ record['zone_id'] }}
    - PrivateZone: {{ record.get('private_zone', False) }}
    - Type: {{ record.get('record_type', 'A') | upper }}
    - ResourceRecords:
      - {{ value }}
    - TTL: {{ record.get('ttl', 60) }}
    - region: {{ region }}
    - require:
      - pip: boto3_install

  {%- else %}
r53_rr_remove_{{ host_name }}_{{ value }}:
  boto3_route53.rr_absent:
    - name: {{ host_name }}.{{ record['zone'] }}
    - HostedZoneId: {{ record['zone_id'] }}
    - Type: {{ record.get('record_type', 'A') | upper }}
    - PrivateZone: {{ record.get('private_zone', False) }}
    - region: {{ region }}
    - require:
      - pip: boto3_install

  {% endif -%}

{% endfor -%}

pillars:

aws:
  route53:
{%- if 'west' in grains['id'] %}
    region: us-west-2
    records:
      # Ensure the zone has a trailing '.'.
    - zone: vpc.internal.
      enabled: True
      zone_id: ABS10239576HTOEWS

{%- elif 'east' in grains['id'] %}
    region: us-east-2
    records:
      # Ensure the zone has a trailing '.'.
    - zone: vpc.internal.
      enabled: True
      zone_id: ZBS10239576HTFFQER

{%- endif %}

logs:

[INFO    ] Python package boto3 was already installed
All specified packages are already installed
[INFO    ] Completed state [boto3] at time 15:08:49.123413 (duration_in_ms=453.416)
[DEBUG   ] The functions from module 'boto3' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded boto3.assign_funcs
[DEBUG   ] The functions from module 'boto3_route53' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded boto3_route53.find_hosted_zone
[DEBUG   ] The functions from module 'boto3_route53' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded boto3_route53.rr_present
[INFO    ] Running state [west-prd.vpc.internal.] at time 15:08:49.133968
[INFO    ] Executing state boto3_route53.rr_present for [west-prd.vpc.internal.]
[DEBUG   ] Starting new HTTP connection (1): 169.254.169.254:80
[DEBUG   ] http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 200 56
[DEBUG   ] Resetting dropped connection: 169.254.169.254
[DEBUG   ] http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 63
[DEBUG   ] Resetting dropped connection: 169.254.169.254
[DEBUG   ] http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/app-deployment-prd-01fn7j51jh9f2fv30k-InstanceRole-NKNZ60XSURTI HTTP/1.1" 200 1582
[DEBUG   ] Starting new HTTPS connection (1): route53.amazonaws.com:443
[DEBUG   ] https://route53.amazonaws.com:443 "GET /2013-04-01/hostedzone/ABS10239576HTOEWS HTTP/1.1" 200 623
[INFO    ] Route 53 public hosted zone None not found
[INFO    ] Route 53 public hosted zone None not found
[INFO    ] Completed state [west-prd.vpc.internal.] at time 15:08:49.554757 (duration_in_ms=420.788)
[DEBUG   ] File /var/cache/salt/minion/accumulator/140353340676544 does not exist, no need to cleanup
[DEBUG   ] Closing AsyncReqChannel instance
[DEBUG   ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'west-prd.example.com', 'tcp://10.155.64.102:4506')
[DEBUG   ] Closing AsyncReqChannel instance
[DEBUG   ] The functions from module 'highstate' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded highstate.output
local:
----------
          ID: boto_install
    Function: pip.installed
        Name: boto
      Result: True
     Comment: Python package boto was already installed
              All specified packages are already installed
     Started: 15:08:47.667951
    Duration: 1001.841 ms
     Changes:
----------
          ID: boto3_install
    Function: pip.installed
        Name: boto3
      Result: True
     Comment: Python package boto3 was already installed
              All specified packages are already installed
     Started: 15:08:48.669997
    Duration: 453.416 ms
     Changes:
----------
          ID: r53_rr_update_west-prd_10.155.65.118
    Function: boto3_route53.rr_present
        Name: west-prd.vpc.internal.
      Result: True
     Comment: Route 53 public hosted zone None not found
     Started: 15:08:49.133969
    Duration: 420.788 ms
     Changes:

Expected behavior DNS records are updated.

Screenshots If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) master ```yaml sudo salt --versions-report Salt Version: Salt: 3006.5 Python Version: Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.2 libgit2: 1.7.1 looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: 1.13.1 python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.14.2 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: amzn 2 locale: utf-8 machine: x86_64 release: 4.14.330-250.540.amzn2.x86_64 system: Linux version: Amazon Linux 2 ``` minion: ```yaml sudo salt-call --versions-report Salt Version: Salt: 3006.5 Python Version: Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: 18.6.1 dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.2 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.14.2 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: amzn 2 locale: utf-8 machine: x86_64 release: 4.14.327-246.539.amzn2.x86_64 system: Linux version: Amazon Linux 2 ```

Additional context Add any other context about the problem here.

japtain-cack commented 8 months ago

Disregard, after further investigation, these are private zones, and the docs for boto3_route53 are different in this regard vs the boto_route53. I was assuming, incorrectly, that the private zone attribute was optional.

boto_route53 states: If using split_dns, specify if this is the private zone. boto3_route53 states: Set to True if the resource record should be in a private zone, False if public.

So, after specifically setting PrivateZone to True, seems to have resolved the issue.