delete_dns_record broken with digitalocean (pagination)

Description of Issue

When querying digitalocean for domain records (both for deletion and stale checks) via https://api.digitalocean.com/v2/domains<DOMAIN>/records, the API now returns at most 20 records in .domain_records. To access further records, one must call it with a page=<PAGE> GET parameter. This change has not been documented (and it is thus not surprising that it broke within salt), but it is rather simple to verify. The .meta.total key contains the total amount of records, and .links.pages contains various keys such as "first", "next", "last" and "prev" that give the URLs to use to get the appropriate information/page.

Setup & Steps to Reproduce Issue

Set up a digitalocean-based salt-cloud environment, with domains, create_dns_domain and delete_dns_domain.
Create a large amount (currently >20) unrelated domain names.
Bootstrap a salt client with a name that will trigger create_dns_domain.
Delete the salt client.
Observe that it does not see the whole set of dns domains and does not find the client's record to delete.

Attached is a log with any identifiable information ripped out:

all domains are changed to id.example.com

all IPs are changed to 127.0.0.1 or ::1 NOTE that as a result of this being ripped out, it is impossible to verify that the compatible DNS record isn't in the collection in question, but I assure you that it isn't.


[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Including configuration from '/etc/salt/cloud.conf.d/highstate.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.conf.d/highstate.conf
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/fileserver.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/fileserver.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/interface.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/interface.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/pillar.conf
[DEBUG   ] Changed git to gitfs in master opts' fileserver_backend list
[DEBUG   ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/digitalocean.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/digitalocean.conf
[DEBUG   ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/digitalocean.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/digitalocean.conf
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Marking 'base64_encode' as a jinja filter
[DEBUG   ] Marking 'base64_decode' as a jinja filter
[DEBUG   ] Marking 'md5' as a jinja filter
[DEBUG   ] Marking 'sha256' as a jinja filter
[DEBUG   ] Marking 'sha512' as a jinja filter
[DEBUG   ] Marking 'hmac' as a jinja filter
[DEBUG   ] Marking 'random_hash' as a jinja filter
[DEBUG   ] Marking 'rand_str' as a jinja filter
[DEBUG   ] Marking 'file_hashsum' as a jinja filter
[DEBUG   ] Marking 'http_query' as a jinja filter
[DEBUG   ] Marking 'strftime' as a jinja filter
[DEBUG   ] Marking 'date_format' as a jinja filter
[DEBUG   ] Marking 'yaml_dquote' as a jinja filter
[DEBUG   ] Marking 'yaml_squote' as a jinja filter
[DEBUG   ] Marking 'yaml_encode' as a jinja filter
[DEBUG   ] Marking 'raise' as a jinja global
[DEBUG   ] Marking 'match' as a jinja test
[DEBUG   ] Marking 'equalto' as a jinja test
[DEBUG   ] Marking 'skip' as a jinja filter
[DEBUG   ] Marking 'sequence' as a jinja filter
[DEBUG   ] Marking 'to_bool' as a jinja filter
[DEBUG   ] Marking 'tojson' as a jinja filter
[DEBUG   ] Marking 'quote' as a jinja filter
[DEBUG   ] Marking 'regex_escape' as a jinja filter
[DEBUG   ] Marking 'regex_search' as a jinja filter
[DEBUG   ] Marking 'regex_match' as a jinja filter
[DEBUG   ] Marking 'regex_replace' as a jinja filter
[DEBUG   ] Marking 'uuid' as a jinja filter
[DEBUG   ] Marking 'unique' as a jinja filter
[DEBUG   ] Marking 'min' as a jinja filter
[DEBUG   ] Marking 'max' as a jinja filter
[DEBUG   ] Marking 'avg' as a jinja filter
[DEBUG   ] Marking 'union' as a jinja filter
[DEBUG   ] Marking 'intersect' as a jinja filter
[DEBUG   ] Marking 'difference' as a jinja filter
[DEBUG   ] Marking 'symmetric_difference' as a jinja filter
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad digital_ocean.optimize_providers: 'digital_ocean.optimize_providers' is not available.
[DEBUG   ] The 'digital_ocean' cloud driver is unable to be optimized.
[DEBUG   ] Could not LazyLoad digital_ocean.list_nodes_min: 'digital_ocean.list_nodes_min' is not available.
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG   ] https://api.digitalocean.com:443 "GET /v2/droplets/?page=1&per_page=200 HTTP/1.1" 200 None
[DEBUG   ] https://api.digitalocean.com/v2/droplets/?page=1&per_page=200
[INFO    ] Destroying in non-parallel mode.
[DEBUG   ] LazyLoaded cloud.fire_event
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/cloud/id.example.com/destroying; data = {u'_stamp': '2019-10-28T16:37:17.290391', 'u'name': u'<SOMENAME>',
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG   ] https://api.digitalocean.com:443 "GET /v2/droplets/?page=1&per_page=200 HTTP/1.1" 200 None
[DEBUG   ] https://api.digitalocean.com/v2/droplets/?page=1&per_page=200
[DEBUG   ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG   ] https://api.digitalocean.com:443 "DELETE /v2/droplets/164713903/ HTTP/1.1" 204 0
[DEBUG   ] https://api.digitalocean.com/v2/droplets/164713903/
[DEBUG   ] Deleting DNS records for id.example.com.
[DEBUG   ] Starting new HTTPS connection (1): api.digitalocean.com
[DEBUG   ] https://api.digitalocean.com:443 "GET /v2/domains/example.com/records HTTP/1.1" 200 None
[DEBUG   ] https://api.digitalocean.com/v2/domains/example.com/records
[DEBUG   ] found DNS records: {u'domain_records': [{u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 1800,
                  u'type': u'SOA',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 1800,
                  u'type': u'NS',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 1800,
                  u'type': u'NS',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 1800,
                  u'type': u'NS',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'A',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'A',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'A',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'A',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'A',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'A',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 43200,
                  u'type': u'CNAME',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 43200,
                  u'type': u'CNAME',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 43200,
                  u'type': u'CNAME',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 43200,
                  u'type': u'CNAME',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 43200,
                  u'type': u'CNAME',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 43200,
                  u'type': u'CNAME',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'A',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'AAAA',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'A',
                  u'weight': None},
                 {u'data': u'127.0.0.1',
                  u'flags': None,
                  'u'id': <SOMEID>,
                  'u'name': u'<SOMENAME>',
                  u'port': None,
                  u'priority': None,
                  u'tag': None,
                  u'ttl': 3600,
                  u'type': u'AAAA',
                  u'weight': None}],
u'links': {u'pages': {u'last': u'https://api.digitalocean.com/v2/domains/example.com/records?page=2',
                   u'next': u'https://api.digitalocean.com/v2/domains/example.com/records?page=2'}},
u'meta': {u'total': 40}}
[DEBUG   ] deleting DNS record IDs: []
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/cloud/id.example.com/destroyed; data = {u'_stamp': '2019-10-28T16:37:18.046830', 'u'name': u'<SOMENAME>',
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] Deleted '/etc/salt/pki/master/minions/id.example.com'
[DEBUG   ] LazyLoaded nested.output
The following virtual machines are set to be destroyed:
do:
digital_ocean:
  id.example.com

Proceed? [N/y] ... proceeding do:

digital_ocean:
    ----------
    id.example.com:
        True

### Versions Report

Salt Version: Salt: 2019.2.2

Dependency Versions: cffi: Not Installed cherrypy: unknown dateutil: 2.6.1 docker-py: Not Installed gitdb: 2.0.3 gitpython: 2.1.8 ioflo: Not Installed Jinja2: 2.10 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: 1.0.7 msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.15+ (default, Oct 7 2019, 17:39:04) python-gnupg: 0.4.1 PyYAML: 3.12 PyZMQ: 16.0.2 RAET: Not Installed smmap: 2.0.3 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.2.5

System Versions: dist: Ubuntu 18.04 bionic locale: UTF-8 machine: x86_64 release: 4.15.0-66-generic system: Linux version: Ubuntu 18.04 bionic


### P.S.

This move is obviously unacceptable on the side of DigitalOcean, as no warning was provided to anyone, as far as I can tell.
However, I don't think any of us have the power to force them to act otherwise at this point, and salt should be able to handle this, one way or another.

saltstack / salt