Autogenerated monitor name can fail NS1 validations if domains are too long

[nickrisaro]

Hello there,

We are using octodns-ns1 version 0.0.7 and we are facing some issues. We are using a dynamic record to setup GeoDNS as explained here however, our domain names are pretty long and when the monitors are being created we get an error back from NS1.

It would be great if either the monitor name can be supplied by us in the config or trimmed to 64 characters before making the API call to NS1.

Our dynamic record file looks like this:

---
xxxxxxx.xxxxxxx:
  dynamic:
    # These are the pools of records that can be referenced and thus used by rules
    pools:
      us-east:
        # Implicitly goes to the backup pool (below) if all values are failing
        # health checks
        type: CNAME
        values:
        - value: xxxxxxx.xx.xxxxxxx.xxxxxxxxx.com.
      us-west:
        # An optional fallback, if all of the records in this pool fail this pool should be tried
        fallback: us-east
        # One or more values for this pool
        type: CNAME
        values:
        - value: xxxxxxx.xx.xxxxxxx.xxxxxxxxx.com.
    # Rules that assign queries to pools
    rules:
    - geos:
      # Geos used in matching queries
      - NA-CA-BC
      - NA-CA-YT
      - NA-US-CA
      - NA-US-OR
      - NA-US-WA
      # The pool to service the query from
      pool: us-west
    # No geos means match all queries, the final rule should generally be a
    # "catch-all", served to any requests that didn't match the preceeding
    # rules. The catch-all is the only case where a pool may be re-used.
    - pool: us-east
  octodns:
    healthcheck:
      path: /health
  ttl: 30
  type: CNAME
  # These values become a non-healthchecked backup/default pool, generally it
  # should be a superset of the catch-all pool and include enough capacity to
  # try and serve all global requests (with degraded performance.) The main
  # case they will come into play is if all dynamic healthchecks are failing,
  # either on the service side or if the providers systems are experiencing
  # problems. They will also be used for when the record is pushed to a
  # provider that doesn't support dynamic records.
  value: xxxxxxx.xx.xxxxxxx.xxxxxxxxx.com.

And the error we get is:

ERROR NS1Client _try: method=create, args=({'name': 'xxxxxxx.xxxxxxx.xxxxxxxxx.com - CNAME - xxxxxxx.xx.xxxxxxx.xxxxxxxxx.com', 'notify_list': [{'config': {'sourceid': 'e2f24f3cecbf331845ec7ee0a5e033f5'}, 'type': 'datafeed'}]},), response=<Response [400]>, body={"message":"Invalid body","details":[{"type":"field-violation","field":"/name","message":"must have length less than 64 but was 72"},{"type":"request-id","message":"02619247-6a7a-4a19-bbba-dc042d421fba"}]}
Traceback (most recent call last):
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1080, in _notifylists_find_or_create
    nl = self._client.notifylists[name]
KeyError: 'xxxxxxx.xxxxxxx.xxxxxxxxx.com - CNAME - xxxxxxx.xx.xxxxxxx.xxxxxxxxx.com'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 274, in _try
    return method(*args, **kwargs)
  File "python/lib/python3.9/site-packages/ns1/rest/monitoring.py", line 91, in create
    return self._make_request(
  File "python/lib/python3.9/site-packages/ns1/rest/resource.py", line 74, in _make_request
Traceback (most recent call last):
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1080, in _notifylists_find_or_create
    return self._transport.send(type, self._make_url(path), **kwargs)
  File "python/lib/python3.9/site-packages/ns1/rest/transport/requests.py", line 126, in send
    resp_headers, jsonOut = self._send(
  File "python/lib/python3.9/site-packages/ns1/rest/transport/requests.py", line 91, in _send
    raise ResourceException("server error", resp, resp.text)
ns1.rest.errors.ResourceException: server error: Invalid body
    nl = self._client.notifylists[name]
KeyError: 'xxxxxxx.xxxxxxx.xxxxxxxxx.com - CNAME - xxxxxxx.xx.xxxxxxx.xxxxxxxxx.com'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "python/bin/octodns-sync", line 8, in <module>
    sys.exit(main())
  File "python/lib/python3.9/site-packages/octodns/cmds/sync.py", line 62, in main
    manager.sync(
  File "python/lib/python3.9/site-packages/octodns/manager.py", line 817, in sync
    total_changes += target.apply(plan)
  File "python/lib/python3.9/site-packages/octodns/provider/base.py", line 298, in apply
    self._apply(plan)
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1839, in _apply
    getattr(self, f'_apply_{class_name}')(ns1_zone, change)
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1781, in _apply_Create
    params, active_monitor_ids = getattr(self, f'_params_for_{_type}')(new)
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1632, in _params_for_CNAME
    return self._params_for_dynamic(record)
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1587, in _params_for_dynamic
    active_monitors, answers = self._generate_answers(record, regions)
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1538, in _generate_answers
    monitor_id, feed_id = self._monitor_sync(
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1334, in _monitor_sync
    monitor_id, feed_id = self._monitor_create(expected)
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1109, in _monitor_create
    nl = self._notifylists_find_or_create(nl_name)
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 1091, in _notifylists_find_or_create
    nl = self._client.notifylists_create(
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 233, in notifylists_create
    nl = self._try(self._notifylists.create, body)
  File "python/lib/python3.9/site-packages/octodns_ns1/__init__.py", line 274, in _try
    return method(*args, **kwargs)
  File "python/lib/python3.9/site-packages/ns1/rest/monitoring.py", line 91, in create
    return self._make_request(
  File "python/lib/python3.9/site-packages/ns1/rest/resource.py", line 74, in _make_request
    return self._transport.send(type, self._make_url(path), **kwargs)
  File "python/lib/python3.9/site-packages/ns1/rest/transport/requests.py", line 126, in send
    resp_headers, jsonOut = self._send(
  File "python/lib/python3.9/site-packages/ns1/rest/transport/requests.py", line 91, in _send
    raise ResourceException("server error", resp, resp.text)
ns1.rest.errors.ResourceException: server error: Invalid body

[ross]

Interesting. Will poke around. First thought is if < 64 chars, it'd hash the value and then truncate things enough to stick the hash on the end, but with so little content it might not reliable hash. Second thought would be to truncate and use some sort of index that won't change, but I don't recall off-hand if such a thing exists where it'd be needed or if it would be consistent.

Failing that I'll look into a way to specify a prefix or something to that effect so that you could take some-really-really-very-long-and verbose-thing-<...> and tell it to be srrvlvt-<...> or similar.

[ross]

Looks like you're running into a name limit on the notify list, you might try using a shared notify list,

  ns1:
    class: octodns_ns1.Ns1Provider
    api_key: env/NS1_API_KEY
    shared_notifylist: true

That'll pick a shared/single notifylist here

https://github.com/octodns/octodns-ns1/blob/7e215dc7cb0ab0bd8522bd189d83b4ac7ee80e68/octodns_ns1/__init__.py#L1136-L1140

Which is https://github.com/octodns/octodns-ns1/blob/main/octodns_ns1/__init__.py#L328

We didn't use that at GitHub, tbh not sure why, maybe just created before that was a thing. My personal/testing setup for octoDNS does have it enabled. Unless you otherwise go in and add actual notification settings to the notify lists they're not going to be doing anything anyway, just a required bit of NS1's setup.

Anyway, you may run into further name length limits elsewhere, but that should get you past the initial one at least and as far as I can remember there's no downside to using them shared one.

You can also run a sync with --debug to get more verbose debug logging. That can be super helpful for me to figure out what's happening since I don't have your exact data/setup.