zone files

[nikor30]

Would import export (bind) zone files be an option in the future ?

Also, now it could be cool to have an deploy button which uses netbox source to deploy DNS records to bind servers :-)

Just saying and wishing.

So far thanks for this great work

[yzguy]

Just a snippet relevant to importing zonefiles with dnspython, and loop through the rdatasets

From Zone File:

zone = dns.zone.from_file('path/to/example-zone.com.db', 'example-zone.com', relativize=False, check_origin=True)

From AXFR (Zone Transfer):

zone = dns.zone.from_xfr(dns.query.xfr('10.10.10.10', 'example-zone.com', relativize=False), relativize=False)

There is also a .from_text() method you can use as well, similar as above. Reference

Then to loop through the records

for (name, ttl, rdata) in zone.iterate_rdatas():
  rdtype = dns.rdatatype.to_text(rdata.rdtype)
  print(name, ttl, rdtype, rdata.to_text())

So I suspect, having some way to Click Import, upload file, then load it and loop through the records and creating/updating/deleting them in Netbox within the plugin is how that would happen.

• Zone File Import Example
• AXFR Import Example
• Looping Through Records Example

[gellis713]

@nikor30 The deployment is really outside of the scope of Netbox being a source of truth. I think what would make sense is a webhook tie-in that fires on create/update/delete of a zone/record/nameserver object. Then your automation tooling can easily handle generation and deployment of bind configuration.

[peteeckel]

@gellis713 I think generating zone files would be pretty much in scope of an export functionality. All the data that are needed for zone files are there, so generating them would only be an export with a very specific template.

Using webhooks that fire on zone/record changes would result in a very large number of zone updates, which would probably not be taken well by slave admins. So doing changes and then manually triggering an export to zone files when you're done looks like a better option to me.

It might be another issue with PowerDNS, where the "zone files" reside in a database instead of flat files on disk. That would indeed be more suited for a webhook. But even then triggering a NOTIFY/*XFR cycle per update will not be a good idea.

[gellis713]

@peteeckel In a typical scenario the webhook receiver builds the config and commits to a dev branch of a given repo, then a merge request would actually handle sending the zone updates. In the context of his question, he wants a deploy button not an export button. Actually deploying config directly to devices is really outside the scope of what Netbox intends to be.

I do agree that a webhook straight to zone update is not desirable.

[peteeckel]

@gellis713 That way it would indeed make much more sense to use a webhook as a way of communicating with a staging system.

A direct export would in any case require some validation (using named-checkzone or whatever is available) to run after exporting the zone files, so triggering a staging system to pull the data from NetBox before actually deploying makes sense as well.

On the other hand I'm still trying to figure out the most sensible workflow for DNS operations in NetBox. In my personal experience, DNS changes usually come in blocks, where quite a number of records are updated in one go and then the actual export/deployment is triggered for all of them, so the zone files can be created, validated and finally pushed to the primary server(s).

What would you think of a hybrid approach - Zone and Record updates don't trigger anything (except, maybe, marking the affected zones "dirty") until some custom script/external API request is executed that increments the affected zones' serial numbers, and that action then triggers a webhook that starts the actual deployment.

It is, I admit, a little bit beyond the scope of NetBox as a source of truth, but not too far and it would IMHO help to improve usability.

[peteeckel]

Just a quick update: I'm now convinced that @gellis713 is right and we should not pursue the idea of actively creating zone files.

I did a quick implementation of an Ansible playbook using the netbox/netbox Ansible collection, specifically the netbox.netbox.nb_lookup plugin using the data from Netbox DNS to create BIND configuration files for master and slaves as well as zone files for forward and reverse zones, and it really is so simple that integrating it (and the corresponding adapters for PowerDNS, Unbound etc.) in the plugin would unnecessarily bloat it and still not be as flexible.

So I will not pursue this approach any further. It has its merits, but the downsides outweigh them and there are simpler alternatives.

[peteeckel]

As far as the discussion went I don't think a direct export functionality doesn't make any sense. There are a couple of good reasons not to directly interfere with the authoritative DNS zones, and there are several options (including Webhooks/Ansible/OctoDNS etc.) to serve the intended purpose without misusing NetBox as an operational tool instead of the source of truth it's supposed to be.

@hatsat32: I suggest closing this issue.

[wolfspyre]

well... not too fast :) there's two glommed requests in this issue...

• 'deploy to mah nameservers' button which I agree is out of scope....
• an export mechanism which is capable of spitting out zonefiles...

I can see the argument for either:

• an export mechanism which emits zonefiles ready for consumption/comitting-to-a-repo
• a prescribed/documented process 'happy path'

That way, at least the users are steered in a consistent direction rather than being told 'not in scope. closing! seeya!' my $.03

[peteeckel]

Hi @wolfspyre, thanks for your input. Let me evaluate a bit why I think there's not much point in providing a specific "export zone file" functionality within NetBox DNS.

Regarding the functionality to deploy directly to a name server we're obviously on the same page. Not only that it's beyond the intended scope of NetBox (the same is true for network components etc., where NetBox also does not support vendor-specific configuration options, see for instance https://github.com/netbox-community/netbox/issues/10148), it also does not make too much sense from an operational point of view, also due to the lack of options to roll back to a defined state.

So let's concentrate on the issue of exporting zone files. I see two arguments against providing that functionality:

• Zone files are not that generic as well. If you're using PowerDNS, for instance, zone files are not the primary means of configuring zone data (though they can be imported).
• Creating zone files is not all it takes to provide the complete configuration of an authoritative server. At the very least configuration (in case of BIND the named.conf for the primary and the secondary nameservers) needs to be created - think of creating new zones or removing existing ones for a simple case. Creating the configuration file for the name server is, however, again covered by the 'we don't want to deploy to nameservers' argument.

Considering this, I don't see creating zone files as an issue that should be covered by the plugin's core functionality.

As for the 'consistent direction', it would be necessary to either limit the users to a specific configuration file layout or make the export function very flexible in terms of what gets exported where. There is not even a common standard between BIND implementations of different Linux distributions (RHEL/Debian follow completely different concepts), and many deployments follow even different configuration file layouts. You may have views, which are supported by BIND but not, for instance, PowerDNS (where you would use a different instance for each view) complicating things further. Said that, there is no 'happy path' that would match everyone's needs. Yes, that all can be solved, but it's not a trivial issue and would massively bloat the plugin's functionality and configuration options.

On the other hand, what instruments do we have?

• Using Ansible and the netbox.netbox.nb_lookup plugin or any other automation tool supporting NetBox to generate the configuration data from the NetBox API. That's the way I currently do it in the customer environments I maintain, and in my own setup as well. Works perfectly, although it requires triggering the export/deployment externally instead of from within NetBox.
• Using OctoDNS with the NetBox DNS-Plugin. I haven't tried that myself, but from the description it looks like a very decent solution abstracting away the differences between the authoritative name server implementations or even cloud solutions.
• Creating a Jinja2 export template for NetBox that can be used to export zones as zone files. That would probably come nearest to what the initial idea was, but there are some restrictions in terms of what NetBox exports can do. The main issue is that an export creates exactly one file, and that exporting only seems to be possible from list views - so one would have to filter for a specific zone, then create the export of that zone to a zone file, which would then be downloaded by the browser. Not exactly a comfortable solution.
• Creating a script for NetBox. This involves a bit of programming, but on the other hand it provides the greatest flexibility in terms of what happens with the data. I'd probably go down that route when I wanted to trigger zone file generation from NetBox DNS. In fact, one could even run the Ansible playbook from the first bullet point from a NetBox script. Scripts can be run from the API as well, providing all options of automation inside and outside of NetBox.
• Probably some more I just can't think of right now.

I already provided a little sample playbook for the Ansible option (which could use some update, I admit, but it can be useful as a starting point anyway). Export templates seem a bit too limited for my taste, but might be useful for versioning in a repo (although importing from zone files would be yet another issue).

It might, however, be an idea to provide a sample implementation of a custom script that dumps the NetBox DNS configuration into a directory of zone files.

Anyway, considering the arguments above I still don't see it as something that would make sense in the plugin itself.

Since you mentioned maintaining versions of the zone files or rather file sets in repositories: Not having any form of consistent versioning (with snapshots, rollback functionality etc.) is probably the thing I'm missing most about NetBox, especially with respect to NetBox DNS. Versioning can currently (and I don't expect that to change, to be honest, since Django does not provide any means for it AFAIK) only be done externally, and moving back to a given dataset is really painful, in most cases involving the restoration of a PostgreSQL backup.

Automatic versioning of generated zone files can, however, be done using Ansible, or with a Custom Script. I don't know what OctoDNS offers in terms of versioning - maybe @jcollie can comment on this.

[peteeckel]

Just to make my point regarding the script solution I hacked a little exporter script for zone data ... nothing fancy, and I wouldn't want to use it in production without further scrutiny, but it basically works.

#!/usr/bin/env python3

from pathlib import Path

from netbox_dns.models import View, Zone, Record

from extras.scripts import Script, StringVar, BooleanVar
from jinja2 import Environment, DictLoader

def rm_tree(path):
    for child in path.iterdir():
        if child.is_file():
            child.unlink()
        else:
            rm_tree(child)

    path.rmdir()

name = "NetBox DNS Exporters"

class ZoneExporter(Script):

    class Meta:
        name = "Zone Exporter"
        description = "This custom script can be used to export zone data to the file system"
        commit_default = True

    export_path = StringVar(
        description="Base path for the zone file export. The exporter will create a subdirectory 'netbox-dns-exporter' below this path",
        default="/home/netbox",
    )

    default_view_name = StringVar(
        description="Default view name for zones without a view",
        default="_default",
    )

    remove_existing_data = BooleanVar(
        description="Clean up existing data before exporting",
        default=True,
    )

    zone_template = '''\
;
; Zone file for zone {{ zone.name }} [{{ zone.view.name }}]
;

$TTL {{ zone.default_ttl }}

{% for record in records -%}
{{ record.name.ljust(32) }}    {{ (record.ttl|string).ljust(8) }} IN {{ record.type.ljust(8) }}    {{ record.value }}
{% endfor %}\
'''
    jinja_env = Environment(loader=DictLoader({"zone_file": zone_template}))
    template = jinja_env.get_template("zone_file")

    def run(self, data, commit):
        views = View.objects.all()

        export_path = Path(data["export_path"]) / "netbox-dns-exporter"

        if data["remove_existing_data"] and export_path.exists():
            self.log_info(f"Deleting the old export path {export_path}")
            try:
                rm_tree(export_path)
            except OSError as exc:
                self.log_failure(f"Could not remove the old export tree: {exc}")
                return

        try:
            export_path.mkdir(parents=False, exist_ok=True)
        except OSError as exc:
            self.log_failure(f"Could not create the export path {exc}")
            return

        for view in views:
            zones = Zone.objects.filter(view=view, status__in=Zone.ACTIVE_STATUS_LIST)
            if len(zones):
                self.log_info(f"Exporting zones for view '{view.name}'")
                self.export_zones(zones, view.name, export_path)

        zones = Zone.objects.filter(view__isnull=True, status__in=Zone.ACTIVE_STATUS_LIST)
        if len(zones):
            self.log_info("Exporting zones without a view")
            self.export_zones(zones, data["default_view_name"], export_path)

    def export_zones(self, zones, view_name, export_path):
        view_path = export_path / view_name

        try:
            view_path.mkdir(parents=True, exist_ok=True)
        except OSError as exc:
            self.log_failure(f"Could not create directory {view_path}: {exc}")
            return

        for zone in zones:
            self.log_info(f"Exporting zone {zone}")
            records = Record.objects.filter(zone=zone, status__in=Record.ACTIVE_STATUS_LIST)

            zone_data = self.template.render({"zone": zone, "records": records})

            zone_file_path = view_path / f"{zone.name}.db"
            try:
                zone_file = open(zone_file_path, "wb")
                zone_file.write(zone_data.encode("UTF-8"))
                zone_file.close()
            except OSError as exc:
                self.log_failure(f"Could not create zone file {zone_file_path}: {exc}")
                continue

Just enable SCRIPTS_ROOT in the configuration, drop the file in the path designated by it and you can export zone files just like that. Obviously it can also be used as a basis for other specific exporters.

[wolfspyre]

That’s TOTALLY a viable step in the direction of a solution …

Document it as alpha. Put it up in the repo, and ask for contributions.

My intention is more to expand the defined happy path, so that users/devs are steered in a more consistent direction and thus refinement contributions are more likely.. 

Could i hack something together to make zonefiles? Probably. Would I pick the same path you did?  maybe in some cases. maybe not in others.

I don’t want to re-invent the unnecessary-friction reduction and mobility-enhancement device.

you have already invented the wheel.

I want to help lube the wheel bearings…or ask why they’re missing :)

I think my point is ‘don’t let the perfect be the enemy of the good. ‘. 

document the process to install and use it, and add it. 

let refinement be tomorrows problem as this is iteratively better imo.

regardless of all of that:

thank you.  seriously. thank you.

There is so much bullshit in the world right now.

So many people dealing with so much hard.

it’s refreshing to see and important to remember that people like helping each other when given the opportunity

❤️🐺W

[peteeckel]

Hi @wolfspyre, thanks for your encouraging words and your enthusiasm.

I just provided a PR containing the script and some documentation for it in the examples directory. Providing 'alpha' functionality in a plugin that is used by some people in production is not such a good idea IMHO, but having the code around as an example with an appropriate disclaimer is probably something that might help anyone requiring this kind of functionality getting a good start.

Could i hack something together to make zonefiles? Probably. Would I pick the same path you did? maybe in some cases. maybe not in others.

And that pretty much sums up the problem with that kind of functionality: The requirements vary in such a broad way that it will be almost impossible (at least, very, very much effort, both in terms of development and in terms of maintenance and testing) to get it right for everyone. Let's see how the 'example code' approach works out.

[wolfspyre]

concur. another way to think about this script is that it could prove to be a good way to validate plug-in functionality in a black-boxy kinda way. if tweaking on other aspects of managed-record-creation/manipulation/etc … this could be a not horrible way to validate functionality…. or it could be a terrible idea. ¯\_(ツ)_/¯

[peteeckel]

I see several possible applications for that kind of functionality ... the script could actually quite easily be tweaked to dump the zone files, check them in to a Git repo, and so enable some kind of semi-manual rollback functionality (the function that NetBox is lacking) for DNS configurations.

Actually working on the script gave me some ideas for future script functionality - while exporting DNS data to authoritative servers is a tricky endeavour, importing data via dnspython and AXFR probably isn't as that is a standardised interface. The trickier part would be to get the DNS servers to answer (ACLs, TSIG keys etc.), but the task looks manageable.

The other question is whether it is really that useful. Thinking of NetBox as a 'source of truth', an import would probably be done once and then NetBox would become the leading system. On the other hand, with dynamic or 'foreign' zones (not being managed within NetBox) it could still make sense. And the initial data migration to NetBox DNS might also be a large task that could be simplified using an AXFR import script - so far I always got the zone data from the admins and converted them to CSV for importing, but that might not always be the case.