Render configuration templates natively

[jeremystretch]

NetBox version

v3.4.3

Feature type

New functionality

Proposed functionality

Render device configurations directly from NetBox, using a new ConfigurationTemplate model (detailed below) and both locally- and remotely-sourced context data (see also #11558). Users will be able to fetch rendered configuration templates via both the NetBox UI and APIs.

Identifying the appropriate configuration template for a specific device/VM may need further exploration, but an approach similar to what we currently do for config contexts is likely needed. It should also be possible to explicitly define the primary template for any supported object (e.g. by adding a config_template ForeignKey to the Device and VirtualMachine models).

Inclusion tags within a template should work relative to the source file's path within its parent source. We'll need to provide a custom environment overloading the get_template method to return the appropriate DataFile instance.

(I originally considered taking this on as a plugin, however IMO there's a fairly strong case for implementing this functionality natively under the under the "source of truth" umbrella.)

Use case

One of the core functions of network automation involves rendering device configurations from a template (typically written in Jinja2 or a similar language) and contextual data from one or more sources of truth (e.g. NetBox). While this can be accomplished using provisioning tools like Ansible, offering this ability natively within NetBox can simplify the toolchain needed.

Database changes

This proposal would entail the creation of at least one new model, to represent each template. (The DataFile model referenced below is detailed in #11558.)

ConfigurationTemplate

• name - User-configured name
• description
• language - Template language (Jinja2, possibly others)
• data_file - Foreign key to DataFile (optional)
• content - Stores template content if not sourced from a DataFile

External dependencies

N/A

[renatoalmeidaoliveira]

@jeremystretch I think it would be nice to have a "composable" environment for configuration templates, for example:

when deploying a device you can have:

• A base template for AAA, SNMP, license and other stuff
• A particular configuration for some device type, or scenario

That way having several base templates, and a configuration template that is a composable of the bases may be useful

Another point is how to add variables inside that template, in such way it could be used to render some configuration to deploy a new device or inside a script.

[ryanmerolle]

@carlbuchmann, @titom73, @ClausHolbechArista

There might be some good synergies here with ansible avd if you have thoughts.

[jeremystretch]

@renatoalmeidaoliveira my intent here is to determine per device the appropriate base template, which can pull in other templates via {% include %} tags. These inclusion templates can be reused by many templates. This approach has worked very well for me in the past, and mimics NetBox's own template hierarchy.

[renatoalmeidaoliveira]

Intresting, very similar to the templates environment, that way the user gonna have several "base" templates and for a full configuration he build a template import the base ones. It seens interesting, but how you gonna import "as a template" some model attribute?

[jeremystretch]

So far, I've got the template rendering working as planned, however I'm getting a little hung up on deciding how best to relate devices to specific templates. To review, each device should reference exactly one (if any) configuration template, which can be rendered using context data to produce the desired configuration of that device.

It's entirely possible to designate a single global base template for all devices, and employ logic to evaluate a device's role, platform, etc. to determine which other relevant template to pull in (via {% include %} tags). However, this is probably not ideal, and I expect that people will want some mechanism native to NetBox for selecting which of several base templates to use for a device. I see a few options, detailed below.

Option A: Add a template ForeignKey to Device

This is pretty straightforward: Just add a ForeignKey to the Device model so that each device can point to its appropriate ConfigTemplate. This allows for very efficient queries at the cost of requiring explicit assignment for each device. It's also the safest option, as a misconfiguration will affect only the single device on which it's made.

Option B: Add a template ForeignKey to DeviceRole and/or Platform

Rather than specifying templates per device, we could specify them per device role and/or platform. This would require much less explicit configuration than option A, however we run into an issue if a device's role and platform each indicate a different template. It's also likely users may want to classify devices by some other attribute, such as assigned tag, for the purpose of determining which template to render.

(This option could be combined with option A, where a device's individually-defined template would take precedence over that associated with its role or platform.)

Option C: Add Many-to-Many Assignments on ConfigTemplate

This approaches the problem from a different direction, associating each template with a set of device roles and/or platforms. To determine the appropriate template for a device, a query is made for a template with matching role & platform. (This is similar in concept to how config contexts are retrieved.) This grants a great deal of flexibility, however it would also require some form of tie-breaking mechanism in case multiple matching templates are found.

For example, suppose you associate template A with device role X and template B with platform Y. A device having role X and platform Y would match on both templates. We would need to include a factor such as numeric weighting to prefer one over the other; or return an error.

As with option B, there's a danger here in that a minor change to a template (e.g. removing an assigned role) can break template rendering for a large number of devices; this may be an acceptable drawback considering the flexibility afforded. However, it also complicates retrieving associated device templates in bulk.

This approach could also be combined with option A, which would provide an alternative, explicit assignment for more conservative deployments.

[falz]

Fond of Option B but allowing option A to override the role/platform per device seems like the sweet spot to me.

Option C sounds good on paper but seems like excess complexity.

Note that we wrote our own config generator using Netbox, and may have further feedback on this based on our experiences once we see what data is available to the template. Will chime in with more details on that when the time is appropriate.

[zn0k]

I concur that option C, while powerful, could get very confusing. Given that the penalty is overwriting the config of a production device and taking it down, simple seems better.

A mixture of options A and B with a slight tweak seems best to me as well. Being able to set defaults via a combination of platform (so we can differentiate between IOS and Junos, for example) and role while retaining the ability to override on a per-device basis by directly assigning a template would be reasonably simple but also allow enough escape hatches to solve slightly trickier situations. I do think it's necessary to allow the combination of platform and role to determine the template - many networks are multi-platform. Or, if you're on Cisco, you're not very unlikely to run multiple dialects between legacy and newer devices.

[PieterL75]

I would like to be able to run multiple templates to a device.

• Initial template to setup the device (hostname, tacacs, snmp, mgmt ip, mgmt route, ....)
• MLAG template to setup an MLAG
• Layer2 template to setup STP, VLAN
• Layer3 template to add routing
• other templates that do small things

An ofcourse, each template has to be valid for that vendor/syntax.

imo, this should only create a config out of a template, but never deploy it direct. That does not seems like the task of NetBox

[jeremystretch]

I would like to be able to run multiple templates to a device.

You can do this by creating a base template which includes other (reusable) templates, in the desired order and subject to any logic asserting specific device attributes. For example:

{% include 'common/system.jinja2' %}
{% include 'common/interfaces.jinja2' %}
{% include 'common/access-lists.jinja2' %}
{% if device.role.slug == 'some-special-role' %}
  {% include 'special/other-stuff.jinja2' %}
{% endif %}

You'll likely have different base templates for each role/platform.

I also want to point out that you do not need to create a corresponding ConfigTemplate instance in NetBox to accomplish this: Included templates can be populated directly from other remotely-sourced data files. You'll only need to create a ConfigTemplate for each template that you want to render directly.

imo, this should only create a config out of a template, but never deploy it direct.

100% :+1: - There are plenty of other tools for that.

[ryanmerolle]

Not trying to pile on, but just share a current workflow to broaden an understanding of how users currently approach.

I currently do this off box, and my templates get applied to a "function" and platform. Functions are assigned to device roles (the netbox model). Here are a few examples:

• NetBox Device Roles: border_leaf, edge_leaf, storage_leaf, spine, internet_firewall, office_firewall, b2b_firewall, console_server, mgmt_core, mgmt_access, console_server, voice_gateway, user_l2_leaf, user_l3_leaf, pdu, ups
• NetBox Platforms: eos, ios, nx-os, opengear, fortios, panos, aos
• Templates Assigned to a network function: b2b_peering, leaf_peering, compute_access, storage_leaf, underlay_spine_leaf, evpn_overlay, vxlan, base, wan

I think map my made up functions to NetBox Device Roles like:

device_role_definitions:
  border_leaf:
    - base (applies the device platform's base)
    - leaf_peering (for lack of a better name, it includes the templates for the ibgp, mlag, lacp, and other similar leaf peer config)
    - wan (bgp, prefix-lists, community-lists, route-maps, circuit terminations w/descriptions, bfd, etc.)
    - underlay_spine_leaf (fiber uplinks, bgp peering, etc.)
    - evpn_overlay (evpn, bgp, etc)
    - vxlan
    - border_leaf (specific border_leaf only config)
    - storage_leaf (specific config if a location is so small that storage_leaf role is consolidated on this device)
  storage_leaf:
    - base (applies the device platform's base)
    - leaf_peering (for lack of a better name, it includes the templates for the ibgp, mlag, lacp, and other similar leaf peer config)
    - underlay_spine_leaf (fiber uplinks, bgp peering, etc.)
    - evpn_overlay (evpn, bgp, etc)
    - vxlan
    - storage_leaf (storage network specific items)
  spine:
    - base (applies the device platform's base)
    - underlay_spine_leaf (fiber uplinks, bgp peering, etc.)
    - evpn_overlay (evpn, bgp, etc)
    - spine (specific spine only config like summary networks)

All of this is defined by using include templates like @jeremystretch mentioned. "Functions" are not really modeled in netbox, but in the templates that define each device role. All of the above templates are created as needed, and if I bring up a JUNOS border_leaf in the future, the template would fail (as it should) until platform specific templates are created for all the boarder_leaf functions.

I probably would continue to do this off box because I produce additional artifacts and upload all to git:

• Single config file per device in cli platform format
• Single config file per device in json/yaml form
• Single markdown document per device
• Topology Docs (markdown, yaml, etc) per site
• Snapshot of netbox data used for the config build (per device, per site, etc)

[ryanmerolle]

Oh I forgot to add, I feel like the Option C, similar to how config_context can be applied ultimate gives use the most flexibility, but its likely a trade off with complexity/confusion.

[jeremystretch]

For now, I've just added the foreign key from Device to ConfigTemplate, since it seems like we'll almost certainly want the ability to specify an explicit assignment regardless. Making really great progress with this already!

[davidgwatkins]

This is really awesome progress! Any chance that could be modified to be Device to Many? We have J2 templates that are CLI for Day 0 / out-of-band / sneakernet, then NETCONF for Day 2 use cases; which gets rendered is based on device status. Perhaps this is covered under the use case for multiple component templates, but we are already using includes.

[jeremystretch]

Pretty happy with where this ended up. No doubt we'll continue to refine & improve the implementation from here, but I'm calling it done for the purposes of this particular FR.

Any chance that could be modified to be Device to Many?

You'll be able to implement conditional logic within the template itself, which could (for example) check for the presence of a flag indicating which type of config to render, and pull in the relevant subordinate template files.

[dmulyalin]

This is great.

Late to the party, but would like to share an idea.

It would be good to associate config templates with services as well. Say service L3VPN for customer assigned to the device, service has template associated matching device platform to produce requied configuration.

[JonasEinfach]

@jeremystretch : Do you have an overview of the device and interface object, which you can use in the Jinja2 Configuration Templates?

For example:

Device:

{{ device.name }} --> working: print device name {{ device.url }} --> not working: should print url? {{ device.serial }} --> working: print sn

Interfaces:

{% for interface in device.interfaces.all() %} {{ interface }} --> working: print interface name {{ interface.enabled }} --> working: print interface enabled value {% endfor %}

Custom Fields:

Can you also access the custom fields of an interface / device?

I try serveral methods to get an overview of all attributes / values / keys of the object ... but not working {{ device.keys() }} / {{ device.values() }} / {{ device.attr() }}

Access netbox data:

I think it would also be a very nice feature, if you can access the complete netbox data via jinja2 templates. For example:

• check the device location and get id
• get all vlans for this location id
• Render the vlans which you get

Maybe it is already possible and I dont know how?