Allow non-unique device and virtual machine names

jeremystretch commented 5 years ago

Environment

Python version: 3.5.2
NetBox version: 2.5.0

Proposed Functionality

Remove the uniqueness constraint on device and virtual machine names, so that multiple objects can be created with the same name. It is worth noting that NetBox technically already supports non-unique names, given that multiple devices can be unnamed (i.e. with a name of NULL).

Use Case

Some users have expressed a desire to support having multiple devices/VMs with the same name.

Database Changes

Remove unique=True from the name field on the Device and VirtualMachine models.

External Dependencies

None

MelanieTanaka commented 5 years ago

I definitely need this. We just evaluated netbox last week and opted to not use it since we have several hundred rack-mounted cable management units that we really do not need unique names for. We just want to give a reasonably descriptive name to each one such as the part number (or even go as generic as "cable manager"), but we can't do this with unique names being enforced.

Also if, as you mentioned, unique names are not really enforced to begin with by allowing empty names, it seems a bit odd.

lampwins commented 5 years ago

@MelanieTanaka to your point specifically, this is already possible today. When a device does not have a name in NetBox, it is displayed using its Device Type's name (which would include the model).

MelanieTanaka commented 5 years ago

@MelanieTanaka to your point specifically, this is already possible today. When a device does not have a name in NetBox, it is displayed using its Device Type's name (which would include the model).

We did notice leaving a name blank would pull a name from somewhere but in our case it was the device "role". Is there an option that would make it behave differently?

bellwood commented 5 years ago

If this resolves #1088 I'm all in favor.

lampwins commented 5 years ago

@MelanieTanaka that is true for the primary display in a rack, however, the hover tooltip includes the device type. So the typical use case is to have a device role of "Cable Manager" and if you wish to track the difference between different types of cable managers, do this with the device type.

I never liked the idea of people having 1000 devices with the actual name "Cable Manager" because inevitably people will build filters based off of the name, and they will have typos resulting in some amount of these devices named "Calbe Manager."

dgomes87 commented 5 years ago

In our scenario there's 3 use cases where we would really need this.

first is building out exact copies of cabinets to offer our solutions. 1 customer = 1 cabinet, each built exactly the same. Devices have the same names/identification tags in each, same IPs (different VRF) and everything else. The only thing differentiating them from one another is the public IPs/VRF, and cabinet name. Right now we can't accurately represent this in netbox so we have had to prefix every device name with the cabinet name. We just started offering this service so it is not a huge issue yet but I see it growing into something severe enough that our techs would want to discard netbox and use something else.

second situation is we have multiple sites built out the same way, and on each site there are devices named the same due to our naming convention being based on a per-site basis. So each site can have a "server-01" for example.

third issue is the same one pointed out by @MelanieTanaka, all our "dumb" objects like cable management and other rack mountable accessories, have no need for unique naming, but for the sake of completeness of documentation we do not want to leave blank names as pointed out by @lampwins, even though we have tested and confirmed this method could technically work, we wouldn't be complying with our standards.

jeremystretch commented 5 years ago

Please remember to :+1: this FR if you're in favor (or :-1: if not).

lampwins commented 5 years ago

Just to solidify my neutral stance on this. IMO the only real use case here is for multiple sites that have networking devices with the same name (e.g. core1 at each site).

I understand people have their own requirements driving their use of NetBox, but I strongly urge against using this feature in order to name rack furniture (so-called "dumb devices" in this thread) all the same thing. Naming a device "Cable Manager" serves no purpose other than to duplicate data and increase the chance of incorrect data (typo in the name) at scale. In such cases, the purpose of this device (and hence its "name") can readily be obtained from its device role and or device type.

I wonder if it would make sense to instead change the constraint to be unique per site?

jvanderaa commented 5 years ago

I'll add another example of a good place where allowing the same name on VMs is in a multi-tenancy environment. Let's say customers have an AD environment and the server names in vCenter and the like show up as say AD01, AD02. Then that is used across multiple locations. This would be the perfect place to allow VMs to have the same name.

amuckart commented 5 years ago

I definitely need this. We just evaluated netbox last week and opted to not use it since we have several hundred rack-mounted cable management units that we really do not need unique names for. We just want to give a reasonably descriptive name to each one such as the part number (or even go as generic as "cable manager"), but we can't do this with unique names being enforced.

I think a better solution to this problem would be to print the device role in the main rack view if the name is empty. That way you can have lots of unnamed cable management that still shows up with a sensible label.

jeremystretch commented 5 years ago

Related: #1966

amuckart commented 5 years ago

If this change does get made I would really like to be able to toggle the behavior. In my environment we have a lot of VMs and multiple different sites/rack groups/racks etc. but name uniqueness is baked into our naming scheme and duplicates are an error.

psuet commented 5 years ago

name uniqueness is baked into our naming scheme and duplicates are an error.

I also name cable managers etc. with unique names following a schema, so this should definitely be toggleable setting if implemented.

DanSheps commented 5 years ago

I think a better way to tackle this, is perhaps have name uniqueness with either a site/rack/rackgroup (or in a VM case a VM cluster).

I don't think allowing globally ununique names would be a good thing IMO.

berahtlv commented 5 years ago

I think a better way to tackle this, is perhaps have name uniqueness with either a site/rack/rackgroup (or in a VM case a VM cluster).

IMO uniqueness parameters should be obligatory, but racking of the equipment is not. For those who need the same names I see more as Context (additional object obligatory field), then name + context should be uniq. In that case devices or VMs can be located in the same site/rack.

DanSheps commented 5 years ago

I am not saying you need a rack, but base it on the rack and rack groups, this would give you a uniqueness constraint if it is in the rack, and if it is not you still have the constraint with just "null" as the rack/rack group.

I don't think adding context as an additional field would be a smart move. If you want to do that, you can just make the context part of your naming scheme.

yarnocobussen commented 5 years ago

@lampwins I agree with you on the naming of dumb devices. However, the dummy display name based on role and type is still there for objects without a name. Because of this, I do not see why this argument should result in such a feature not being implemented.

@amuckart @DanSheps If a duplicate name is disallowed only in some cases, what stops you from checking first that the name is not in use yet, within the boundaries that you have set for where this uniqueness is required?

To elaborate: Netbox users with add or change permissions on a field, are currently allowed to input data in any format supported by the field. Netbox cannot enforce company (naming) conventions for those fields on its own. Imho, if your company has strict conventions and workflows, you shouldn't be manually entering data into Netbox to begin with.

In our company, we're using Ansible Tower as our main automation tool. When combined with the Netbox API, it trivializes the enforcement of conventions and pre-checks for things such as duplicates.

In a world where some enter data manually and some don't, A global toggle seems the correct way to go about this. This same option exists for prefixes. Nice and consistent.

jbakklund commented 5 years ago

NetBox should preferably follow the ANSI/TIA-606-B and ISO/IEC TR14763 standards requiring each identifier to be unique. This is though not a requirement that applies to the text on a label, as the location of the label is (usually) obvious to the reader. The requirement for uniqueness on labels can therefore be limited to the perimeters of the site, building, room, or rack.

So, you are still in conformity with the standard e.g. by labelling a PDU only as 'PDU-A' or 'PDU-B' but the complete identifier listed in NetBox will still have to be unique e.g. by including a hierarchic list of prefixes based on e.g. country, site, building, room, row, rack, elevation, etc.

hSaria commented 5 years ago

As others have said, I would suggest that the uniqueness constraint for this and many other objects is based on the siblings.

This approach is very scalable, predictable, and hierarchical; just take the parent and replicate it – making sure that it is unique amongst its siblings, of course. An example of a hierarchy can be:

Devices are unique in the same rack
Racks are unique in the same rack group
Rack groups are unique in the same site

This concept is already implemented for many objects in NetBox. It just needs to be extended to the generic parent/child relationship model, rather than selectively to some places but not others.

Daniel-Dietz commented 5 years ago

I would need the non uniqueness for patch panels as they are labeled identical on both sides and I have now thrown the rack in there the make them unique. Not a great way to model this as I want to represent reality and not make it too different. Would it be possible to make this a configurable setting true or false?

ragzilla commented 5 years ago

Another thumbs up for non uniqueness for patch panels. Right now we have to label our panels in Netbox with rack/panel nomenclature which doesn't match the physical labelling of the panel. I'd love to see a uniqueness constraint within the rack as @hSaria proposed.

candlerb commented 5 years ago

Let me try to make this more concrete. People have proposed that device names might need to be unique:

Globally;
Within a site;
Within a rack;
Within a rackgroup.

If selecting one of these was a global choice, there are still a couple of edge cases to consider:

If naming is unique within a rack, what happens for devices which are in a site but not a rack? (One option is that such device names are only unique within the set of "no rack" devices in the site. Another option is that such device names would be unique across all racks in the site, i.e. cannot conflict with a device name in any other rack)
If naming is unique within a rackgroup, then as well as devices with no rack, what happens for devices which are in a rack but the rack is not in a rackgroup? (Maybe the set of "no rackgroup" racks in that site would be considered a de-facto rackgroup; or naming would be unique only within that rack. If there are no rackgroups in a site, this would equate to site-unique or rack-unique naming respectively)

Then there's the question of whether a global setting is general-purpose enough for everyone's requirements. Maybe you want different uniqueness settings for different sites, or different device roles? Maybe in one particular rackgroup, the naming needs to be unique per tenant?

In the most general case, you could make this selectable on a device-by-device basis: each device can have a drop-down for the naming scope ("global", "site-unique", "rack-unique" etc).

Now, if device A has a name which is "site-unique", and device B has a name which is "rack-unique", then obviously device B's rack is within a site, so names A and B have to conflict. Doing this in model logic isn't too hard. Doing it at the database level is a bit trickier - but welcome to postgres exclusion constraints. It gets worse with tenant, which can span sites. All seems rather complicated to me.

The other major issue around this is how to deal with CSV import and export, if device names are no longer unique.

If it's a global setting, then one possibility is that every import and export which includes a device name also has to include columns for site name or rackgroup name or rack name, as appropriate. So for example: if you had chosen to make device names site-unique, and you want to CSV-import IP Addresses with "device" and "interface_name" columns, then "site" would also have to be provided here as an extra column, and would be mandatory if "device" is not null.

Another option is to make the import/export names composite: e.g. in the device name column you can put site:rackgroup:rack:name. If you provide a bare name, and it's not unique, this would become an error. You could use a subset of those fields, e.g. site:::name, if that was sufficient to be unique. But beware that there might be one device "foo" within a rack, and another device "foo" not within a rack - that is, "no rack" and "unspecified rack" are different things. Maybe you'd need site:*:*:name or site:%:name for "unspecified rackgroup and rack".

Even so, I think I prefer composite names over having variable, extra mandatory columns.

Of course, you could simply store these composite names within the "name" field today, but it's not so pretty in the UI; also it doesn't automatically update the name if you change the site or rack of a device.

Another proposal was to make naming unique at sibling level only. This is easy to build: dcim_device has a uniqueness constraint on (name, rack_id) [^1]; dcim_rack has a uniqueness constraint on (name, group_id); dcim_rackgroup has a uniqueness constraint on (name, site_id). However this has the same CSV import/export problem. Also it doesn't help people who want to retain global uniqueness, or want site-wide unique names.

Taking the "sibling" approach to its limit, one might argue that site names only need to be unique within a region, but that's going too far I think.

[^1]: This still allows unnamed devices; nulls are not considered identical in postgres indexes.

Daniel-Dietz commented 5 years ago

@candlerb How about using Device Roles to define Uniqueness either Globally, Within a site, Within a rack, Within a rackgroup or not at at all. Going from there this could solve your edge cases.

candlerb commented 5 years ago

A related issue is around rack names. What form of uniqueness is currently enforced? At the database level I see:

\d dcim_rack
...
    "dcim_rack_group_id_facility_id_f16a53ae_uniq" UNIQUE CONSTRAINT, btree (group_id, facility_id)
    "dcim_rack_group_id_name_846f3826_uniq" UNIQUE CONSTRAINT, btree (group_id, name)

\d dcim_rackgroup
...
    "dcim_rackgroup_site_id_c9bd921f_uniq" UNIQUE CONSTRAINT, btree (site_id, name)

So the answer is: rack names are unique within a rackgroup, and rackgroup names are unique within a site. That is, LON1:room4:101 is a different rack to LON2:room4:101 (site:rackgroup:rack)

Aside: I think there is an error in this logic. A rack doesn't have to be associated with a rackgroup; ~~and if it's not, then as far as I can see, its name becomes globally unique, rather than site unique.~~

In postgres, a unique index does not consider nulls to be identical, so if there are two racks with the same name, this is permitted as long as they are not associated with a rackgroup.

Raised separately as #2883.

candlerb commented 5 years ago

I realised another issue with non-unique device names is when creating connections (i.e. cables).

The user interface for this is:

You navigate to the A end device
You click "connect" button next to the relevant interface
It displays the create cable page
You then have to select three things: the device (by entering part of its name and selecting from a dropdown of matches); the port type; and the port.

If device names are not unique, then at very least the devices will need to be labeled in in an unambiguous way (such as site:rackgroup:rack:name) so you can select the correct one.

Aside: it is almost certain that the cable you're creating is to a device in the same site; it would be helpful to restrict the list in this way, if that's not already done. And in many cases it will be within the same rack - but not always.

Therefore another approach is to add a dropdown for "B-end rack", showing all racks in the site, with A end's rack pre-selected. That would force you to select a different rack if the device you want is not in the same rack.

I think it is reasonable to do this, because when you're adding a physical cable, you almost certainly know which racks the two ends go between.

candlerb commented 5 years ago

@Daniel-Dietz:

How about using Device Roles to define Uniqueness either Globally, Within a site, Within a rack, Within a rackgroup or not at at all. Going from there this could solve your edge cases.

Cna you clarifying what you're proposing?

Suppose I have device roles "router" and "switch". Are you saying that I can configure role "router" should have globally unique names, but role "switch" should have rack-unique names? If so, the same issues I raised before will occur.

Or are you saying that each role should have its own namespace, so the uniqueness rules for routers and switches would be evaluated independently? That's a different proposal again. One consequence of that is, regardless of what uniqueness constraints I set for routers and switches, I could always have a router called "foo" and a switch called "foo". And therefore the device function rather than location becomes part of its unique identifier in exports etc. I don't like that: particularly if you have multiple roles (e.g. core switch, distribution switch, edge switch)

candlerb commented 5 years ago

Just for comparison, I had a look at what Snipe-IT does.

Locations can have parents, like Regions in Netbox. But Snipe-IT enforces global uniqueness of location names. If you create Room 101 in Site A, you can't have Room 101 in Site B. You have to give them globally unique names like "Site A-Room 101" and "Site B-Room 101". Hence this punts the issue of location naming uniqueness to the user.

But as for assets: names are optional, and can conflict. The thing which makes an asset unique is the asset tag, which is a mandatory unique field (which you can choose or can be auto-assigned). It doesn't care about names at all.

Netbox uses names differently:

the name acts as a sort of "functional name" rather than unique asset identifier - for example, if you swap out a faulty device, the replacement gets the same name as the original
the name is used as the identifier for CSV uploads.

DanSheps commented 5 years ago

Looking at the issues over the past few days, one thing that will need to change as a result of this, if this is accepted, is how devices and related fields are exported.

For example, inventory items. Currently inventory items is imported/exported by Device Name, however if device name is not unique, then the import will fail (results>1). The solution would be to introduce any other uniqueness parameters, however then those fields would need to be required in order to successfully import. Alternatively, move to some unique "slug" (pk is not viable as it is unique to the netbox instance only, so if you imported into a new instance you aren't guaranteed to be associating correctly. Asset Tag and Serial number are optional so currently they are not viable.

jeremystretch commented 5 years ago

One feature coming in v2.6 is the ability to reference related objects in API requests by nested attributes (#3077). I think it would be very cool if we could replicate this approach to bulk imports as well.

Because devices can be unnamed, CSV import fields which reference a device utilize our custom FlexibleModelChoiceField, which accepts either a name or a numeric primary key in the form {123}. It should be possible to extend this so that the field accepts any JSON (or JSON-like) dictionary that can be used as a queryset filter. The field would also interpret a bare string as a name by default. For example:

A value of router1 becomes Device.objects.get(name='router1')

A value of {123} becomes Device.objects.get(pk=123)

A value of {site.name: "North Arlen", name: "switch4"} becomes Device.objects.get(site__name='North Arlen', name='switch4')

This would allow a user to craft arbitrary references that are most suitable to the context within which they're importing objects. In the event the provided attributes return more than a single device (or they return zero devices), an exception is raised and validation fails.

Edit: I've opened #3147 to capture this proposal.

jeremystretch commented 5 years ago

After digesting all the feedback provided here, I think it makes sense to loosen the uniqueness requirement on device and VM names to the site/cluster level. I also think we should include tenant assignment, such that two different tenants can have a device with the same name in the same site. The modification I propose would look like this:

class Device:
    name = models.CharField(unique=False)  # No longer a unique field
    ...
    class Meta:
        constraints = [
            UniqueConstraint(
                fields=['site', 'tenant', 'name'],
                condition=Q(name__isnull=False)
            )
        ]

class VirtualMachine:
    name = models.CharField(unique=False)  # No longer a unique field
    ...
    class Meta:
        constraints = [
            UniqueConstraint(fields=['cluster', 'tenant', 'name'])
        ]

This should address the concerns raised regarding the use of "dumb names" (e.g. "Patch panel"), while still providing sufficient flexibility to satisfy most use cases. I should note that although some folks have suggested binding uniqueness to rack or rack group, this untenable as there is no requirement in NetBox for a device to be racked. There have also been proposals to implement some form of dynamic constraint, however we are not in a position to accommodate such a complex approach.

If there are no strong objections (and possibly even if there are), I'm going to introduce the change above, likely in the v2.6 release.

candlerb commented 5 years ago

Yes, I think it makes sense.

You didn't make it clear whether VMs would also have the tenant scope. (I am inclined that they shouldn't; a VM cluster probably won't be happy with two VMs with the same name. If you are running public hosting then the name is probably some unique instance identifier anyway)

For any CSV import which references a device/VM, you will need to ensure that it has the ability to disambiguate the device. Ones I can find:

IP address import currently has a device column but no site or tenant column, and virtual_machine but not cluster.
Inventory item import
Cable import
Secrets import

Those new columns could still be optional, but raise an error if multiple devices or virtual machines matched the given name.

The corresponding CSV exports also need extra columns: that is, any export which references a device and/or VM needs to have site/cluster/tenant columns. I haven't checked them all, but I note that Secrets export only gives device,role,name,plaintext.

jvanderaa commented 5 years ago

On the topic of VM names to sites/clusters, I would advocate for a per tenant option in there. In the particular scenario that I have at the moment is that some teams didn't want to track down to the sites or cluster level, and we have created a "catch all" cluster (and site). Obviously if this is the requirement we just have some work to do, but wanted to comment.

vosdev commented 4 years ago

A much needed improvement! Can finally rename my cable management / patch panels from "cable management 1", "cable management 2" to "cable management"!

DanSheps commented 4 years ago

Just as a FYI, you can not give cable management a name and just add it to the rack. It will be fine without a name.

marvi commented 4 years ago

This is not committed yet, right?

My use case is that I have a requirement to have a test site with the same ip/device names together with the production environment. We use tenants and VRFs to manage this. It would be really good if device and VM names could be non-unique or at least only unique within a tenant.

DanSheps commented 4 years ago

It has not been committed yet

lmgonzalezl commented 4 years ago

It would be interesting to have this possibility. In our case we have many patch panels (Device at the end) that have the same name only that located in different tenants.

regards

jeremystretch commented 4 years ago

After starting on this, I realized that this will be challenging to implement.

fields=['site', 'tenant', 'name']

As pointed out in this StackOverflow discussion, NULL does not equal NULL within the context of this evaluation. So for example, if we try to create two identical devices:

name,site,tenant
Router1,Site A,None
Router1,Site A,None

This will be allowed, as the lack of a tenant assignment for either device prevents us from violating the uniqueness constraint (i.e. NULL != NULL). As I understand it, this is a PostgreSQL limitation.

So, we have two options:

Implement custom validation (as proposed in the StackOverflow thread) to manually check for conflicting objects.
Further restrict the constraint to (name, site) OR (name, tenant). This is more flexible than the current model, but less flexible than what was proposed in this FR. For instance, two tenants can have identically-named devices in different sites, but not in the same site.

jeremystretch commented 4 years ago

I've submitted PR #3740, which takes option 1 (implementing manual uniqueness validation) as I don't think option 2 is very appealing.

candlerb commented 4 years ago

As I understand it, this is a PostgreSQL limitation

You can get the behaviour you want using something like

CREATE UNIQUE INDEX NameIndex ON dcim_device (COALESCE(site_id,0), COALESCE(tenant_id,0), name);

(also mentioned in #2883). Whether the ORM lets you apply this easily is another matter, but I presume a migration can use raw SQL?

netbox-community / netbox