"nb-dt-import.py --vendors d-link" throws "Error creating manufacturers"

[basictheprogram]

netbox-community 7969f4a05cf1 (v3.4.6) running in Docker

$ docker --version Docker version 23.0.1, build a5ee5b1

I do keep pin postgres:14-apline

cat docker-compose.override.yml
version: '3.4'
services:
  netbox:
    ports:
      - 8000:8080
    restart: unless-stopped
  postgres:
    image: postgres:14-alpine

I run the nb-dt-import.py on macOS

(venv) $ sw_vers ProductName: macOS ProductVersion: 12.6.3 BuildVersion: 21G419

(venv) $ python --version Python 3.9.16

(venv) $ nb-dt-import.py --vendors d-link
Package devicetype-library is already installed, updating /Users/tanner/projects/netbox/Netbox-Device-Type-Library-Import/repo
1 Vendors Found
3 Device-Types Found
Error creating manufacturers
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DES-3200-52
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DES-1210-28
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DGS-1210-52
Modules Enabled. Creating Modules...
0 Module Vendors Found
0 Module-Types Found
---
0 devices created
0 interfaces/ports updated
0 manufacturers created

[danner26]

Hey @basictheprogram, since we released v2.0.0 of the import script, I just want to confirm which version you are on.

Could you let me know if the import repo is up to date?

[basictheprogram]

Is there a --version or specific way to get the script release?

The import script I use is from the master branch on github, identified as upstream but my origin is synced with upstream.

git remote -v
origin  git@github.com:basictheprogram/Netbox-Device-Type-Library-Import.git (fetch)
origin  git@github.com:basictheprogram/Netbox-Device-Type-Library-Import.git (push)
upstream    https://github.com/minitriga/Netbox-Device-Type-Library-Import (fetch)
upstream    https://github.com/minitriga/Netbox-Device-Type-Library-Import (push)

git pull upstream master
From https://github.com/minitriga/Netbox-Device-Type-Library-Import
 * branch            master     -> FETCH_HEAD
Already up to date.

And my repo

git remote -v
origin  git@github.com:basictheprogram/devicetype-library.git (fetch)
origin  git@github.com:basictheprogram/devicetype-library.git (push)
upstream    https://github.com/netbox-community/devicetype-library.git (fetch)
upstream    https://github.com/netbox-community/devicetype-library.git (push)

git pull upstream master
From https://github.com/netbox-community/devicetype-library
 * branch            master     -> FETCH_HEAD
Already up to date.

[danner26]

A --version is a good idea to add, the old version does not have that. I'll get that added to the new version

That being said, it looks like you are using a fork of the old upstream device type library import repo which was previously a personal repository. It is now managed under the NetBox community organization

To get the latest release, I believe you can either:

1. Sync changes on your personal fork (from the UI or via the cli), GitHub should be smart enough to pull them from the new location (under NetBox Community, since we migrated the repo rather than creating a new copy. If this fails go to option 2)
2. Clone the repo from the new location and merge your configuration details into it

I suggest you do option 2, because this option will allow you to acquire new releases simply by executing a git pull rather than syncing the upstream and then running the pull command

[danner26]

Once you are on the new release, let me know if the issue persists. Feel free to comment back if you have any issues!

[basictheprogram]

Documentation bug at https://github.com/netbox-community/Device-Type-Library-Import?

Getting Started states git clone https://github.com/minitriga/Netbox-Device-Type-Library-Import

Confirm this is a typo. If so, I will submit a PR to fix it.

[danner26]

Yup that's an obvious overlook on my part. I would appreciate the PR!

https://github.com/netbox-community/Device-Type-Library-Import.git

[basictheprogram]

Same problem using all the right GitHub repos. I let nb-dt-import pull repo.

(venv) $ git remote -v
origin  git@github.com:basictheprogram/Netbox-Device-Type-Library-Import.git (fetch)
origin  git@github.com:basictheprogram/Netbox-Device-Type-Library-Import.git (push)
upstream    https://github.com/netbox-community/Device-Type-Library-Import.git (fetch)   # community repo
upstream    https://github.com/netbox-community/Device-Type-Library-Import.git (push)    # community repo

(venv) $ git pull upstream master
From https://github.com/netbox-community/Device-Type-Library-Import
 * branch            master     -> FETCH_HEAD
Already up to date.

(venv) $ file repo
repo: cannot open `repo' (No such file or directory)
(venv) $ ./nb-dt-import.py --vendors d-link
Package Installed https://github.com/netbox-community/devicetype-library.git
1 Vendors Found
3 Device-Types Found
Error creating manufacturers
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DES-3200-52
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DES-1210-28
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DGS-1210-52
Modules Enabled. Creating Modules...
0 Module Vendors Found
0 Module-Types Found
---
0 devices created
0 interfaces/ports updated
0 manufacturers created

[basictheprogram]

Thought maybe the problem was just the D-Link manufacturer. Did a full run of nb-dt-import.py and several manufacturers have problems. Maybe it's my instance of Netbox? Missed some database migration(s)?

Problematic Manufacturers

``` ./nb-dt-import.py Package devicetype-library is already installed, updating /Users/tanner/projects/netbox/Netbox-Device-Type-Library-Import/repo 103 Vendors Found 2215 Device-Types Found Error creating manufacturers Error '[{"__all__":["Subdevice role of device type (RS1619xs+) must be set to \"parent\" to allow device bays."]}]' creating Device Bay Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: Ubiquiti UniFi Cloud Key Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'Neousys', 'slug': 'neousys'}"]} creating device type: Neousys Nuvo-8240GC Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: APC Smart-UPS SMT1500RM2U Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: APC Smart-UPS SMT1500 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: APC Smart-UPS SRT3000RMXLT Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'Albis-Elcon', 'slug': 'albis-elcon'}"]} creating device type: Albis-Elcon BIG2862 Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'Albis-Elcon', 'slug': 'albis-elcon'}"]} creating device type: Albis-Elcon BIG4862 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: A10 Thunder 6430 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: A10 Thunder 4440 ADC Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'EnGenius', 'slug': 'engenius'}"]} creating device type: EnGenius EWS1200-28TFP Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'EnGenius', 'slug': 'engenius'}"]} creating device type: EnGenius EWS5912FP Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'EnGenius', 'slug': 'engenius'}"]} creating device type: EnGenius EWS360AP Error {"front_image":["The submitted data was not a file. Check the encoding type on the form."],"rear_image":["The submitted data was not a file. Check the encoding type on the form."]} creating device type: Nokia 7220 IXR-H3 32QSFPDD 2SFP+ Error '[{"__all__":["Subdevice role of device type (PowerVault MD3200) must be set to \"parent\" to allow device bays."]},{"__all__":["Subdevice role of device type (PowerVault MD3200) must be set to \"parent\" to allow device bays."]}]' creating Device Bay Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: Dell PowerEdge T110 Error '[{"__all__":["Subdevice role of device type (PowerVault ME4024) must be set to \"parent\" to allow device bays."]},{"__all__":["Subdevice role of device type (PowerVault ME4024) must be set to \"parent\" to allow device bays."]}]' creating Device Bay Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DES-3200-52 Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DES-1210-28 Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'D-Link', 'slug': 'd-link'}"]} creating device type: D-Link DGS-1210-52 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: ADVA FSP 150-XG418-100G-CFP2 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: ADVA FSP 150-XG418-100G Modules Enabled. Creating Modules... 31 Module Vendors Found 290 Module-Types Found --- 9 devices created 146 interfaces/ports updated 0 manufacturers created ```

[danner26]

Absolutely. It looks like you already have a bunch of devices in your NetBox instance. Currently the script isn't able to update existing devices, but that is coming down the line.

If you try deleting one of the offending devices in NetBox and then reimporting it, it should solve the issue. Otherwise you can manually fix the difference and try again

[basictheprogram]

Netbox cannot find any of the problematic devices. I know there are several D-Link devices that are causing problems but the manufacturer report show zero devices associated with the D-Link manufacturer

I know a problematic device is DES-1210-28, searching for that name in Netbox does not find the device

Tried psql (command line) and getting some really unusual results?

I find the D-Link manufacturer

SELECT * FROM public.dcim_manufacturer WHERE name LIKE '%D-link%';

"2022-07-12 22:31:50.799663+00" "2023-03-25 16:37:16.584804+00" "{}"    60  "D-link"    "d-link"    

But I cannot find any of the D-Link device_types

SELECT * FROM public.dcim_devicetype WHERE model LIKE '%DES%';
SELECT * FROM public.dcim_devicetype WHERE model LIKE '%DGS%';

I do not know how to delete the offending device(s) as I think the error is the script cannot insert the devices.

[danner26]

Sorry about that, I wasn't clear with my last message. The D-Link devices seem problematic, I will look into that

Specifically the devices like the SRT3000RMXLT which say they already exist are the ones I was referring to

I am on mobile right now, so I'll take a deeper look later on

[basictheprogram]

SRT3000RMXLT is found as a Device Type and Manufacturer on my Netbox

The Manufacturers version shows an interesting Changelog, created 2022-07-12 but updated 5 minutes ago (which is approximately when I last ran the script)

Deleted the SRT3000RMXLT "Manufacturer" and now things work for --vendors d-link!

./nb-dt-import.py --vendors d-link
Package devicetype-library is already installed, updating /Users/tanner/projects/netbox/Netbox-Device-Type-Library-Import/repo
1 Vendors Found
3 Device-Types Found
Modules Enabled. Creating Modules...
0 Module Vendors Found
0 Module-Types Found
---
3 devices created
146 interfaces/ports updated
1 manufacturers created

[basictheprogram]

Still have problems with a full run of the script. This example is EnGenius:

Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'EnGenius', 'slug': 'engenius'}"]} creating device type: EnGenius EWS1200-28TFP
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'EnGenius', 'slug': 'engenius'}"]} creating device type: EnGenius EWS5912FP
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'EnGenius', 'slug': 'engenius'}"]} creating device type: EnGenius EWS360AP

Searching my Netbox for engenius comes back with zero results. But --vendors works? Fixes the problem?

nb-dt-import.py --vendors engenius

``` ./nb-dt-import.py --vendors engenius Package devicetype-library is already installed, updating /Users/tanner/projects/netbox/Netbox-Device-Type-Library-Import/repo 1 Vendors Found 3 Device-Types Found Modules Enabled. Creating Modules... 0 Module Vendors Found 0 Module-Types Found --- 3 devices created 47 interfaces/ports updated 1 manufacturers created ```

The next full run fails, but the engenius errors are gone

nb-dt-import.py (full run)

``` ./nb-dt-import.py Package devicetype-library is already installed, updating /Users/tanner/projects/netbox/Netbox-Device-Type-Library-Import/repo 103 Vendors Found 2215 Device-Types Found Error creating manufacturers Error '[{"__all__":["Subdevice role of device type (RS1619xs+) must be set to \"parent\" to allow device bays."]}]' creating Device Bay Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: Ubiquiti UniFi Cloud Key Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'Neousys', 'slug': 'neousys'}"]} creating device type: Neousys Nuvo-8240GC Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: APC Smart-UPS SMT1500RM2U Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: APC Smart-UPS SMT1500 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: APC Smart-UPS SRT3000RMXLT Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'Albis-Elcon', 'slug': 'albis-elcon'}"]} creating device type: Albis-Elcon BIG2862 Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'Albis-Elcon', 'slug': 'albis-elcon'}"]} creating device type: Albis-Elcon BIG4862 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: A10 Thunder 6430 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: A10 Thunder 4440 ADC Error {"front_image":["The submitted data was not a file. Check the encoding type on the form."],"rear_image":["The submitted data was not a file. Check the encoding type on the form."]} creating device type: Nokia 7220 IXR-H3 32QSFPDD 2SFP+ Error '[{"__all__":["Subdevice role of device type (PowerVault MD3200) must be set to \"parent\" to allow device bays."]},{"__all__":["Subdevice role of device type (PowerVault MD3200) must be set to \"parent\" to allow device bays."]}]' creating Device Bay Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: Dell PowerEdge T110 Error '[{"__all__":["Subdevice role of device type (PowerVault ME4024) must be set to \"parent\" to allow device bays."]},{"__all__":["Subdevice role of device type (PowerVault ME4024) must be set to \"parent\" to allow device bays."]}]' creating Device Bay Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: ADVA FSP 150-XG418-100G-CFP2 Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: ADVA FSP 150-XG418-100G Modules Enabled. Creating Modules... 31 Module Vendors Found 290 Module-Types Found --- 0 devices created 0 interfaces/ports updated 0 manufacturers created ```

Not a cure all. Missing devices for ADVA

Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: ADVA FSP 150-XG418-100G-CFP2
Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: ADVA FSP 150-XG418-100G

Even --vendors adva doesn't work

./nb-dt-import.py --vendors  adva
Package devicetype-library is already installed, updating /Users/tanner/projects/netbox/Netbox-Device-Type-Library-Import/repo
1 Vendors Found
6 Device-Types Found
Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: ADVA FSP 150-XG418-100G-CFP2
Error {"__all__":["Device type with this Manufacturer and Slug already exists."]} creating device type: ADVA FSP 150-XG418-100G
Modules Enabled. Creating Modules...
0 Module Vendors Found
0 Module-Types Found
---
0 devices created
0 interfaces/ports updated
0 manufacturers created

Could there be a limit on the length of the slug that's being checked?

slug: xg418-100g-cfp2 slug: xg418-100g

Or is this about your comment that the script cannot update things?

[basictheprogram]

Was able to fix the Albis-Elcon issues with

(venv)$ nb-dt-import.py --vendors   Albis-Elcon

[basictheprogram]

Fixed the Neousys issue by deleting the Manufacturer with in Netbox and running the script

(venv)$ nb-dt-import.py --vendors   Neousys
Package devicetype-library is already installed, updating /Users/tanner/projects/netbox/Netbox-Device-Type-Library-Import/repo
1 Vendors Found
1 Device-Types Found
Modules Enabled. Creating Modules...
0 Module Vendors Found
0 Module-Types Found
---
1 devices created
8 interfaces/ports updated
1 manufacturers created

[danner26]

Hey @basictheprogram the majority of the issues you are going to run into are related to pre-existing items in NetBox. As mentioned before we cannot currently update a device/model type so the only way to fix this is by deleting/readding the devices. Also as you saw, the manufacturer names need to be written correctly for it to work.

Is everything working for you now? Since this also doesnt seem like an issue with the devicetype-library itself, I suggest we move this over to the (Device-Type-Library-Import)[https://github.com/netbox-community/Device-Type-Library-Import] if you need further assistance