Add support for importing NetBox 2.11.x data

[bsmeding]

Environment

• Nautobot version: 1.0.1
• nautobot-netbox-importer version: 1.4.0

Proposed Functionality

Get support for an exported json file of Netbox 2.11.x

Use Case

Currently is is not working when setting the version to 2.10.8

(nautobot) nautobot@debian:~$ nautobot-server import_netbox_json netbox_120721.json 2.10.8
05:42:36 info     Loading NetBox JSON data into memory...
  filename: {'netbox_120721.json'}
Traceback (most recent call last):
  File "/opt/nautobot/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/opt/nautobot/lib/python3.7/site-packages/nautobot/core/cli.py", line 62, in main
    initializer=_configure_settings,  # Called after defaults
  File "/opt/nautobot/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/opt/nautobot/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/nautobot/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/nautobot/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/nautobot/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobot/lib/python3.7/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 40, in handle
    data = json.load(options["json_file"])
  File "/usr/lib/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 3016066 (char 3016065)

[glennmatthews]

Thanks for the report!

This doesn't look like with the importer itself; that's Python saying that your NetBox JSON dump isn't a valid JSON file. Can you check the file and make sure it wasn't truncated or something like that?

[bsmeding]

Yes probably you're right, the last characters are null, "tagged_vlans": []}} so it didnt close correct ] Is there a limit on items/devices?

[glennmatthews]

File size shouldn't be a limit, but available RAM can be when running the importer. Can you try re-exporting the data from NetBox?

[sklemer1]

To get back to the title of this issue -- In order to import 2.11 changes data we had to change the exported json:

jq '[ .[] | if (.fields.postchange_data != null) then .fields.object_data = .fields.postchange_data else .fields.object_data = .fields.prechange_data end | del(.fields.prechange_data, .fields.postchange_data) ]'

This was straight forward according to the changes from netbox 2.10 to 2.11. (just put postchange_data into object_data except for cases where there is no postchange_change (e.g. deletes) where we take it from prechange_data).

But it happened that there were still fields with object_data=null which I cannot explain and just deleted with

`` jq '[ .[] | select(.fields.object_data != null)]'

[glennmatthews]

Adding known NetBox changes that would need to be accommodated for full 2.11 support:

• Added ProviderNetwork model
  • Should be straightforward so long as we update the minimum Nautobot supported version to at least 1.3, as this model was ported more-or-less-directly into Nautobot 1.3
• Added SiteGroup model
  • There's no direct equivalent to this model in Nautobot at this time, so for the time being it will probably just need to be explicitly ignored by the importer, perhaps with a warning message if SiteGroup data is present in the source.
• Added JournalEntry model
  • Nautobot 1.4 and later has a Note model, it should be relatively straightforward to translate one to the other.
• Renamed RackGroup to Location
  • This needs to still map to Nautobot's RackGroup model; Nautobot Locations are a different thing.
  • Renaming of Rack.group to Rack.location also needs to be accounted for.
• As noted above, the change to NetBox's ObjectChange model that replaced object_data with separate prechange_data and postchange_data fields.
  • Nautobot made a similar enhancement in version 1.3 with the addition of object_data_v2 field in https://github.com/nautobot/nautobot/pull/1389; some translation between the two features will be necessary.
• Change from VLAN.site foreign-key to VLAN.scope generic-foreign-key, permitting scoping of VLANs to Regions, SiteGroups, ~RackGroups~ ("Locations") and Racks as well as Sites.
  • There's no direct equivalent to this feature in Nautobot at this time, although this may be possible to at least partially accommodate in Nautobot 2.0 when Region and Site become special cases of Nautobot's Location model.
  • In the short term we might just discard any VLANs that are scoped to anything other than a Site.
  • As a more complex piece of work, we can translate VLANs scoped to RackGroups/Locations and Racks back to scoping to the parent Site of these records (although we may have to account for potential deduplication of data to satisfy uniqueness constraints).
  • Conversely, for VLANs scoped to Regions or SiteGroups, we might want to duplicate those VLANs to each of their descendant Sites.
• Device.location field
  • No direct equivalent in Nautobot, probably should ignore/discard this field for now
• Cable.mark_connected field
  • No direct equivalent in Nautobot, ignore/discard
• Interface.parent field
  • Equivalent functionality is present in Nautobot 1.4 and later, needs to be supported in the importer.

[chadell]

Drop in favor of NetBox 3.x