Netbox importer: Exception while importing: ipam_ipaddress_nat_inside_id_a45fb7c5_fk_ipam_ipaddress_id

rubenjimenezit commented 4 months ago

Environment

Python version: Python 3.9.18
Nautobot version: Nautobot v2.2.0
nautobot-netbox-importer version: 2.0.0
Django version: 3.2.25

Expected Behavior

Data extracted from Netbox using command (several excludes were added to avoid errors found while importing in Nautobot): python3 /opt/netbox-3.7.0/netbox/manage.py dumpdata --traceback --format=json --exclude extras.ObjectChange --exclude extras.Script --exclude extras.Report --exclude django_rq --exclude extras.imageattachment --exclude social_django.usersocialauth --output=/tmp/netbox_data.json

Data expected to be imported into Nautobot

Observed Behavior

Data not imported Several "ImporterIssue" found like the following at the beginning of the import process: ImporterIssue(uid='e3ac8932-01d2-506a-afd1-bd5c4123950d', name='', issue_type='TypeError', message="'int' object is not iterable") or ImporterIssue(uid='98', name='', issue_type='ValueError', message='Content type id mismatch: 76 != 75') or ImporterIssue(uid='00c4393e-9fa4-563d-b411-876c8ef99e8e', name='', issue_type='SourceFieldImporterIssue', message="{'u_height': 'Invalid source value 0.5, truncated to 0'}")

Import process crashes with the following exception:

2024-05-22 14:17:19 [info     ] Created successfully           action=create diffs={'+': {'created': datetime.datetime(2022, 3, 17, 0, 0, tzinfo=datetime.timezone.utc), 'last_updated': None, 'name': 'vSphere'}} dst=<NautobotAdapter "Nautobot"> flags=<DiffSyncFlags.NONE: 0> model=virtualization_clustertype src=<NetBoxAdapter "NetBox"> status=success unique_id=710443dd-84a1-510f-9c56-163b76e6143b
2024-05-22 14:17:19 [debug    ] Attempting model create        action=create diffs={'+': {'created': datetime.datetime(2022, 3, 17, 0, 0, tzinfo=datetime.timezone.utc), 'last_updated': None, 'name': 'VM Host'}} dst=<NautobotAdapter "Nautobot"> flags=<DiffSyncFlags.NONE: 0> model=virtualization_clustergroup src=<NetBoxAdapter "NetBox"> unique_id=bf29b8b9-9601-5c0f-890b-fd123684f0ce
2024-05-22 14:17:19 [info     ] Created successfully           action=create diffs={'+': {'created': datetime.datetime(2022, 3, 17, 0, 0, tzinfo=datetime.timezone.utc), 'last_updated': None, 'name': 'VM Host'}} dst=<NautobotAdapter "Nautobot"> flags=<DiffSyncFlags.NONE: 0> model=virtualization_clustergroup src=<NetBoxAdapter "NetBox"> status=success unique_id=bf29b8b9-9601-5c0f-890b-fd123684f0ce
2024-05-22 14:17:19 [info     ] Sync complete                  dst=<NautobotAdapter "Nautobot"> flags=<DiffSyncFlags.NONE: 0> src=<NetBoxAdapter "NetBox">
Traceback (most recent call last):
  File "/opt/nautobot/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
psycopg2.errors.ForeignKeyViolation: insert or update on table "ipam_ipaddress" violates foreign key constraint "ipam_ipaddress_nat_inside_id_a45fb7c5_fk_ipam_ipaddress_id"
DETAIL:  Key (nat_inside_id)=(648d4310-a261-54f8-a786-64ae06dca166) is not present in table "ipam_ipaddress".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/nautobot/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot/core/cli/__init__.py", line 286, in main
    execute_from_command_line([sys.argv[0], *unparsed_args])
  File "/opt/nautobot/lib64/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/opt/nautobot/lib64/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/nautobot/lib64/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/nautobot/lib64/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_netbox_importer/management/commands/import_netbox.py", line 89, in handle
    adapter.import_to_nautobot()
  File "/opt/nautobot/lib64/python3.9/site-packages/nautobot_netbox_importer/diffsync/adapters/netbox.py", line 97, in import_to_nautobot
    self._atomic_import()
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/nautobot/lib64/python3.9/site-packages/django/db/transaction.py", line 246, in __exit__
    connection.commit()
  File "/opt/nautobot/lib64/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/opt/nautobot/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 266, in commit
    self._commit()
  File "/opt/nautobot/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
  File "/opt/nautobot/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 242, in _commit
    return self.connection.commit()
django.db.utils.IntegrityError: insert or update on table "ipam_ipaddress" violates foreign key constraint "ipam_ipaddress_nat_inside_id_a45fb7c5_fk_ipam_ipaddress_id"
DETAIL:  Key (nat_inside_id)=(648d4310-a261-54f8-a786-64ae06dca166) is not present in table "ipam_ipaddress".

Steps to Reproduce

In Netbox. Export data with command: nautobot-server import_netbox --bypass-data-validation --save-json-summary-path /tmp/netbox_summary.json --save-text-summary-path /tmp/netbox_summary.txt /tmp/netbox_data.json >> /tmp/netbox_clioutput.txt 2>> /tmp/netbox_clioutput.txt
Copy netbox_data.json file to Nautobot machine
In Nautobot. Import data with command: python3 /opt/netbox-3.7.0/netbox/manage.py dumpdata --traceback --format=json --exclude extras.ObjectChange --exclude extras.Script --exclude extras.Report --exclude django_rq --exclude extras.imageattachment --exclude social_django.usersocialauth --output=/tmp/netbox_data.json

gsnider2195 commented 1 month ago

There are multiple issues being reported here so I'm going to address each of them below.

`ImporterIssue(uid='98', name='', issue_type='ValueError', message='Content type id mismatch: 76 != 75')`

I was able to replicate this by changing the order of the content types in my netbox dump. This code is relying on the content types being provided in ascending order in the dump file. @snaselj is this intentional?

`ImporterIssue(uid='00c4393e-9fa4-563d-b411-876c8ef99e8e', name='', issue_type='SourceFieldImporterIssue', message="{'u_height': 'Invalid source value 0.5, truncated to 0'}")`

This appears to be working as intended. A feature request is open for half-height devices here but for now this needs to be changed on the source.

`DETAIL: Key (nat_inside_id)=(648d4310-a261-54f8-a786-64ae06dca166) is not present in table "ipam_ipaddress".`

There's probably another issue preventing the referenced IP Address from being imported but it's not provided in the issue report. If you still have access to the data can you provide the complete log?

snaselj commented 1 month ago

This code is relying on the content types being provided in ascending order in the dump file. @snaselj is this intentional?

@gsnider2195

The reason for this was to generate NetBox content type IDs based on the order in the source file if IDs were not provided in the source file. This can probably be updated just to issue a warning instead of raising an exception, in case, the content type ID is provided.

rubenjimenezit commented 3 weeks ago

gsnider2195

Thanks for checking on this. We are now trying onboarding app 4.0 so we can get the info directly from the networking devices. Still a task in progress by now :)

gsnider2195 commented 3 weeks ago

@rubenjimenezit please let us know if you try to use netbox importer again and still encounter this problem. I'm going to close this issue now since I was unable to reproduce the issue but if you can provide example json data that reproduces the issue we will reopen it. Thanks!

nautobot / nautobot-app-netbox-importer