SSOT Upgrade failed

[ghost]

Environment

• Python version:
• Nautobot version: 2.0.5
• nautobot-ssot version: 2.0.[01] -> 2.0.2

Expected Behavior

nautobot-server post_upgrade fails when upgrading ssot from 2.0.0 to 2.0.2

Observed Behavior

nautobotv2@sa-prod-integration-sd-78:~$ nautobot-server post_upgrade
13:36:50.882 DEBUG   nautobot.ssot        __init__.py                  _add_integrations() :
  Registering job <class 'nautobot_ssot.integrations.ipfabric.jobs.IpFabricDataSource'> from /opt/nautobotv2/nautobot-plugin-ssot/nautobot_ssot/integrations/ipfabric/jobs.py
13:36:50.882 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot_ssot.integrations.ipfabric.jobs.IpFabricDataSource
13:36:50.895 DEBUG   nautobot.ssot        utils.py        each_enabled_integration_module() :
  Integration ipfabric does not have a urls module, skipping.
Grafana ChatOps integration is not available.
NoneType: None
13:36:51.234 DEBUG   nautobot.core.celery __init__.py        import_jobs_as_celery_tasks() :
  Importing system Jobs
13:36:51.235 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositorySync
13:36:51.235 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositoryDryRun
13:36:52.167 DEBUG   nautobot.ssot        __init__.py                              ready() :
  Registering signals for /opt/nautobotv2/nautobot-plugin-ssot/nautobot_ssot/integrations/ipfabric/signals.py
Performing database migrations...
Operations to perform:
  Apply all migrations: admin, auth, circuits, contenttypes, database, dcim, django_celery_beat, django_celery_results, extras, ipam, nautobot_chatops, nautobot_ssot, sessions, social_django, taggit, tenancy, users, virtualization
Running migrations:
  No migrations to apply.
13:36:54.611 DEBUG   nautobot.core.celery __init__.py        import_jobs_as_celery_tasks() :
  Importing system Jobs
13:36:54.621 INFO    nautobot.extras.utils utils.py        refresh_job_model_from_job_class() :
  Refreshed Job "SSoT - IPFabric: IPFabric ⟹ Nautobot" from <IpFabricDataSource>
13:36:54.628 INFO    nautobot.extras.utils utils.py        refresh_job_model_from_job_class() :
  Refreshed Job "System Jobs: Git Repository: Sync" from <GitRepositorySync>
13:36:54.636 INFO    nautobot.extras.utils utils.py        refresh_job_model_from_job_class() :
  Refreshed Job "System Jobs: Git Repository: Dry-Run" from <GitRepositoryDryRun>
Traceback (most recent call last):
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/query.py", line 581, in get_or_create
    return self.get(**kwargs), False
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
__fake__.DoesNotExist: CustomField matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "extras_customfield_slug_ed27c4fe_uniq"
DETAIL:  Key (key)=(last_synced_from_sor) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/nautobotv2/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/opt/nautobotv2/lib/python3.9/site-packages/nautobot/core/cli/__init__.py", line 54, in main
    run_app(
  File "/opt/nautobotv2/lib/python3.9/site-packages/nautobot/core/runner/runner.py", line 297, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobotv2/lib/python3.9/site-packages/nautobot/core/management/commands/post_upgrade.py", line 100, in handle
    call_command(
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/__init__.py", line 181, in call_command
    return command.execute(*args, **defaults)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 268, in handle
    emit_post_migrate_signal(
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/core/management/sql.py", line 42, in emit_post_migrate_signal
    models.signals.post_migrate.send(
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/opt/nautobotv2/lib/python3.9/site-packages/nautobot/core/apps/__init__.py", line 798, in post_migrate_send_nautobot_database_ready
    nautobot_database_ready.send(sender=app_conf, app_config=app_conf, **kwargs)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/opt/nautobotv2/nautobot-plugin-ssot/nautobot_ssot/integrations/ipfabric/signals.py", line 80, in nautobot_database_ready_callback
    create_custom_field("last_synced_from_sor", "Last sync from System of Record", synced_from_models, apps=apps)
  File "/opt/nautobotv2/nautobot-plugin-ssot/nautobot_ssot/integrations/ipfabric/signals.py", line 29, in create_custom_field
    custom_field, _ = CustomField.objects.get_or_create(
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/query.py", line 588, in get_or_create
    return self.create(**params), True
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobotv2/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "extras_customfield_slug_ed27c4fe_uniq"
DETAIL:  Key (key)=(last_synced_from_sor) already exists.

Steps to Reproduce

1. Upgrade ssot plugin to 2.0.2 from 2.0.0 or 2.0.1

[ghost]

rollback to v2.0.1 works.

[ghost]

deleted last_synced_from_sor and reran

[glennmatthews]

Likely the use of CustomField.objects.get_or_create() in create_custom_field needs to be adapted, perhaps something like:

        custom_field, _ = CustomField.objects.get_or_create(
            key=key,
            type=CustomFieldTypeChoices.TYPE_TEXT,
            defaults={"label": label},
        )

so that a difference in label doesn't result in trying to create a custom field instance with a duplicate key.

[whitej6]

Issue likely caused by one of the following scenarios

1. CustomField was previously manually created with the same key
2. Another app published a CustomField with the same key, verified there are no conflicts in this app
3. Upgraded to 2.0.2 which created the CustomField in Nautobot then was edited causing the conflict.

I agree with @glennmatthews comment, but could lead to a split brain scenario with 2 apps publishing conflicting CustomFields with different definitions. There is a roadmap item to revisit publishing CustomFields with official apps for issues like this.

[jdrew82]

@justinjeffery-ipf Do you have any other SSoT Apps integrated into your Nautobot environment? Just trying to see if it is possible that we have another App potentially creating this CustomField before I try to implement the fix that @glennmatthews mentioned.

[ghost]

Nope maybe it was because I was testing a different branch now that I think of it?

[jdrew82]

It is quite possible if you had done an update or change to use a different branch that had the same CustomField could cause a conflict, yes.