search

netbox-community / netbox

The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/free-netbox-cloud/
http://netboxlabs.com/oss/netbox/
Apache License 2.0
15.77k stars 2.54k forks source link

ProgrammingError exception when upgrading to v4.0 #15605

Closed opericgithub closed 5 months ago

opericgithub commented 5 months ago

Deployment Type

Self-hosted

NetBox Version

v3.7.4 and v3.6.9

Python Version

3.10

Steps to Reproduce

I'm not sure whether this is the right place, but please correct me if I should post this issue in some other place. I tried to do the upgrade from v3.7.4 as well as from v3.6.9, to the new v4.0.x (x = either feature, or beta).

First I tried upgrading to feature branch few days ago, and now I tried upgrading to v4.0-beta1. I was using the following commands:

cd /opt/netbox
sudo git pull origin v4.0-beta1
sudo ./upgrade.sh

Then I get the following error messages:

Installing collected packages: pyasn1, pyasn1-modules, python-ldap, django-auth-ldap
Successfully installed django-auth-ldap-4.7.0 pyasn1-0.6.0 pyasn1-modules-0.4.0 python-ldap-3.4.4
Applying database migrations (python3 netbox/manage.py migrate)...
Operations to perform:
  Apply all migrations: account, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, vpn, wireless
Running migrations:
  Applying circuits.0043_circuittype_color... OK
  Applying core.0006_datasource_type_remove_choices... OK
  Applying core.0007_job_add_error_field... OK
  Applying core.0008_contenttype_proxy... OK
  Applying core.0009_configrevision... OK
  Applying core.0010_gfk_indexes... OK
  Applying dcim.0183_devicetype_exclude_from_utilization... OK
  Applying dcim.0184_protect_child_interfaces... OK
  Applying dcim.0185_gfk_indexes... OK
  Applying dcim.0186_location_facility... OK
  Applying extras.0099_cachedvalue_ordering... OK
  Applying extras.0100_customfield_ui_attrs... OK
  Applying extras.0101_eventrule... OK
  Applying extras.0102_move_configrevision... OK
  Applying extras.0103_gfk_indexes... OK
  Applying extras.0104_stagedchange_remove_change_logging... OK
  Applying extras.0105_customfield_min_max_values... OK
  Applying extras.0106_bookmark_user_cascade_deletion... OK
  Applying extras.0107_cachedvalue_extras_cachedvalue_object... OK
  Applying extras.0108_convert_reports_to_scripts... OK
  Applying extras.0109_script_model... OK
  Applying extras.0110_remove_eventrule_action_parameters... OK
  Applying extras.0111_rename_content_types...Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
  File "/opt/netbox/venv/lib/python3.10/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedTable: relation "extras_customfield_content_types_id_seq" does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 106, in database_forwards
    self._run_sql(schema_editor, self.sql)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 133, in _run_sql
    schema_editor.execute(statement, params=None)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/postgresql/schema.py", line 45, in execute
    return super().execute(sql, params)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 201, in execute
    cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
  File "/opt/netbox/venv/lib/python3.10/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: relation "extras_customfield_content_types_id_seq" does not exist

Am I doing something wrong? Is there anything else I should check for? I am also afraid that I won't be able to do the upgrade to the final version v4.0 in few weeks.

Expected Behavior

I expect to upgrade to the new (beta) version, as I was able to do this in the past using the same commands.

Observed Behavior

I get the error: django.db.utils.ProgrammingError: relation "extras_customfield_content_types_id_seq" does not exist Detailed log messages are given above.

jeremystretch commented 5 months ago

I tried to do the upgrade from v3.7.4 as well as from v3.6.9, to the new v4.0.x

You cannot upgrade from v3.6 directly to v4.0: Per the documentation you'll need to first upgrade to v3.7 to ensure all migrations are applied prior to being squashed.

NetBox can generally be upgraded directly to any newer release with no interim steps, with the one exception being incrementing major versions. This can be done only from the most recent minor release of the major version.

However, it should work fine when upgrading from v3.7.x. On a clean install of NetBox v3.7.4 (starting with an empty database), I have extras_customfield_content_types_id_seq:

netbox=> \d extras_customfield_content_types_id_seq
          Sequence "public.extras_customfield_content_types_id_seq"
  Type  | Start | Minimum |       Maximum       | Increment | Cycles? | Cache 
--------+-------+---------+---------------------+-----------+---------+-------
 bigint |     1 |       1 | 9223372036854775807 |         1 | no      |     1
Sequence for identity column: public.extras_customfield_content_types.id

And after switching to the v4.0-beta code base, all migrations apply successfully:

$ ./manage.py migrate
Operations to perform:
  Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, vpn, wireless
Running migrations:
  Applying dcim.0186_location_facility... OK
  Applying extras.0108_convert_reports_to_scripts... OK
  Applying extras.0109_script_model... OK
  Applying extras.0110_remove_eventrule_action_parameters... OK
  Applying extras.0111_rename_content_types... OK
  Applying extras.0112_tag_update_object_types... OK
  Applying extras.0113_customfield_rename_object_type... OK
  Applying tenancy.0015_contactassignment_rename_content_type... OK
  Applying users.0005_alter_user_table... OK
  Applying users.0006_custom_group_model... OK
  Applying users.0007_objectpermission_update_object_types... OK
  Applying users.0008_flip_objectpermission_assignments... OK
  Applying vpn.0005_rename_indexes... OK

With your original v3.7.4 database in place, do you see extras_customfield_content_types_id_seq defined? (You can use manage.py dbshell to enter the PostgreSQL command prompt and type \d to list all tables.)

opericgithub commented 5 months ago

With this:

cd /opt/netbox/netbox
python3 manage.py dbshell

I get the following:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/django/utils/connection.py", line 58, in __getitem__
    return getattr(self._connections, alias)
  File "/usr/local/lib/python3.10/dist-packages/asgiref/local.py", line 105, in __getattr__
    raise AttributeError(f"{self!r} object has no attribute {key!r}")
AttributeError: <asgiref.local.Local object at 0x7f0a43267610> object has no attribute 'default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 414, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/commands/dbshell.py", line 28, in handle
    connection = connections[options["database"]]
  File "/usr/local/lib/python3.10/dist-packages/django/utils/connection.py", line 60, in __getitem__
    if alias not in self.settings:
  File "/usr/local/lib/python3.10/dist-packages/django/utils/functional.py", line 49, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.10/dist-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 87, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 74, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 183, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/netbox/netbox/netbox/settings.py", line 22, in <module>
    from netbox.config import PARAMS
  File "/opt/netbox/netbox/netbox/config/__init__.py", line 9, in <module>
    from .parameters import PARAMS
  File "/opt/netbox/netbox/netbox/config/parameters.py", line 2, in <module>
    from django.contrib.postgres.forms import SimpleArrayField
  File "/usr/local/lib/python3.10/dist-packages/django/contrib/postgres/forms/__init__.py", line 3, in <module>
    from .ranges import *  # NOQA
  File "/usr/local/lib/python3.10/dist-packages/django/contrib/postgres/forms/ranges.py", line 1, in <module>
    from psycopg2.extras import DateRange, DateTimeTZRange, NumericRange
ModuleNotFoundError: No module named 'psycopg2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.10/dist-packages/django/core/management/base.py", line 427, in run_from_argv
    connections.close_all()
  File "/usr/local/lib/python3.10/dist-packages/django/db/utils.py", line 212, in close_all
    for alias in self:
  File "/usr/local/lib/python3.10/dist-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/functional.py", line 49, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/usr/local/lib/python3.10/dist-packages/django/db/utils.py", line 148, in configure_settings
    databases = super().configure_settings(databases)
  File "/usr/local/lib/python3.10/dist-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 87, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 74, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.10/dist-packages/django/conf/__init__.py", line 183, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/netbox/netbox/netbox/settings.py", line 22, in <module>
    from netbox.config import PARAMS
  File "/opt/netbox/netbox/netbox/config/__init__.py", line 9, in <module>
    from .parameters import PARAMS
  File "/opt/netbox/netbox/netbox/config/parameters.py", line 2, in <module>
    from django.contrib.postgres.forms import SimpleArrayField
  File "/usr/local/lib/python3.10/dist-packages/django/contrib/postgres/forms/__init__.py", line 3, in <module>
    from .ranges import *  # NOQA
  File "/usr/local/lib/python3.10/dist-packages/django/contrib/postgres/forms/ranges.py", line 1, in <module>
    from psycopg2.extras import DateRange, DateTimeTZRange, NumericRange
ModuleNotFoundError: No module named 'psycopg2'
jeremystretch commented 5 months ago

ModuleNotFoundError: No module named 'psycopg2'

You're missing at least one required library, which indicates that the installation/upgrade instructions were not followed. Please go back and ensure that your v3.7 deployment is fully functional before attempting to upgrade to v4.0.

opericgithub commented 5 months ago

Thanks, I beleive I now installed all django packages. Then I tried this:

netbox=> \d extras_customfield_content_types_id_seq
Did not find any relation named "extras_customfield_content_types_id_seq".

Here are all the tables (347 rows):

netbox=> \d
                              List of relations
 Schema |                      Name                      |   Type   | Owner  
--------+------------------------------------------------+----------+--------
 public | auth_group                                     | table    | netbox
 public | auth_group_id_seq                              | sequence | netbox
 public | auth_group_permissions                         | table    | netbox
 public | auth_group_permissions_id_seq                  | sequence | netbox
 public | auth_permission                                | table    | netbox
 public | auth_permission_id_seq                         | sequence | netbox
 public | auth_user                                      | table    | netbox
 public | auth_user_groups                               | table    | netbox
 public | auth_user_groups_id_seq                        | sequence | netbox
 public | auth_user_id_seq                               | sequence | netbox
 public | auth_user_user_permissions                     | table    | netbox
 public | auth_user_user_permissions_id_seq              | sequence | netbox
 public | circuits_circuit                               | table    | netbox
 public | circuits_circuit_id_seq                        | sequence | netbox
 public | circuits_circuittermination                    | table    | netbox
 public | circuits_circuittermination_id_seq             | sequence | netbox
 public | circuits_circuittype                           | table    | netbox
 public | circuits_circuittype_id_seq                    | sequence | netbox
 public | circuits_provider                              | table    | netbox
 public | circuits_provider_asns                         | table    | netbox
 public | circuits_provider_asns_id_seq                  | sequence | netbox
 public | circuits_provider_id_seq                       | sequence | netbox
 public | circuits_provideraccount                       | table    | netbox
 public | circuits_provideraccount_id_seq                | sequence | netbox
 public | circuits_providernetwork                       | table    | netbox
 public | circuits_providernetwork_id_seq                | sequence | netbox
 public | core_autosyncrecord                            | table    | netbox
 public | core_autosyncrecord_id_seq                     | sequence | netbox
 public | core_configrevision                            | table    | netbox
 public | core_datafile                                  | table    | netbox
 public | core_datafile_id_seq                           | sequence | netbox
 public | core_datasource                                | table    | netbox
 public | core_datasource_id_seq                         | sequence | netbox
 public | core_job                                       | table    | netbox
 public | core_job_id_seq                                | sequence | netbox
 public | core_managedfile                               | table    | netbox
 public | core_managedfile_id_seq                        | sequence | netbox
 public | dcim_cable                                     | table    | netbox
 public | dcim_cable_id_seq                              | sequence | netbox
 public | dcim_cablepath                                 | table    | netbox
 public | dcim_cablepath_id_seq                          | sequence | netbox
 public | dcim_cabletermination                          | table    | netbox
 public | dcim_cabletermination_id_seq                   | sequence | netbox
 public | dcim_consoleport                               | table    | netbox
 public | dcim_consoleport_id_seq                        | sequence | netbox
 public | dcim_consoleporttemplate                       | table    | netbox
 public | dcim_consoleporttemplate_id_seq                | sequence | netbox
 public | dcim_consoleserverport                         | table    | netbox
 public | dcim_consoleserverport_id_seq                  | sequence | netbox
 public | dcim_consoleserverporttemplate                 | table    | netbox
 public | dcim_consoleserverporttemplate_id_seq          | sequence | netbox
 public | dcim_device                                    | table    | netbox
 public | dcim_device_id_seq                             | sequence | netbox
 public | dcim_devicebay                                 | table    | netbox
 public | dcim_devicebay_id_seq                          | sequence | netbox
 public | dcim_devicebaytemplate                         | table    | netbox
 public | dcim_devicebaytemplate_id_seq                  | sequence | netbox
 public | dcim_devicerole                                | table    | netbox
 public | dcim_devicerole_id_seq                         | sequence | netbox
 public | dcim_devicetype                                | table    | netbox
 public | dcim_devicetype_id_seq                         | sequence | netbox
 public | dcim_frontport                                 | table    | netbox
 public | dcim_frontport_id_seq                          | sequence | netbox
 public | dcim_frontporttemplate                         | table    | netbox
 public | dcim_frontporttemplate_id_seq                  | sequence | netbox
 public | dcim_interface                                 | table    | netbox
 public | dcim_interface_id_seq                          | sequence | netbox
 public | dcim_interface_tagged_vlans                    | table    | netbox
 public | dcim_interface_tagged_vlans_id_seq             | sequence | netbox
 public | dcim_interface_vdcs                            | table    | netbox
 public | dcim_interface_vdcs_id_seq                     | sequence | netbox
 public | dcim_interface_wireless_lans                   | table    | netbox
 public | dcim_interface_wireless_lans_id_seq            | sequence | netbox
 public | dcim_interfacetemplate                         | table    | netbox
 public | dcim_interfacetemplate_id_seq                  | sequence | netbox
 public | dcim_inventoryitem                             | table    | netbox
 public | dcim_inventoryitem_id_seq                      | sequence | netbox
 public | dcim_inventoryitemrole                         | table    | netbox
 public | dcim_inventoryitemrole_id_seq                  | sequence | netbox
 public | dcim_inventoryitemtemplate                     | table    | netbox
 public | dcim_inventoryitemtemplate_id_seq              | sequence | netbox
 public | dcim_location                                  | table    | netbox
 public | dcim_manufacturer                              | table    | netbox
 public | dcim_manufacturer_id_seq                       | sequence | netbox
 public | dcim_module                                    | table    | netbox
 public | dcim_module_id_seq                             | sequence | netbox
 public | dcim_module_id_seq1                            | sequence | netbox
 public | dcim_modulebay                                 | table    | netbox
 public | dcim_modulebay_id_seq                          | sequence | netbox
 public | dcim_modulebaytemplate                         | table    | netbox
 public | dcim_modulebaytemplate_id_seq                  | sequence | netbox
 public | dcim_moduletype                                | table    | netbox
 public | dcim_moduletype_id_seq                         | sequence | netbox
 public | dcim_platform                                  | table    | netbox
 public | dcim_platform_id_seq                           | sequence | netbox
 public | dcim_powerfeed                                 | table    | netbox
 public | dcim_powerfeed_id_seq                          | sequence | netbox
 public | dcim_poweroutlet                               | table    | netbox
 public | dcim_poweroutlet_id_seq                        | sequence | netbox
 public | dcim_poweroutlettemplate                       | table    | netbox
 public | dcim_poweroutlettemplate_id_seq                | sequence | netbox
 public | dcim_powerpanel                                | table    | netbox
 public | dcim_powerpanel_id_seq                         | sequence | netbox
 public | dcim_powerport                                 | table    | netbox
 public | dcim_powerport_id_seq                          | sequence | netbox
 public | dcim_powerporttemplate                         | table    | netbox
 public | dcim_powerporttemplate_id_seq                  | sequence | netbox
 public | dcim_rack                                      | table    | netbox
 public | dcim_rack_id_seq                               | sequence | netbox
 public | dcim_rackgroup_id_seq                          | sequence | netbox
 public | dcim_rackreservation                           | table    | netbox
 public | dcim_rackreservation_id_seq                    | sequence | netbox
 public | dcim_rackrole                                  | table    | netbox
 public | dcim_rackrole_id_seq                           | sequence | netbox
 public | dcim_rearport                                  | table    | netbox
 public | dcim_rearport_id_seq                           | sequence | netbox
 public | dcim_rearporttemplate                          | table    | netbox
 public | dcim_rearporttemplate_id_seq                   | sequence | netbox
 public | dcim_region                                    | table    | netbox
 public | dcim_region_id_seq                             | sequence | netbox
 public | dcim_site                                      | table    | netbox
 public | dcim_site_asns                                 | table    | netbox
 public | dcim_site_asns_id_seq                          | sequence | netbox
 public | dcim_site_id_seq                               | sequence | netbox
 public | dcim_sitegroup                                 | table    | netbox
 public | dcim_sitegroup_id_seq                          | sequence | netbox
 public | dcim_virtualchassis                            | table    | netbox
 public | dcim_virtualchassis_id_seq                     | sequence | netbox
 public | dcim_virtualdevicecontext                      | table    | netbox
 public | dcim_virtualdevicecontext_id_seq               | sequence | netbox
 public | django_admin_log                               | table    | netbox
 public | django_admin_log_id_seq                        | sequence | netbox
 public | django_content_type                            | table    | netbox
 public | django_content_type_id_seq                     | sequence | netbox
 public | django_migrations                              | table    | netbox
 public | django_migrations_id_seq                       | sequence | netbox
 public | django_session                                 | table    | netbox
 public | extras_bookmark                                | table    | netbox
 public | extras_bookmark_id_seq                         | sequence | netbox
 public | extras_branch                                  | table    | netbox
 public | extras_branch_id_seq                           | sequence | netbox
 public | extras_cachedvalue                             | table    | netbox
 public | extras_configcontext                           | table    | netbox
 public | extras_configcontext_cluster_groups            | table    | netbox
 public | extras_configcontext_cluster_groups_id_seq     | sequence | netbox
 public | extras_configcontext_cluster_types             | table    | netbox
 public | extras_configcontext_cluster_types_id_seq      | sequence | netbox
 public | extras_configcontext_clusters                  | table    | netbox
 public | extras_configcontext_clusters_id_seq           | sequence | netbox
 public | extras_configcontext_device_types              | table    | netbox
 public | extras_configcontext_device_types_id_seq       | sequence | netbox
 public | extras_configcontext_id_seq                    | sequence | netbox
 public | extras_configcontext_locations                 | table    | netbox
 public | extras_configcontext_locations_id_seq          | sequence | netbox
 public | extras_configcontext_platforms                 | table    | netbox
 public | extras_configcontext_platforms_id_seq          | sequence | netbox
 public | extras_configcontext_regions                   | table    | netbox
 public | extras_configcontext_regions_id_seq            | sequence | netbox
 public | extras_configcontext_roles                     | table    | netbox
 public | extras_configcontext_roles_id_seq              | sequence | netbox
 public | extras_configcontext_site_groups               | table    | netbox
 public | extras_configcontext_site_groups_id_seq        | sequence | netbox
 public | extras_configcontext_sites                     | table    | netbox
 public | extras_configcontext_sites_id_seq              | sequence | netbox
 public | extras_configcontext_tags                      | table    | netbox
 public | extras_configcontext_tags_id_seq               | sequence | netbox
 public | extras_configcontext_tenant_groups             | table    | netbox
 public | extras_configcontext_tenant_groups_id_seq      | sequence | netbox
 public | extras_configcontext_tenants                   | table    | netbox
 public | extras_configcontext_tenants_id_seq            | sequence | netbox
 public | extras_configrevision_id_seq                   | sequence | netbox
 public | extras_configtemplate                          | table    | netbox
 public | extras_configtemplate_id_seq                   | sequence | netbox
 public | extras_customfield                             | table    | netbox
 public | extras_customfield_content_types               | table    | netbox
 public | extras_customfield_id_seq                      | sequence | netbox
 public | extras_customfield_obj_type_id_seq             | sequence | netbox
 public | extras_customfieldchoiceset                    | table    | netbox
 public | extras_customfieldchoiceset_id_seq             | sequence | netbox
 public | extras_customlink                              | table    | netbox
 public | extras_customlink_content_types                | table    | netbox
 public | extras_customlink_content_types_id_seq         | sequence | netbox
 public | extras_customlink_id_seq                       | sequence | netbox
 public | extras_dashboard                               | table    | netbox
 public | extras_dashboard_id_seq                        | sequence | netbox
 public | extras_eventrule                               | table    | netbox
 public | extras_eventrule_content_types                 | table    | netbox
 public | extras_eventrule_content_types_id_seq          | sequence | netbox
 public | extras_eventrule_id_seq                        | sequence | netbox
 public | extras_exporttemplate                          | table    | netbox
 public | extras_exporttemplate_content_types            | table    | netbox
 public | extras_exporttemplate_content_types_id_seq     | sequence | netbox
 public | extras_exporttemplate_id_seq                   | sequence | netbox
 public | extras_imageattachment                         | table    | netbox
 public | extras_imageattachment_id_seq                  | sequence | netbox
 public | extras_journalentry                            | table    | netbox
 public | extras_journalentry_id_seq                     | sequence | netbox
 public | extras_objectchange                            | table    | netbox
 public | extras_objectchange_id_seq                     | sequence | netbox
 public | extras_savedfilter                             | table    | netbox
 public | extras_savedfilter_content_types               | table    | netbox
 public | extras_savedfilter_content_types_id_seq        | sequence | netbox
 public | extras_savedfilter_id_seq                      | sequence | netbox
 public | extras_stagedchange                            | table    | netbox
 public | extras_stagedchange_id_seq                     | sequence | netbox
 public | extras_tag                                     | table    | netbox
 public | extras_tag_id_seq                              | sequence | netbox
 public | extras_tag_object_types                        | table    | netbox
 public | extras_tag_object_types_id_seq                 | sequence | netbox
 public | extras_taggeditem                              | table    | netbox
 public | extras_taggeditem_id_seq                       | sequence | netbox
 public | extras_webhook                                 | table    | netbox
 public | extras_webhook_id_seq                          | sequence | netbox
 public | ipam_aggregate                                 | table    | netbox
 public | ipam_aggregate_id_seq                          | sequence | netbox
 public | ipam_asn                                       | table    | netbox
 public | ipam_asn_id_seq                                | sequence | netbox
 public | ipam_asnrange                                  | table    | netbox
 public | ipam_asnrange_id_seq                           | sequence | netbox
 public | ipam_fhrpgroup                                 | table    | netbox
 public | ipam_fhrpgroup_id_seq                          | sequence | netbox
 public | ipam_fhrpgroupassignment                       | table    | netbox
 public | ipam_fhrpgroupassignment_id_seq                | sequence | netbox
 public | ipam_ipaddress                                 | table    | netbox
 public | ipam_ipaddress_id_seq                          | sequence | netbox
 public | ipam_iprange                                   | table    | netbox
 public | ipam_iprange_id_seq                            | sequence | netbox
 public | ipam_l2vpn_export_targets_id_seq               | sequence | netbox
 public | ipam_l2vpn_id_seq                              | sequence | netbox
 public | ipam_l2vpn_import_targets_id_seq               | sequence | netbox
 public | ipam_l2vpntermination_id_seq                   | sequence | netbox
 public | ipam_prefix                                    | table    | netbox
 public | ipam_prefix_id_seq                             | sequence | netbox
 public | ipam_rir                                       | table    | netbox
 public | ipam_rir_id_seq                                | sequence | netbox
 public | ipam_role                                      | table    | netbox
 public | ipam_role_id_seq                               | sequence | netbox
 public | ipam_routetarget                               | table    | netbox
 public | ipam_routetarget_id_seq                        | sequence | netbox
 public | ipam_service                                   | table    | netbox
 public | ipam_service_id_seq                            | sequence | netbox
 public | ipam_service_ipaddresses                       | table    | netbox
 public | ipam_service_ipaddresses_id_seq                | sequence | netbox
 public | ipam_servicetemplate                           | table    | netbox
 public | ipam_servicetemplate_id_seq                    | sequence | netbox
 public | ipam_vlan                                      | table    | netbox
 public | ipam_vlan_id_seq                               | sequence | netbox
 public | ipam_vlangroup                                 | table    | netbox
 public | ipam_vlangroup_id_seq                          | sequence | netbox
 public | ipam_vrf                                       | table    | netbox
 public | ipam_vrf_export_targets                        | table    | netbox
 public | ipam_vrf_export_targets_id_seq                 | sequence | netbox
 public | ipam_vrf_id_seq                                | sequence | netbox
 public | ipam_vrf_import_targets                        | table    | netbox
 public | ipam_vrf_import_targets_id_seq                 | sequence | netbox
 public | nextbox_ui_plugin_savedtopology                | table    | netbox
 public | nextbox_ui_plugin_savedtopology_id_seq         | sequence | netbox
 public | secrets_secret                                 | table    | netbox
 public | secrets_secret_id_seq                          | sequence | netbox
 public | secrets_secretrole                             | table    | netbox
 public | secrets_secretrole_id_seq                      | sequence | netbox
 public | secrets_sessionkey                             | table    | netbox
 public | secrets_sessionkey_id_seq                      | sequence | netbox
 public | secrets_userkey                                | table    | netbox
 public | secrets_userkey_id_seq                         | sequence | netbox
 public | social_auth_association                        | table    | netbox
 public | social_auth_association_id_seq                 | sequence | netbox
 public | social_auth_code                               | table    | netbox
 public | social_auth_code_id_seq                        | sequence | netbox
 public | social_auth_nonce                              | table    | netbox
 public | social_auth_nonce_id_seq                       | sequence | netbox
 public | social_auth_partial                            | table    | netbox
 public | social_auth_partial_id_seq                     | sequence | netbox
 public | social_auth_usersocialauth                     | table    | netbox
 public | social_auth_usersocialauth_id_seq              | sequence | netbox
 public | taggit_tag                                     | table    | netbox
 public | taggit_tag_id_seq                              | sequence | netbox
 public | taggit_taggeditem                              | table    | netbox
 public | taggit_taggeditem_id_seq                       | sequence | netbox
 public | tenancy_contact                                | table    | netbox
 public | tenancy_contact_id_seq                         | sequence | netbox
 public | tenancy_contactassignment                      | table    | netbox
 public | tenancy_contactassignment_id_seq               | sequence | netbox
 public | tenancy_contactgroup                           | table    | netbox
 public | tenancy_contactgroup_id_seq                    | sequence | netbox
 public | tenancy_contactrole                            | table    | netbox
 public | tenancy_contactrole_id_seq                     | sequence | netbox
 public | tenancy_tenant                                 | table    | netbox
 public | tenancy_tenant_id_seq                          | sequence | netbox
 public | tenancy_tenantgroup                            | table    | netbox
 public | tenancy_tenantgroup_id_seq                     | sequence | netbox
 public | users_objectpermission                         | table    | netbox
 public | users_objectpermission_groups                  | table    | netbox
 public | users_objectpermission_groups_id_seq           | sequence | netbox
 public | users_objectpermission_id_seq                  | sequence | netbox
 public | users_objectpermission_object_types            | table    | netbox
 public | users_objectpermission_object_types_id_seq     | sequence | netbox
 public | users_objectpermission_users                   | table    | netbox
 public | users_objectpermission_users_id_seq            | sequence | netbox
 public | users_token                                    | table    | netbox
 public | users_token_id_seq                             | sequence | netbox
 public | users_userconfig                               | table    | netbox
 public | users_userconfig_id_seq                        | sequence | netbox
 public | virtualization_cluster                         | table    | netbox
 public | virtualization_cluster_id_seq                  | sequence | netbox
 public | virtualization_clustergroup                    | table    | netbox
 public | virtualization_clustergroup_id_seq             | sequence | netbox
 public | virtualization_clustertype                     | table    | netbox
 public | virtualization_clustertype_id_seq              | sequence | netbox
 public | virtualization_virtualdisk                     | table    | netbox
 public | virtualization_virtualdisk_id_seq              | sequence | netbox
 public | virtualization_virtualmachine                  | table    | netbox
 public | virtualization_virtualmachine_id_seq           | sequence | netbox
 public | virtualization_vminterface                     | table    | netbox
 public | virtualization_vminterface_id_seq              | sequence | netbox
 public | virtualization_vminterface_tagged_vlans        | table    | netbox
 public | virtualization_vminterface_tagged_vlans_id_seq | sequence | netbox
 public | vpn_ikepolicy                                  | table    | netbox
 public | vpn_ikepolicy_id_seq                           | sequence | netbox
 public | vpn_ikepolicy_proposals                        | table    | netbox
 public | vpn_ikepolicy_proposals_id_seq                 | sequence | netbox
 public | vpn_ikeproposal                                | table    | netbox
 public | vpn_ikeproposal_id_seq                         | sequence | netbox
 public | vpn_ipsecpolicy                                | table    | netbox
 public | vpn_ipsecpolicy_id_seq                         | sequence | netbox
 public | vpn_ipsecpolicy_proposals                      | table    | netbox
 public | vpn_ipsecpolicy_proposals_id_seq               | sequence | netbox
 public | vpn_ipsecprofile                               | table    | netbox
 public | vpn_ipsecprofile_id_seq                        | sequence | netbox
 public | vpn_ipsecproposal                              | table    | netbox
 public | vpn_ipsecproposal_id_seq                       | sequence | netbox
 public | vpn_l2vpn                                      | table    | netbox
 public | vpn_l2vpn_export_targets                       | table    | netbox
 public | vpn_l2vpn_import_targets                       | table    | netbox
 public | vpn_l2vpntermination                           | table    | netbox
 public | vpn_tunnel                                     | table    | netbox
 public | vpn_tunnel_id_seq                              | sequence | netbox
 public | vpn_tunnelgroup                                | table    | netbox
 public | vpn_tunnelgroup_id_seq                         | sequence | netbox
 public | vpn_tunneltermination                          | table    | netbox
 public | vpn_tunneltermination_id_seq                   | sequence | netbox
 public | wireless_wirelesslan                           | table    | netbox
 public | wireless_wirelesslan_id_seq                    | sequence | netbox
 public | wireless_wirelesslangroup                      | table    | netbox
 public | wireless_wirelesslangroup_id_seq               | sequence | netbox
 public | wireless_wirelesslink                          | table    | netbox
 public | wireless_wirelesslink_id_seq                   | sequence | netbox
(347 rows)
candlerb commented 5 months ago

Thanks, I beleive I now installed all django packages.

You should not have to install anything.

You should allow Netbox to create a virtualenv and populate it; this is all done using the ./upgrade.sh script. If you have a problem with running upgrade.sh, please show it.

If you find you have to install packages manually then there is definitely some other problem: you need to stop and fix that problem, before continuing.

EDIT: if you did this (as you say):

cd /opt/netbox/netbox
python3 manage.py dbshell

and you got that error, it means you probably hadn't activated the virtualenv before running the command. You definitely should not install django packages globally to workaround that issue.

Actually, there's a safer way to run it:

/opt/netbox/venv/bin/python /opt/netbox/netbox/manage.py dbshell

This will activate the virtualenv for you (regardless of whether it's currently activated or not)

opericgithub commented 5 months ago

Actually, there's a safer way to run it:

/opt/netbox/venv/bin/python /opt/netbox/netbox/manage.py dbshell

This will activate the virtualenv for you (regardless of whether it's currently activated or not)

Yes, after I run this command, I can immediately list all tables with "\d" command, as Jeremy asked.

You should allow Netbox to create a virtualenv and populate it; this is all done using the ./upgrade.sh script. If you have a problem with running upgrade.sh, please show it.

I can run upgrade.sh without problems:

cd /opt/netbox
sudo ./upgrade.sh

...
Removing old virtual environment...
Creating a new virtual environment at /opt/netbox/venv...
Updating pip (pip install --upgrade pip)...
Requirement already satisfied: pip in ./venv/lib/python3.10/site-packages (22.0.2)
...  
Installing local dependencies (pip install -r local_requirements.txt)...
Collecting django-auth-ldap (from -r local_requirements.txt (line 1))
  Using cached django_auth_ldap-4.7.0-py3-none-any.whl.metadata (7.2 kB)
Requirement already satisfied: Django>=3.2 in ./venv/lib/python3.10/site-packages (from django-auth-ldap->-r local_requirements.txt (line 1)) (4.2.11)
...
  wireless.wirelesslink... No objects found.
Completed. Total entries: 36437
Removing expired user sessions (python3 netbox/manage.py clearsessions)...
Upgrade complete! Don't forget to restart the NetBox services:

Even after successful execution of this script on the stable version 3.7.4, I am not able to upgrade to beta.

I mean, first I do this: sudo ./upgrade.sh Everything is OK. Then this: sudo git pull origin v4.0-beta1 then again this: sudo ./upgrade.sh and I get stucked with the message from the first post (relation "extras_customfield_content_types_id_seq" does not exist).

And really, when one looks at the tables I posted above, there is a missing row with sequence regarding the custom field content type. As far as I understand there should be sequence for every table (vpntunnel, vpntunnel_id_seq, dcim_site, dcim_site_id_seq etc).

tomasz-c commented 5 months ago

I have this problem too. ./upgrade.sh on version 3.7.4 goes correctly, but there is no extras_customfield_content_types_id_seq in the database

netbox=> \d extras_customfield_content_types_id_seq
Did not find any relation named "extras_customfield_content_types_id_seq".

Trying to force migration to extras on version 3.7.4 reports no changes:

/opt/netbox/venv/bin/python /opt/netbox/netbox/manage.py migrate extras
Operations to perform:
  Apply all migrations: extras
Running migrations:
  No migrations to apply.

So I have the same error when trying to update to v4.0-beta1. I have been updating the netbox for a long time, maybe this table was created only for new installations and the update omitted it?

candlerb commented 5 months ago

I mean, first I do this: sudo ./upgrade.sh Everything is OK. Then this: sudo git pull origin v4.0-beta1 then again this: sudo ./upgrade.sh and I get stucked with the message from the first post (relation "extras_customfield_content_types_id_seq" does not exist).

It's broken for me too; some of the sequences and indexes in my system (and your system, and some others it seems) have different names to the ones that the migration scripts are expecting, and some constraints are missing.

I was able to make the migrations complete with some temporary patches: https://github.com/netbox-community/netbox/discussions/15601#discussioncomment-8998869

But there's some underlying issue about how machines which have a long history of upgrades having different constraint names than those currently expected.

opericgithub commented 5 months ago

Regarding the long history of upgrades, which other users have also discussed, it might be worth mentioning that we initially had version 2.8, then progressed subsequently to 2.11, followed by 3.0, and subsequently to each minor version (3.1, 3.2, etc.).

At one moment we did clean install of 3.x (don't remember the particular version) and did migrate all data from the same version from another server.

PaulR282 commented 5 months ago

I'm running into this issue while trying to start a new container with the v4.0-beta1:

Traceback (most recent call last):
  File "/opt/netbox/netbox/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/commands/shell.py", line 127, in handle
    exec(sys.stdin.read(), globals())
  File "<string>", line 3, in <module>
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 196, in __get__
    raise AttributeError(
AttributeError: Manager isn't available; 'auth.User' has been swapped for 'users.User'
jhammond-git commented 5 months ago

To add to the data set, I had the same issues as documented in https://github.com/netbox-community/netbox/discussions/15601#discussioncomment-8998869.

The VM I installed onto is a brand new ubuntu 22.04 machine with a fresh install of 3.7.4, but the database that was imported was upgraded over the years from 1.9.5, or at least, that's the first version I know we had on our production box.

ingvaldlorentzen commented 5 months ago

I'm running into this issue while trying to start a new container with the v4.0-beta1:

AttributeError: Manager isn't available; 'auth.User' has been swapped for 'users.User'

I've got this exact issue, and I can't figure out where it is coming from. Wouldn't mind looking into it, but I have no idea where to start.

v0tti commented 5 months ago

I am also unable to upgrade to v4.0-beta1 because ./upgrade.sh looks for reports and scripts that are not synced from the data-source yet, thus the files do not exist:

  Applying extras.0109_script_model...Traceback (most recent call last):
  File "/opt/netbox-4.0-beta1/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/opt/netbox-4.0-beta1/venv/lib64/python3.11/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox-4.0-beta1/netbox/extras/migrations/0109_script_model.py", line 93, in update_scripts
    for script_name in get_module_scripts(module):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox-4.0-beta1/netbox/extras/migrations/0109_script_model.py", line 63, in get_module_scripts
    module = loader.load_module()
             ^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 605, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1120, in load_module
  File "<frozen importlib._bootstrap_external>", line 945, in load_module
  File "<frozen importlib._bootstrap>", line 290, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 721, in _load
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 936, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1073, in get_code
  File "<frozen importlib._bootstrap_external>", line 1130, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/netbox-4.0-beta1/netbox/reports/devices.py'
v0tti commented 5 months ago 
FileNotFoundError: [Errno 2] No such file or directory: '/opt/netbox-4.0-beta1/netbox/reports/devices.py'

After fixing this by manually creating the mentioned files I am also running into the django.db.utils.ProgrammingError: relation "extras_customfield_content_types_id_seq" does not exist problem.

jeremystretch commented 5 months ago

I've hidden several comments in the thread which are not relevant to the reported behavior above. Please remember to open a separate bug report for unrelated issues.

jeremystretch commented 5 months ago

Could someone who is experiencing this issue please post the list of tables which exist in their database prior to attempting the upgrade to v4.0? This can be done by running manage.py dbshell and issuing the \d console command.)

I'm specifically interested in determining how extras_customfield_content_types_id_seq is named in older databases.

a084ed22 commented 5 months ago

I have a setup from the 2.11 era, gradually updated, currently on 3.7.5. I don't have a sequence for extras_customfield_content_types_id_seq.

tobiasge commented 5 months ago

Could someone who is experiencing this issue please post the list of tables which exist in their database prior to attempting the upgrade to v4.0? This can be done by running manage.py dbshell and issuing the \d console command.)

I'm specifically interested in determining how extras_customfield_content_types_id_seq is named in older databases.

This is the list from our Netbox 3.7.5 (first installation on this DB was from 2.5 or 2.4):

netbox-prod=# \d
                                List of relations
 Schema |                      Name                      |   Type   |    Owner
--------+------------------------------------------------+----------+-------------
 public | auth_group                                     | table    | netbox-prod
 public | auth_group_id_seq                              | sequence | netbox-prod
 public | auth_group_permissions                         | table    | netbox-prod
 public | auth_group_permissions_id_seq                  | sequence | netbox-prod
 public | auth_permission                                | table    | netbox-prod
 public | auth_permission_id_seq                         | sequence | netbox-prod
 public | auth_user                                      | table    | netbox-prod
 public | auth_user_groups                               | table    | netbox-prod
 public | auth_user_groups_id_seq                        | sequence | netbox-prod
 public | auth_user_id_seq                               | sequence | netbox-prod
 public | auth_user_user_permissions                     | table    | netbox-prod
 public | auth_user_user_permissions_id_seq              | sequence | netbox-prod
 public | circuits_circuit                               | table    | netbox-prod
 public | circuits_circuit_id_seq                        | sequence | netbox-prod
 public | circuits_circuittermination                    | table    | netbox-prod
 public | circuits_circuittermination_id_seq             | sequence | netbox-prod
 public | circuits_circuittype                           | table    | netbox-prod
 public | circuits_circuittype_id_seq                    | sequence | netbox-prod
 public | circuits_provider                              | table    | netbox-prod
 public | circuits_provider_asns                         | table    | netbox-prod
 public | circuits_provider_asns_id_seq                  | sequence | netbox-prod
 public | circuits_provider_id_seq                       | sequence | netbox-prod
 public | circuits_provideraccount                       | table    | netbox-prod
 public | circuits_provideraccount_id_seq                | sequence | netbox-prod
 public | circuits_providernetwork                       | table    | netbox-prod
 public | circuits_providernetwork_id_seq                | sequence | netbox-prod
 public | core_autosyncrecord                            | table    | netbox-prod
 public | core_autosyncrecord_id_seq                     | sequence | netbox-prod
 public | core_configrevision                            | table    | netbox-prod
 public | core_datafile                                  | table    | netbox-prod
 public | core_datafile_id_seq                           | sequence | netbox-prod
 public | core_datasource                                | table    | netbox-prod
 public | core_datasource_id_seq                         | sequence | netbox-prod
 public | core_job                                       | table    | netbox-prod
 public | core_job_id_seq                                | sequence | netbox-prod
 public | core_managedfile                               | table    | netbox-prod
 public | core_managedfile_id_seq                        | sequence | netbox-prod
 public | dcim_cable                                     | table    | netbox-prod
 public | dcim_cable_id_seq                              | sequence | netbox-prod
 public | dcim_cablepath                                 | table    | netbox-prod
 public | dcim_cablepath_id_seq                          | sequence | netbox-prod
 public | dcim_cabletermination                          | table    | netbox-prod
 public | dcim_cabletermination_id_seq                   | sequence | netbox-prod
 public | dcim_consoleport                               | table    | netbox-prod
 public | dcim_consoleport_id_seq                        | sequence | netbox-prod
 public | dcim_consoleporttemplate                       | table    | netbox-prod
 public | dcim_consoleporttemplate_id_seq                | sequence | netbox-prod
 public | dcim_consoleserverport                         | table    | netbox-prod
 public | dcim_consoleserverport_id_seq                  | sequence | netbox-prod
 public | dcim_consoleserverporttemplate                 | table    | netbox-prod
 public | dcim_consoleserverporttemplate_id_seq          | sequence | netbox-prod
 public | dcim_device                                    | table    | netbox-prod
 public | dcim_device_id_seq                             | sequence | netbox-prod
 public | dcim_devicebay                                 | table    | netbox-prod
 public | dcim_devicebay_id_seq                          | sequence | netbox-prod
 public | dcim_devicebaytemplate                         | table    | netbox-prod
 public | dcim_devicebaytemplate_id_seq                  | sequence | netbox-prod
 public | dcim_devicerole                                | table    | netbox-prod
 public | dcim_devicerole_id_seq                         | sequence | netbox-prod
 public | dcim_devicetype                                | table    | netbox-prod
 public | dcim_devicetype_id_seq                         | sequence | netbox-prod
 public | dcim_frontport                                 | table    | netbox-prod
 public | dcim_frontport_id_seq                          | sequence | netbox-prod
 public | dcim_frontporttemplate                         | table    | netbox-prod
 public | dcim_frontporttemplate_id_seq                  | sequence | netbox-prod
 public | dcim_interface                                 | table    | netbox-prod
 public | dcim_interface_id_seq                          | sequence | netbox-prod
 public | dcim_interface_tagged_vlans                    | table    | netbox-prod
 public | dcim_interface_tagged_vlans_id_seq             | sequence | netbox-prod
 public | dcim_interface_vdcs                            | table    | netbox-prod
 public | dcim_interface_vdcs_id_seq                     | sequence | netbox-prod
 public | dcim_interface_wireless_lans                   | table    | netbox-prod
 public | dcim_interface_wireless_lans_id_seq            | sequence | netbox-prod
 public | dcim_interfacetemplate                         | table    | netbox-prod
 public | dcim_interfacetemplate_id_seq                  | sequence | netbox-prod
 public | dcim_inventoryitem                             | table    | netbox-prod
 public | dcim_inventoryitem_id_seq                      | sequence | netbox-prod
 public | dcim_inventoryitemrole                         | table    | netbox-prod
 public | dcim_inventoryitemrole_id_seq                  | sequence | netbox-prod
 public | dcim_inventoryitemtemplate                     | table    | netbox-prod
 public | dcim_inventoryitemtemplate_id_seq              | sequence | netbox-prod
 public | dcim_location                                  | table    | netbox-prod
 public | dcim_manufacturer                              | table    | netbox-prod
 public | dcim_manufacturer_id_seq                       | sequence | netbox-prod
 public | dcim_module                                    | table    | netbox-prod
 public | dcim_module_id_seq                             | sequence | netbox-prod
 public | dcim_module_id_seq1                            | sequence | netbox-prod
 public | dcim_modulebay                                 | table    | netbox-prod
 public | dcim_modulebay_id_seq                          | sequence | netbox-prod
 public | dcim_modulebaytemplate                         | table    | netbox-prod
 public | dcim_modulebaytemplate_id_seq                  | sequence | netbox-prod
 public | dcim_moduletype                                | table    | netbox-prod
 public | dcim_moduletype_id_seq                         | sequence | netbox-prod
 public | dcim_platform                                  | table    | netbox-prod
 public | dcim_platform_id_seq                           | sequence | netbox-prod
 public | dcim_powerfeed                                 | table    | netbox-prod
 public | dcim_powerfeed_id_seq                          | sequence | netbox-prod
 public | dcim_poweroutlet                               | table    | netbox-prod
 public | dcim_poweroutlet_id_seq                        | sequence | netbox-prod
 public | dcim_poweroutlettemplate                       | table    | netbox-prod
 public | dcim_poweroutlettemplate_id_seq                | sequence | netbox-prod
 public | dcim_powerpanel                                | table    | netbox-prod
 public | dcim_powerpanel_id_seq                         | sequence | netbox-prod
 public | dcim_powerport                                 | table    | netbox-prod
 public | dcim_powerport_id_seq                          | sequence | netbox-prod
 public | dcim_powerporttemplate                         | table    | netbox-prod
 public | dcim_powerporttemplate_id_seq                  | sequence | netbox-prod
 public | dcim_rack                                      | table    | netbox-prod
 public | dcim_rack_id_seq                               | sequence | netbox-prod
 public | dcim_rackgroup_id_seq                          | sequence | netbox-prod
 public | dcim_rackreservation                           | table    | netbox-prod
 public | dcim_rackreservation_id_seq                    | sequence | netbox-prod
 public | dcim_rackrole                                  | table    | netbox-prod
 public | dcim_rackrole_id_seq                           | sequence | netbox-prod
 public | dcim_rearport                                  | table    | netbox-prod
 public | dcim_rearport_id_seq                           | sequence | netbox-prod
 public | dcim_rearporttemplate                          | table    | netbox-prod
 public | dcim_rearporttemplate_id_seq                   | sequence | netbox-prod
 public | dcim_region                                    | table    | netbox-prod
 public | dcim_region_id_seq                             | sequence | netbox-prod
 public | dcim_site                                      | table    | netbox-prod
 public | dcim_site_asns                                 | table    | netbox-prod
 public | dcim_site_asns_id_seq                          | sequence | netbox-prod
 public | dcim_site_id_seq                               | sequence | netbox-prod
 public | dcim_sitegroup                                 | table    | netbox-prod
 public | dcim_sitegroup_id_seq                          | sequence | netbox-prod
 public | dcim_virtualchassis                            | table    | netbox-prod
 public | dcim_virtualchassis_id_seq                     | sequence | netbox-prod
 public | dcim_virtualdevicecontext                      | table    | netbox-prod
 public | dcim_virtualdevicecontext_id_seq               | sequence | netbox-prod
 public | django_admin_log                               | table    | netbox-prod
 public | django_admin_log_id_seq                        | sequence | netbox-prod
 public | django_content_type                            | table    | netbox-prod
 public | django_content_type_id_seq                     | sequence | netbox-prod
 public | django_migrations                              | table    | netbox-prod
 public | django_migrations_id_seq                       | sequence | netbox-prod
 public | django_session                                 | table    | netbox-prod
 public | extras_bookmark                                | table    | netbox-prod
 public | extras_bookmark_id_seq                         | sequence | netbox-prod
 public | extras_branch                                  | table    | netbox-prod
 public | extras_branch_id_seq                           | sequence | netbox-prod
 public | extras_cachedvalue                             | table    | netbox-prod
 public | extras_configcontext                           | table    | netbox-prod
 public | extras_configcontext_cluster_groups            | table    | netbox-prod
 public | extras_configcontext_cluster_groups_id_seq     | sequence | netbox-prod
 public | extras_configcontext_cluster_types             | table    | netbox-prod
 public | extras_configcontext_cluster_types_id_seq      | sequence | netbox-prod
 public | extras_configcontext_clusters                  | table    | netbox-prod
 public | extras_configcontext_clusters_id_seq           | sequence | netbox-prod
 public | extras_configcontext_device_types              | table    | netbox-prod
 public | extras_configcontext_device_types_id_seq       | sequence | netbox-prod
 public | extras_configcontext_id_seq                    | sequence | netbox-prod
 public | extras_configcontext_locations                 | table    | netbox-prod
 public | extras_configcontext_locations_id_seq          | sequence | netbox-prod
 public | extras_configcontext_platforms                 | table    | netbox-prod
 public | extras_configcontext_platforms_id_seq          | sequence | netbox-prod
 public | extras_configcontext_regions                   | table    | netbox-prod
 public | extras_configcontext_regions_id_seq            | sequence | netbox-prod
 public | extras_configcontext_roles                     | table    | netbox-prod
 public | extras_configcontext_roles_id_seq              | sequence | netbox-prod
 public | extras_configcontext_site_groups               | table    | netbox-prod
 public | extras_configcontext_site_groups_id_seq        | sequence | netbox-prod
 public | extras_configcontext_sites                     | table    | netbox-prod
 public | extras_configcontext_sites_id_seq              | sequence | netbox-prod
 public | extras_configcontext_tags                      | table    | netbox-prod
 public | extras_configcontext_tags_id_seq               | sequence | netbox-prod
 public | extras_configcontext_tenant_groups             | table    | netbox-prod
 public | extras_configcontext_tenant_groups_id_seq      | sequence | netbox-prod
 public | extras_configcontext_tenants                   | table    | netbox-prod
 public | extras_configcontext_tenants_id_seq            | sequence | netbox-prod
 public | extras_configrevision_id_seq                   | sequence | netbox-prod
 public | extras_configtemplate                          | table    | netbox-prod
 public | extras_configtemplate_id_seq                   | sequence | netbox-prod
 public | extras_customfield                             | table    | netbox-prod
 public | extras_customfield_content_types               | table    | netbox-prod
 public | extras_customfield_id_seq                      | sequence | netbox-prod
 public | extras_customfield_obj_type_id_seq             | sequence | netbox-prod
 public | extras_customfieldchoiceset                    | table    | netbox-prod
 public | extras_customfieldchoiceset_id_seq             | sequence | netbox-prod
 public | extras_customlink                              | table    | netbox-prod
 public | extras_customlink_content_types                | table    | netbox-prod
 public | extras_customlink_content_types_id_seq         | sequence | netbox-prod
 public | extras_customlink_id_seq                       | sequence | netbox-prod
 public | extras_dashboard                               | table    | netbox-prod
 public | extras_dashboard_id_seq                        | sequence | netbox-prod
 public | extras_eventrule                               | table    | netbox-prod
 public | extras_eventrule_content_types                 | table    | netbox-prod
 public | extras_eventrule_content_types_id_seq          | sequence | netbox-prod
 public | extras_eventrule_id_seq                        | sequence | netbox-prod
 public | extras_exporttemplate                          | table    | netbox-prod
 public | extras_exporttemplate_content_types            | table    | netbox-prod
 public | extras_exporttemplate_content_types_id_seq     | sequence | netbox-prod
 public | extras_exporttemplate_id_seq                   | sequence | netbox-prod
 public | extras_imageattachment                         | table    | netbox-prod
 public | extras_imageattachment_id_seq                  | sequence | netbox-prod
 public | extras_journalentry                            | table    | netbox-prod
 public | extras_journalentry_id_seq                     | sequence | netbox-prod
 public | extras_objectchange                            | table    | netbox-prod
 public | extras_objectchange_id_seq                     | sequence | netbox-prod
 public | extras_savedfilter                             | table    | netbox-prod
 public | extras_savedfilter_content_types               | table    | netbox-prod
 public | extras_savedfilter_content_types_id_seq        | sequence | netbox-prod
 public | extras_savedfilter_id_seq                      | sequence | netbox-prod
 public | extras_stagedchange                            | table    | netbox-prod
 public | extras_stagedchange_id_seq                     | sequence | netbox-prod
 public | extras_tag                                     | table    | netbox-prod
 public | extras_tag_id_seq                              | sequence | netbox-prod
 public | extras_tag_object_types                        | table    | netbox-prod
 public | extras_tag_object_types_id_seq                 | sequence | netbox-prod
 public | extras_taggeditem                              | table    | netbox-prod
 public | extras_taggeditem_id_seq                       | sequence | netbox-prod
 public | extras_webhook                                 | table    | netbox-prod
 public | extras_webhook_id_seq                          | sequence | netbox-prod
 public | ipam_aggregate                                 | table    | netbox-prod
 public | ipam_aggregate_id_seq                          | sequence | netbox-prod
 public | ipam_asn                                       | table    | netbox-prod
 public | ipam_asn_id_seq                                | sequence | netbox-prod
 public | ipam_asnrange                                  | table    | netbox-prod
 public | ipam_asnrange_id_seq                           | sequence | netbox-prod
 public | ipam_fhrpgroup                                 | table    | netbox-prod
 public | ipam_fhrpgroup_id_seq                          | sequence | netbox-prod
 public | ipam_fhrpgroupassignment                       | table    | netbox-prod
 public | ipam_fhrpgroupassignment_id_seq                | sequence | netbox-prod
 public | ipam_ipaddress                                 | table    | netbox-prod
 public | ipam_ipaddress_id_seq                          | sequence | netbox-prod
 public | ipam_iprange                                   | table    | netbox-prod
 public | ipam_iprange_id_seq                            | sequence | netbox-prod
 public | ipam_l2vpn_export_targets_id_seq               | sequence | netbox-prod
 public | ipam_l2vpn_id_seq                              | sequence | netbox-prod
 public | ipam_l2vpn_import_targets_id_seq               | sequence | netbox-prod
 public | ipam_l2vpntermination_id_seq                   | sequence | netbox-prod
 public | ipam_prefix                                    | table    | netbox-prod
 public | ipam_prefix_id_seq                             | sequence | netbox-prod
 public | ipam_rir                                       | table    | netbox-prod
 public | ipam_rir_id_seq                                | sequence | netbox-prod
 public | ipam_role                                      | table    | netbox-prod
 public | ipam_role_id_seq                               | sequence | netbox-prod
 public | ipam_routetarget                               | table    | netbox-prod
 public | ipam_routetarget_id_seq                        | sequence | netbox-prod
 public | ipam_service                                   | table    | netbox-prod
 public | ipam_service_id_seq                            | sequence | netbox-prod
 public | ipam_service_ipaddresses                       | table    | netbox-prod
 public | ipam_service_ipaddresses_id_seq                | sequence | netbox-prod
 public | ipam_servicetemplate                           | table    | netbox-prod
 public | ipam_servicetemplate_id_seq                    | sequence | netbox-prod
 public | ipam_vlan                                      | table    | netbox-prod
 public | ipam_vlan_id_seq                               | sequence | netbox-prod
 public | ipam_vlangroup                                 | table    | netbox-prod
 public | ipam_vlangroup_id_seq                          | sequence | netbox-prod
 public | ipam_vrf                                       | table    | netbox-prod
 public | ipam_vrf_export_targets                        | table    | netbox-prod
 public | ipam_vrf_export_targets_id_seq                 | sequence | netbox-prod
 public | ipam_vrf_id_seq                                | sequence | netbox-prod
 public | ipam_vrf_import_targets                        | table    | netbox-prod
 public | ipam_vrf_import_targets_id_seq                 | sequence | netbox-prod
 public | secrets_secret                                 | table    | netbox-prod
 public | secrets_secret_id_seq                          | sequence | netbox-prod
 public | secrets_secretrole                             | table    | netbox-prod
 public | secrets_secretrole_id_seq                      | sequence | netbox-prod
 public | secrets_sessionkey                             | table    | netbox-prod
 public | secrets_sessionkey_id_seq                      | sequence | netbox-prod
 public | secrets_userkey                                | table    | netbox-prod
 public | secrets_userkey_id_seq                         | sequence | netbox-prod
 public | social_auth_association                        | table    | netbox-prod
 public | social_auth_association_id_seq                 | sequence | netbox-prod
 public | social_auth_code                               | table    | netbox-prod
 public | social_auth_code_id_seq                        | sequence | netbox-prod
 public | social_auth_nonce                              | table    | netbox-prod
 public | social_auth_nonce_id_seq                       | sequence | netbox-prod
 public | social_auth_partial                            | table    | netbox-prod
 public | social_auth_partial_id_seq                     | sequence | netbox-prod
 public | social_auth_usersocialauth                     | table    | netbox-prod
 public | social_auth_usersocialauth_id_seq              | sequence | netbox-prod
 public | taggit_tag                                     | table    | netbox-prod
 public | taggit_tag_id_seq                              | sequence | netbox-prod
 public | taggit_taggeditem                              | table    | netbox-prod
 public | taggit_taggeditem_id_seq                       | sequence | netbox-prod
 public | tenancy_contact                                | table    | netbox-prod
 public | tenancy_contact_id_seq                         | sequence | netbox-prod
 public | tenancy_contactassignment                      | table    | netbox-prod
 public | tenancy_contactassignment_id_seq               | sequence | netbox-prod
 public | tenancy_contactgroup                           | table    | netbox-prod
 public | tenancy_contactgroup_id_seq                    | sequence | netbox-prod
 public | tenancy_contactrole                            | table    | netbox-prod
 public | tenancy_contactrole_id_seq                     | sequence | netbox-prod
 public | tenancy_tenant                                 | table    | netbox-prod
 public | tenancy_tenant_id_seq                          | sequence | netbox-prod
 public | tenancy_tenantgroup                            | table    | netbox-prod
 public | tenancy_tenantgroup_id_seq                     | sequence | netbox-prod
 public | users_objectpermission                         | table    | netbox-prod
 public | users_objectpermission_groups                  | table    | netbox-prod
 public | users_objectpermission_groups_id_seq           | sequence | netbox-prod
 public | users_objectpermission_id_seq                  | sequence | netbox-prod
 public | users_objectpermission_object_types            | table    | netbox-prod
 public | users_objectpermission_object_types_id_seq     | sequence | netbox-prod
 public | users_objectpermission_users                   | table    | netbox-prod
 public | users_objectpermission_users_id_seq            | sequence | netbox-prod
 public | users_token                                    | table    | netbox-prod
 public | users_token_id_seq                             | sequence | netbox-prod
 public | users_userconfig                               | table    | netbox-prod
 public | users_userconfig_id_seq                        | sequence | netbox-prod
 public | virtualization_cluster                         | table    | netbox-prod
 public | virtualization_cluster_id_seq                  | sequence | netbox-prod
 public | virtualization_clustergroup                    | table    | netbox-prod
 public | virtualization_clustergroup_id_seq             | sequence | netbox-prod
 public | virtualization_clustertype                     | table    | netbox-prod
 public | virtualization_clustertype_id_seq              | sequence | netbox-prod
 public | virtualization_virtualdisk                     | table    | netbox-prod
 public | virtualization_virtualdisk_id_seq              | sequence | netbox-prod
 public | virtualization_virtualmachine                  | table    | netbox-prod
 public | virtualization_virtualmachine_id_seq           | sequence | netbox-prod
 public | virtualization_vminterface                     | table    | netbox-prod
 public | virtualization_vminterface_id_seq              | sequence | netbox-prod
 public | virtualization_vminterface_tagged_vlans        | table    | netbox-prod
 public | virtualization_vminterface_tagged_vlans_id_seq | sequence | netbox-prod
 public | vpn_ikepolicy                                  | table    | netbox-prod
 public | vpn_ikepolicy_id_seq                           | sequence | netbox-prod
 public | vpn_ikepolicy_proposals                        | table    | netbox-prod
 public | vpn_ikepolicy_proposals_id_seq                 | sequence | netbox-prod
 public | vpn_ikeproposal                                | table    | netbox-prod
 public | vpn_ikeproposal_id_seq                         | sequence | netbox-prod
 public | vpn_ipsecpolicy                                | table    | netbox-prod
 public | vpn_ipsecpolicy_id_seq                         | sequence | netbox-prod
 public | vpn_ipsecpolicy_proposals                      | table    | netbox-prod
 public | vpn_ipsecpolicy_proposals_id_seq               | sequence | netbox-prod
 public | vpn_ipsecprofile                               | table    | netbox-prod
 public | vpn_ipsecprofile_id_seq                        | sequence | netbox-prod
 public | vpn_ipsecproposal                              | table    | netbox-prod
 public | vpn_ipsecproposal_id_seq                       | sequence | netbox-prod
 public | vpn_l2vpn                                      | table    | netbox-prod
 public | vpn_l2vpn_export_targets                       | table    | netbox-prod
 public | vpn_l2vpn_import_targets                       | table    | netbox-prod
 public | vpn_l2vpntermination                           | table    | netbox-prod
 public | vpn_tunnel                                     | table    | netbox-prod
 public | vpn_tunnel_id_seq                              | sequence | netbox-prod
 public | vpn_tunnelgroup                                | table    | netbox-prod
 public | vpn_tunnelgroup_id_seq                         | sequence | netbox-prod
 public | vpn_tunneltermination                          | table    | netbox-prod
 public | vpn_tunneltermination_id_seq                   | sequence | netbox-prod
 public | wireless_wirelesslan                           | table    | netbox-prod
 public | wireless_wirelesslan_id_seq                    | sequence | netbox-prod
 public | wireless_wirelesslangroup                      | table    | netbox-prod
 public | wireless_wirelesslangroup_id_seq               | sequence | netbox-prod
 public | wireless_wirelesslink                          | table    | netbox-prod
 public | wireless_wirelesslink_id_seq                   | sequence | netbox-prod
(345 rows)

I think it's named extras_customfield_obj_type_id_seq:

netbox-dev=# \d extras_customfield_content_types
                              Table "public.extras_customfield_content_types"
     Column     |  Type   | Collation | Nullable |                         Default
----------------+---------+-----------+----------+---------------------------------------------------------
 id             | integer |           | not null | nextval('extras_customfield_obj_type_id_seq'::regclass)
jeremystretch commented 5 months ago

I think it's named extras_customfield_obj_type_id_seq

Yep, that seems to be the issue. This was from the old obj_type field on the CustomField model from before custom fields were completely reimplemented way back in NetBox v2.10. Unfortunately the sequence was never renamed accordingly for upgraded deployments.

We can tweak the v4.0 migration to check for either sequence name and rename it accordingly.

jeremystretch commented 5 months ago

@opericgithub @a084ed22 @tobiasge (and anyone else experiencing this issue), could you try checking out the 15605-migrate-old-sequence branch and attempting the migration again? Should be fixed.

tobiasge commented 5 months ago

The content types are now working for out database.

Applying extras.0111_rename_content_types... OK

But now I'm seeing a new error:

Applying users.0006_custom_group_model...
psycopg.errors.ForeignKeyViolation: update or delete on table "auth_group" violates foreign key constraint "auth_group_permissions_group_id_b120cbf9_fk_auth_group_id" on table "auth_group_permissions"
DETAIL:  Key (id)=(10) is still referenced from table "auth_group_permissions".
a084ed22 commented 5 months ago

I can confirm that the fix allows the specific migration to work, however I'm later stuck on this:

Running migrations:
  Applying users.0008_flip_objectpermission_assignments...Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedObject: constraint "users_objectpermissi_objectpermission_id_2f7cc117_fk_users_obj" for table "users_group_object_permissions" does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 37, in database_forwards
    database_operation.database_forwards(
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 106, in database_forwards
    self._run_sql(schema_editor, self.sql)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 133, in _run_sql
    schema_editor.execute(statement, params=None)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/postgresql/schema.py", line 45, in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 201, in execute
    cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: constraint "users_objectpermissi_objectpermission_id_2f7cc117_fk_users_obj" for table "users_group_object_permissions" does not exist
opericgithub commented 5 months ago

@opericgithub @a084ed22 @tobiasge (and anyone else experiencing this issue), could you try checking out the 15605-migrate-old-sequence branch and attempting the migration again? Should be fixed.

Sure. I tried checking out to that branch, upgrade completes without errors, but I stay on the same version (3.7.5). Maybe I'm doing something wrong?

cd /opt/netbox
sudo git pull origin 15605-migrate-old-sequence

remote: Enumerating objects: 11713, done.
remote: Counting objects: 100% (6519/6519), done.
remote: Compressing objects: 100% (1721/1721), done.
remote: Total 5115 (delta 4177), reused 4181 (delta 3374), pack-reused 0
Receiving objects: 100% (5115/5115), 8.24 MiB | 23.97 MiB/s, done.
Resolving deltas: 100% (4177/4177), completed with 527 local objects.
From https://github.com/netbox-community/netbox
 * branch                15605-migrate-old-sequence -> FETCH_HEAD

hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint: 
hint:   git config pull.rebase false  # merge (the default strategy)
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint: 
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

sudo ./upgrade.sh 

You are installing (or upgrading to) NetBox version 3.7.5
Using Python 3.10.12
Removing old virtual environment...
Creating a new virtual environment at /opt/netbox/venv...
Updating pip (pip install --upgrade pip)...
...
Successfully installed django-auth-ldap-4.8.0 pyasn1-0.6.0 pyasn1-modules-0.4.0 python-ldap-3.4.4
Applying database migrations (python3 netbox/manage.py migrate)...
Operations to perform:
  Apply all migrations: account, admin, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, vpn, wireless
Running migrations:
  No migrations to apply.
...
Completed. Total entries: 36437
Removing expired user sessions (python3 netbox/manage.py clearsessions)...
Upgrade complete! Don't forget to restart the NetBox services:
  > sudo systemctl restart netbox netbox-rq
jeremystretch commented 5 months ago

I'm going to merge PR #15661 and consider this fixed as the above feedback indicates that the extras.0111_rename_content_types migration now completes successfully even for very old databases.

If you continue to experience different errors when attempting to upgrade, please submit a new bug report with details.

@opericgithub Try just running the migration command (manage.py migrate) after checking out the branch, instead of the whole upgrade script.