search

netbox-community / netbox

The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/free-netbox-cloud/
http://netboxlabs.com/oss/netbox/
Apache License 2.0
15.78k stars 2.54k forks source link

Error upgrading from 4.0.11 to 4.1.0 #17401

Open avbor opened 1 week ago

avbor commented 1 week ago

Deployment Type

Self-hosted

NetBox Version

v4.1.0

Python Version

3.12

Steps to Reproduce

Trying to upgrade from 4.0.11 to 4.1.0 rise migration error.

Expected Behavior

No errors during upgrade.

Observed Behavior

Please don't close this issue without investigate, we have several closed unresolved issues in this repo and netbox-docker. Like https://github.com/netbox-community/netbox/issues/17352 and https://github.com/netbox-community/netbox-docker/issues/1314 In https://github.com/netbox-community/netbox-docker/issues/1313 @tobiasge wrote Needs to be fixed in Netbox itself. https://github.com/netbox-community/netbox-docker/issues/1313#issuecomment-2333929966

Also some discussions abount this problem - https://github.com/netbox-community/netbox/discussions/17382 and https://github.com/netbox-community/netbox/discussions/17388

Log from netbox side:

Running migrations:
Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedTable: relation "extras_objectchange" does not exist
LINE 1: ...ct_type_id", "extras_objectchange"."user_id" FROM "extras_ob...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/netbox/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox/netbox/extras/migrations/0117_move_objectchange.py", line 8, in update_content_types
    ContentType.objects.filter(app_label='core', model='objectchange').delete()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/query.py", line 1187, in delete
    collector.collect(del_query)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/deletion.py", line 348, in collect
    if getattr(on_delete, "lazy_sub_objs", False) or sub_objs:
                                                     ^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/query.py", line 414, in __bool__
    self._fetch_all()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: relation "extras_objectchange" does not exist
LINE 1: ...ct_type_id", "extras_objectchange"."user_id" FROM "extras_ob...

log from postgre:

2024-09-06 14:38:46.022 UTC [1] LOG:  starting PostgreSQL 16.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
2024-09-06 14:38:46.023 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-09-06 14:38:46.023 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-09-06 14:38:46.030 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-09-06 14:38:46.040 UTC [29] LOG:  database system was shut down at 2024-09-06 14:38:44 UTC
2024-09-06 14:38:46.058 UTC [1] LOG:  database system is ready to accept connections
2024-09-06 14:38:55.130 UTC [40] LOG:  statement: SELECT "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_content_type" WHERE (("django_content_type"."app_label" = 'auth' AND "django_content_type"."model" IN ('group', 'permission')) OR ("django_content_type"."app_label" = 'contenttypes' AND "django_content_type"."model" IN ('contenttype')) OR ("django_content_type"."app_label" = 'sessions' AND "django_content_type"."model" IN ('session')) OR ("django_content_type"."app_label" = 'social_django' AND "django_content_type"."model" IN ('usersocialauth', 'partial', 'nonce', 'code', 'association')) OR ("django_content_type"."app_label" = 'taggit' AND "django_content_type"."model" IN ('taggeditem', 'tag')) OR ("django_content_type"."app_label" = 'core' AND "django_content_type"."model" IN ('managedfile', 'job', 'datasource', 'datafile', 'objectchange', 'autosyncrecord', 'configrevision')) OR ("django_content_type"."app_label" = 'users' AND "django_content_type"."model" IN ('group', 'token', 'objectpermission', 'userconfig', 'user')) OR ("django_content_type"."app_label" = 'circuits' AND "django_content_type"."model" IN ('provider', 'circuittermination', 'providernetwork', 'circuitgroup', 'circuit', 'provideraccount', 'circuittype', 'circuitgroupassignment')) OR ("django_content_type"."app_label" = 'dcim' AND "django_content_type"."model" IN ('frontporttemplate', 'powerport', 'powerporttemplate', 'platform', 'consoleserverport', 'interface', 'manufacturer', 'poweroutlettemplate', 'devicerole', 'consoleport', 'consoleserverporttemplate', 'rackreservation', 'site', 'racktype', 'frontport', 'rearport', 'module', 'moduletype', 'virtualdevicecontext', 'region', 'modulebay', 'inventoryitemtemplate', 'devicetype', 'devicebay', 'virtualchassis', 'devicebaytemplate', 'poweroutlet', 'inventoryitemrole', 'interfacetemplate', 'cablepath', 'cable', 'device', 'rearporttemplate', 'modulebaytemplate', 'powerpanel', 'consoleporttemplate', 'rack', 'rackrole', 'location', 'powerfeed', 'inventoryitem', 'cabletermination', 'sitegroup')) OR ("django_content_type"."app_label" = 'ipam' AND "django_content_type"."model" IN ('servicetemplate', 'iprange', 'vlan', 'fhrpgroupassignment', 'asnrange', 'role', 'ipaddress', 'service', 'vlangroup', 'fhrpgroup', 'routetarget', 'aggregate', 'rir', 'prefix', 'vrf', 'asn')) OR ("django_content_type"."app_label" = 'extras' AND "django_content_type"."model" IN ('bookmark', 'imageattachment', 'cachedvalue', 'notification', 'taggeditem', 'journalentry', 'eventrule', 'stagedchange', 'configtemplate', 'webhook', 'customfield', 'subscription', 'tag', 'savedfilter', 'configcontext', 'customfieldchoiceset', 'branch', 'customlink', 'notificationgroup', 'script', 'dashboard', 'exporttemplate')) OR ("django_content_type"."app_label" = 'tenancy' AND "django_content_type"."model" IN ('contact', 'contactassignment', 'tenant', 'tenantgroup', 'contactgroup', 'contactrole')) OR ("django_content_type"."app_label" = 'virtualization' AND "django_content_type"."model" IN ('clustergroup', 'virtualdisk', 'vminterface', 'cluster', 'clustertype', 'virtualmachine')) OR ("django_content_type"."app_label" = 'vpn' AND "django_content_type"."model" IN ('tunnel', 'ikeproposal', 'ipsecproposal', 'l2vpntermination', 'tunnelgroup', 'tunneltermination', 'ipsecpolicy', 'ipsecprofile', 'l2vpn', 'ikepolicy')) OR ("django_content_type"."app_label" = 'wireless' AND "django_content_type"."model" IN ('wirelesslangroup', 'wirelesslan', 'wirelesslink')) OR ("django_content_type"."app_label" = 'django_rq' AND "django_content_type"."model" IN ('queue')) OR ("django_content_type"."app_label" = 'netbox_dns' AND "django_content_type"."model" IN ('zone', 'recordtemplate', 'registrar', 'view', 'record', 'zonetemplate', 'registrationcontact', 'nameserver')))
2024-09-06 14:38:56.719 UTC [40] LOG:  statement:
                    SELECT
                        c.relname,
                        CASE
                            WHEN c.relispartition THEN 'p'
                            WHEN c.relkind IN ('m', 'v') THEN 'v'
                            ELSE 't'
                        END,
                        obj_description(c.oid, 'pg_class')
                    FROM pg_catalog.pg_class c
                    LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    WHERE c.relkind IN ('f', 'm', 'p', 'r', 'v')
                        AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                        AND pg_catalog.pg_table_is_visible(c.oid)

2024-09-06 14:38:56.762 UTC [40] LOG:  statement: SELECT "django_migrations"."id", "django_migrations"."app", "django_migrations"."name", "django_migrations"."applied" FROM "django_migrations"
2024-09-06 14:38:56.781 UTC [40] LOG:  statement:
                    SELECT
                        c.relname,
                        CASE
                            WHEN c.relispartition THEN 'p'
                            WHEN c.relkind IN ('m', 'v') THEN 'v'
                            ELSE 't'
                        END,
                        obj_description(c.oid, 'pg_class')
                    FROM pg_catalog.pg_class c
                    LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    WHERE c.relkind IN ('f', 'm', 'p', 'r', 'v')
                        AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                        AND pg_catalog.pg_table_is_visible(c.oid)

2024-09-06 14:38:56.797 UTC [40] LOG:  statement: SELECT "django_migrations"."id", "django_migrations"."app", "django_migrations"."name", "django_migrations"."applied" FROM "django_migrations"
2024-09-06 14:39:08.064 UTC [48] LOG:  statement: SELECT "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_content_type" WHERE (("django_content_type"."app_label" = 'auth' AND "django_content_type"."model" IN ('permission', 'group')) OR ("django_content_type"."app_label" = 'contenttypes' AND "django_content_type"."model" IN ('contenttype')) OR ("django_content_type"."app_label" = 'sessions' AND "django_content_type"."model" IN ('session')) OR ("django_content_type"."app_label" = 'social_django' AND "django_content_type"."model" IN ('code', 'nonce', 'usersocialauth', 'association', 'partial')) OR ("django_content_type"."app_label" = 'taggit' AND "django_content_type"."model" IN ('taggeditem', 'tag')) OR ("django_content_type"."app_label" = 'core' AND "django_content_type"."model" IN ('datasource', 'job', 'autosyncrecord', 'managedfile', 'objectchange', 'configrevision', 'datafile')) OR ("django_content_type"."app_label" = 'users' AND "django_content_type"."model" IN ('user', 'token', 'group', 'userconfig', 'objectpermission')) OR ("django_content_type"."app_label" = 'circuits' AND "django_content_type"."model" IN ('circuittype', 'provideraccount', 'circuit', 'circuitgroupassignment', 'circuittermination', 'providernetwork', 'provider', 'circuitgroup')) OR ("django_content_type"."app_label" = 'dcim' AND "django_content_type"."model" IN ('cablepath', 'consoleporttemplate', 'virtualchassis', 'powerport', 'interface', 'cable', 'rackrole', 'consoleserverport', 'inventoryitemrole', 'moduletype', 'rack', 'region', 'modulebaytemplate', 'devicerole', 'powerporttemplate', 'poweroutlettemplate', 'poweroutlet', 'devicebay', 'manufacturer', 'cabletermination', 'module', 'modulebay', 'site', 'frontporttemplate', 'powerfeed', 'platform', 'rearporttemplate', 'rearport', 'interfacetemplate', 'consoleport', 'virtualdevicecontext', 'device', 'devicetype', 'powerpanel', 'frontport', 'location', 'devicebaytemplate', 'inventoryitemtemplate', 'sitegroup', 'inventoryitem', 'racktype', 'rackreservation', 'consoleserverporttemplate')) OR ("django_content_type"."app_label" = 'ipam' AND "django_content_type"."model" IN ('vrf', 'routetarget', 'role', 'rir', 'servicetemplate', 'asn', 'aggregate', 'fhrpgroup', 'prefix', 'iprange', 'vlangroup', 'asnrange', 'service', 'ipaddress', 'fhrpgroupassignment', 'vlan')) OR ("django_content_type"."app_label" = 'extras' AND "django_content_type"."model" IN ('customfield', 'bookmark', 'configcontext', 'webhook', 'tag', 'customfieldchoiceset', 'dashboard', 'customlink', 'journalentry', 'stagedchange', 'taggeditem', 'branch', 'imageattachment', 'notification', 'script', 'exporttemplate', 'configtemplate', 'cachedvalue', 'eventrule', 'subscription', 'savedfilter', 'notificationgroup')) OR ("django_content_type"."app_label" = 'tenancy' AND "django_content_type"."model" IN ('contactassignment', 'tenant', 'tenantgroup', 'contact', 'contactgroup', 'contactrole')) OR ("django_content_type"."app_label" = 'virtualization' AND "django_content_type"."model" IN ('cluster', 'vminterface', 'clustertype', 'virtualdisk', 'virtualmachine', 'clustergroup')) OR ("django_content_type"."app_label" = 'vpn' AND "django_content_type"."model" IN ('ipsecprofile', 'l2vpntermination', 'tunnel', 'ipsecpolicy', 'tunneltermination', 'ikeproposal', 'tunnelgroup', 'ipsecproposal', 'ikepolicy', 'l2vpn')) OR ("django_content_type"."app_label" = 'wireless' AND "django_content_type"."model" IN ('wirelesslangroup', 'wirelesslan', 'wirelesslink')) OR ("django_content_type"."app_label" = 'django_rq' AND "django_content_type"."model" IN ('queue')) OR ("django_content_type"."app_label" = 'netbox_dns' AND "django_content_type"."model" IN ('registrar', 'zone', 'record', 'recordtemplate', 'nameserver', 'zonetemplate', 'registrationcontact', 'view')))
2024-09-06 14:39:09.654 UTC [48] LOG:  statement:
                    SELECT
                        c.relname,
                        CASE
                            WHEN c.relispartition THEN 'p'
                            WHEN c.relkind IN ('m', 'v') THEN 'v'
                            ELSE 't'
                        END,
                        obj_description(c.oid, 'pg_class')
                    FROM pg_catalog.pg_class c
                    LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    WHERE c.relkind IN ('f', 'm', 'p', 'r', 'v')
                        AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                        AND pg_catalog.pg_table_is_visible(c.oid)

2024-09-06 14:39:09.698 UTC [48] LOG:  statement: SELECT "django_migrations"."id", "django_migrations"."app", "django_migrations"."name", "django_migrations"."applied" FROM "django_migrations"
2024-09-06 14:39:09.717 UTC [48] LOG:  statement:
                    SELECT
                        c.relname,
                        CASE
                            WHEN c.relispartition THEN 'p'
                            WHEN c.relkind IN ('m', 'v') THEN 'v'
                            ELSE 't'
                        END,
                        obj_description(c.oid, 'pg_class')
                    FROM pg_catalog.pg_class c
                    LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    WHERE c.relkind IN ('f', 'm', 'p', 'r', 'v')
                        AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                        AND pg_catalog.pg_table_is_visible(c.oid)

2024-09-06 14:39:09.732 UTC [48] LOG:  statement: SELECT "django_migrations"."id", "django_migrations"."app", "django_migrations"."name", "django_migrations"."applied" FROM "django_migrations"
2024-09-06 14:39:20.724 UTC [56] LOG:  statement: SELECT "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_content_type" WHERE (("django_content_type"."app_label" = 'auth' AND "django_content_type"."model" IN ('group', 'permission')) OR ("django_content_type"."app_label" = 'contenttypes' AND "django_content_type"."model" IN ('contenttype')) OR ("django_content_type"."app_label" = 'sessions' AND "django_content_type"."model" IN ('session')) OR ("django_content_type"."app_label" = 'social_django' AND "django_content_type"."model" IN ('partial', 'nonce', 'code', 'association', 'usersocialauth')) OR ("django_content_type"."app_label" = 'taggit' AND "django_content_type"."model" IN ('tag', 'taggeditem')) OR ("django_content_type"."app_label" = 'core' AND "django_content_type"."model" IN ('managedfile', 'datafile', 'job', 'configrevision', 'objectchange', 'autosyncrecord', 'datasource')) OR ("django_content_type"."app_label" = 'users' AND "django_content_type"."model" IN ('user', 'group', 'userconfig', 'objectpermission', 'token')) OR ("django_content_type"."app_label" = 'circuits' AND "django_content_type"."model" IN ('circuit', 'provideraccount', 'circuitgroup', 'circuitgroupassignment', 'circuittype', 'providernetwork', 'provider', 'circuittermination')) OR ("django_content_type"."app_label" = 'dcim' AND "django_content_type"."model" IN ('powerpanel', 'devicerole', 'manufacturer', 'location', 'cable', 'devicetype', 'powerport', 'platform', 'racktype', 'consoleserverporttemplate', 'rackreservation', 'poweroutlettemplate', 'device', 'cabletermination', 'region', 'powerporttemplate', 'inventoryitemrole', 'moduletype', 'site', 'devicebay', 'module', 'interface', 'frontporttemplate', 'virtualdevicecontext', 'rackrole', 'virtualchassis', 'inventoryitem', 'modulebaytemplate', 'rearporttemplate', 'interfacetemplate', 'cablepath', 'sitegroup', 'powerfeed', 'consoleporttemplate', 'frontport', 'consoleserverport', 'devicebaytemplate', 'inventoryitemtemplate', 'poweroutlet', 'rearport', 'consoleport', 'modulebay', 'rack')) OR ("django_content_type"."app_label" = 'ipam' AND "django_content_type"."model" IN ('asn', 'vlangroup', 'iprange', 'ipaddress', 'vrf', 'routetarget', 'rir', 'prefix', 'servicetemplate', 'aggregate', 'asnrange', 'role', 'service', 'vlan', 'fhrpgroup', 'fhrpgroupassignment')) OR ("django_content_type"."app_label" = 'extras' AND "django_content_type"."model" IN ('stagedchange', 'imageattachment', 'exporttemplate', 'notificationgroup', 'tag', 'eventrule', 'webhook', 'journalentry', 'customfield', 'customlink', 'configcontext', 'customfieldchoiceset', 'savedfilter', 'bookmark', 'subscription', 'dashboard', 'cachedvalue', 'script', 'notification', 'taggeditem', 'configtemplate', 'branch')) OR ("django_content_type"."app_label" = 'tenancy' AND "django_content_type"."model" IN ('contact', 'contactrole', 'tenant', 'tenantgroup', 'contactgroup', 'contactassignment')) OR ("django_content_type"."app_label" = 'virtualization' AND "django_content_type"."model" IN ('cluster', 'vminterface', 'virtualmachine', 'virtualdisk', 'clustergroup', 'clustertype')) OR ("django_content_type"."app_label" = 'vpn' AND "django_content_type"."model" IN ('l2vpn', 'ipsecproposal', 'ikepolicy', 'tunnel', 'tunneltermination', 'ipsecpolicy', 'tunnelgroup', 'ipsecprofile', 'l2vpntermination', 'ikeproposal')) OR ("django_content_type"."app_label" = 'wireless' AND "django_content_type"."model" IN ('wirelesslangroup', 'wirelesslink', 'wirelesslan')) OR ("django_content_type"."app_label" = 'django_rq' AND "django_content_type"."model" IN ('queue')) OR ("django_content_type"."app_label" = 'netbox_dns' AND "django_content_type"."model" IN ('nameserver', 'zonetemplate', 'view', 'record', 'registrar', 'recordtemplate', 'registrationcontact', 'zone')))
2024-09-06 14:39:22.333 UTC [56] LOG:  statement:
                    SELECT
                        c.relname,
                        CASE
                            WHEN c.relispartition THEN 'p'
                            WHEN c.relkind IN ('m', 'v') THEN 'v'
                            ELSE 't'
                        END,
                        obj_description(c.oid, 'pg_class')
                    FROM pg_catalog.pg_class c
                    LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    WHERE c.relkind IN ('f', 'm', 'p', 'r', 'v')
                        AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                        AND pg_catalog.pg_table_is_visible(c.oid)

2024-09-06 14:39:22.375 UTC [56] LOG:  statement: SELECT "django_migrations"."id", "django_migrations"."app", "django_migrations"."name", "django_migrations"."applied" FROM "django_migrations"
2024-09-06 14:39:22.395 UTC [56] LOG:  statement:
                    SELECT
                        c.relname,
                        CASE
                            WHEN c.relispartition THEN 'p'
                            WHEN c.relkind IN ('m', 'v') THEN 'v'
                            ELSE 't'
                        END,
                        obj_description(c.oid, 'pg_class')
                    FROM pg_catalog.pg_class c
                    LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    WHERE c.relkind IN ('f', 'm', 'p', 'r', 'v')
                        AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                        AND pg_catalog.pg_table_is_visible(c.oid)

2024-09-06 14:39:22.412 UTC [56] LOG:  statement: SELECT "django_migrations"."id", "django_migrations"."app", "django_migrations"."name", "django_migrations"."applied" FROM "django_migrations"
2024-09-06 14:39:22.887 UTC [56] LOG:  statement:
                    SELECT
                        c.relname,
                        CASE
                            WHEN c.relispartition THEN 'p'
                            WHEN c.relkind IN ('m', 'v') THEN 'v'
                            ELSE 't'
                        END,
                        obj_description(c.oid, 'pg_class')
                    FROM pg_catalog.pg_class c
                    LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    WHERE c.relkind IN ('f', 'm', 'p', 'r', 'v')
                        AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                        AND pg_catalog.pg_table_is_visible(c.oid)

2024-09-06 14:39:22.981 UTC [56] LOG:  statement: BEGIN
2024-09-06 14:39:22.981 UTC [56] LOG:  statement: ALTER TABLE "extras_objectchange" RENAME TO "core_objectchange"
2024-09-06 14:39:22.989 UTC [56] LOG:  statement: ALTER TABLE extras_objectchange_id_seq  RENAME TO core_objectchange_id_seq
2024-09-06 14:39:22.992 UTC [56] LOG:  statement: ALTER INDEX extras_objectchange_pkey  RENAME TO core_objectchange_pkey
2024-09-06 14:39:23.000 UTC [56] LOG:  statement: ALTER INDEX extras_obje_changed_927fe5_idx  RENAME TO core_objectchange_changed_object_type_id_cha_79a9ed1e
2024-09-06 14:39:23.004 UTC [56] LOG:  statement: ALTER INDEX extras_obje_related_bfcdef_idx  RENAME TO core_objectchange_related_object_type_id_rel_a71d604a
2024-09-06 14:39:23.009 UTC [56] LOG:  statement: ALTER INDEX extras_objectchange_changed_object_type_id_b755bb60  RENAME TO core_objectchange_changed_object_type_id_2070ade6
2024-09-06 14:39:23.012 UTC [56] LOG:  statement: ALTER INDEX extras_objectchange_related_object_type_id_fe6e521f  RENAME TO core_objectchange_related_object_type_id_b80958af
2024-09-06 14:39:23.016 UTC [56] LOG:  statement: ALTER INDEX extras_objectchange_request_id_4ae21e90  RENAME TO core_objectchange_request_id_d9d160ac
2024-09-06 14:39:23.019 UTC [56] LOG:  statement: ALTER INDEX extras_objectchange_time_224380ea  RENAME TO core_objectchange_time_800f60a5
2024-09-06 14:39:23.022 UTC [56] LOG:  statement: ALTER INDEX extras_objectchange_user_id_7fdf8186  RENAME TO core_objectchange_user_id_2b2142be
2024-09-06 14:39:23.025 UTC [56] LOG:  statement: ALTER TABLE core_objectchange RENAME CONSTRAINT extras_objectchange_changed_object_id_check TO core_objectchange_changed_object_id_check
2024-09-06 14:39:23.029 UTC [56] LOG:  statement: ALTER TABLE core_objectchange RENAME CONSTRAINT extras_objectchange_related_object_id_check TO core_objectchange_related_object_id_check
2024-09-06 14:39:23.032 UTC [56] LOG:  statement: ALTER TABLE core_objectchange RENAME CONSTRAINT extras_objectchange_changed_object_type__b755bb60_fk_django_co TO core_objectchange_changed_object_type_id_2070ade6
2024-09-06 14:39:23.036 UTC [56] LOG:  statement: ALTER TABLE core_objectchange RENAME CONSTRAINT extras_objectchange_related_object_type__fe6e521f_fk_django_co TO core_objectchange_related_object_type_id_b80958af
2024-09-06 14:39:23.039 UTC [56] LOG:  statement: ALTER TABLE core_objectchange RENAME CONSTRAINT extras_objectchange_user_id_7fdf8186_fk_auth_user_id TO core_objectchange_user_id_2b2142be
2024-09-06 14:39:23.043 UTC [56] LOG:  statement: SELECT "django_content_type"."id", "django_content_type"."app_label", "django_content_type"."model" FROM "django_content_type" WHERE ("django_content_type"."app_label" = 'core' AND "django_content_type"."model" = 'objectchange')
2024-09-06 14:39:23.046 UTC [56] LOG:  statement: SELECT "auth_permission"."id", "auth_permission"."name", "auth_permission"."content_type_id", "auth_permission"."codename" FROM "auth_permission" INNER JOIN "django_content_type" ON ("auth_permission"."content_type_id" = "django_content_type"."id") WHERE "auth_permission"."content_type_id" IN (166) ORDER BY "django_content_type"."app_label" ASC, "django_content_type"."model" ASC, "auth_permission"."codename" ASC
2024-09-06 14:39:23.054 UTC [56] LOG:  statement: SELECT "core_autosyncrecord"."id", "core_autosyncrecord"."object_id", "core_autosyncrecord"."datafile_id", "core_autosyncrecord"."object_type_id" FROM "core_autosyncrecord" WHERE "core_autosyncrecord"."object_type_id" IN (166)
2024-09-06 14:39:23.061 UTC [56] LOG:  statement: SELECT "core_job"."id", "core_job"."object_id", "core_job"."name", "core_job"."created", "core_job"."scheduled", "core_job"."interval", "core_job"."started", "core_job"."completed", "core_job"."status", "core_job"."data", "core_job"."job_id", "core_job"."object_type_id", "core_job"."user_id", "core_job"."error" FROM "core_job" WHERE "core_job"."object_type_id" IN (166) ORDER BY "core_job"."created" DESC
2024-09-06 14:39:23.070 UTC [56] LOG:  statement: SELECT "core_objectchange"."id", "core_objectchange"."time", "core_objectchange"."user_name", "core_objectchange"."request_id", "core_objectchange"."action", "core_objectchange"."changed_object_id", "core_objectchange"."related_object_id", "core_objectchange"."object_repr", "core_objectchange"."prechange_data", "core_objectchange"."postchange_data", "core_objectchange"."changed_object_type_id", "core_objectchange"."related_object_type_id", "core_objectchange"."user_id" FROM "core_objectchange" WHERE "core_objectchange"."changed_object_type_id" IN (166) ORDER BY "core_objectchange"."time" DESC
2024-09-06 14:39:23.077 UTC [56] LOG:  statement: SELECT "core_objectchange"."id", "core_objectchange"."time", "core_objectchange"."user_name", "core_objectchange"."request_id", "core_objectchange"."action", "core_objectchange"."changed_object_id", "core_objectchange"."related_object_id", "core_objectchange"."object_repr", "core_objectchange"."prechange_data", "core_objectchange"."postchange_data", "core_objectchange"."changed_object_type_id", "core_objectchange"."related_object_type_id", "core_objectchange"."user_id" FROM "core_objectchange" WHERE "core_objectchange"."related_object_type_id" IN (166) ORDER BY "core_objectchange"."time" DESC
2024-09-06 14:39:23.082 UTC [56] LOG:  statement: SELECT "dcim_inventoryitem"."id", "dcim_inventoryitem"."created", "dcim_inventoryitem"."last_updated", "dcim_inventoryitem"."custom_field_data", "dcim_inventoryitem"."name", "dcim_inventoryitem"."_name", "dcim_inventoryitem"."label", "dcim_inventoryitem"."description", "dcim_inventoryitem"."part_id", "dcim_inventoryitem"."serial", "dcim_inventoryitem"."asset_tag", "dcim_inventoryitem"."discovered", "dcim_inventoryitem"."lft", "dcim_inventoryitem"."rght", "dcim_inventoryitem"."tree_id", "dcim_inventoryitem"."level", "dcim_inventoryitem"."device_id", "dcim_inventoryitem"."manufacturer_id", "dcim_inventoryitem"."parent_id", "dcim_inventoryitem"."role_id", "dcim_inventoryitem"."component_id", "dcim_inventoryitem"."component_type_id" FROM "dcim_inventoryitem" WHERE "dcim_inventoryitem"."component_type_id" IN (166) ORDER BY "dcim_inventoryitem"."device_id" ASC, "dcim_inventoryitem"."parent_id" ASC, "dcim_inventoryitem"."_name" ASC
2024-09-06 14:39:23.098 UTC [56] LOG:  statement: SELECT "dcim_inventoryitemtemplate"."id", "dcim_inventoryitemtemplate"."created", "dcim_inventoryitemtemplate"."last_updated", "dcim_inventoryitemtemplate"."name", "dcim_inventoryitemtemplate"."_name", "dcim_inventoryitemtemplate"."label", "dcim_inventoryitemtemplate"."description", "dcim_inventoryitemtemplate"."component_id", "dcim_inventoryitemtemplate"."part_id", "dcim_inventoryitemtemplate"."lft", "dcim_inventoryitemtemplate"."rght", "dcim_inventoryitemtemplate"."tree_id", "dcim_inventoryitemtemplate"."level", "dcim_inventoryitemtemplate"."component_type_id", "dcim_inventoryitemtemplate"."device_type_id", "dcim_inventoryitemtemplate"."manufacturer_id", "dcim_inventoryitemtemplate"."parent_id", "dcim_inventoryitemtemplate"."role_id" FROM "dcim_inventoryitemtemplate" WHERE "dcim_inventoryitemtemplate"."component_type_id" IN (166) ORDER BY "dcim_inventoryitemtemplate"."device_type_id" ASC, "dcim_inventoryitemtemplate"."parent_id" ASC, "dcim_inventoryitemtemplate"."_name" ASC
2024-09-06 14:39:23.108 UTC [56] LOG:  statement: SELECT "dcim_cabletermination"."id", "dcim_cabletermination"."cable_end", "dcim_cabletermination"."termination_id", "dcim_cabletermination"."cable_id", "dcim_cabletermination"."termination_type_id", "dcim_cabletermination"."_device_id", "dcim_cabletermination"."_rack_id", "dcim_cabletermination"."_location_id", "dcim_cabletermination"."_site_id", "dcim_cabletermination"."created", "dcim_cabletermination"."last_updated" FROM "dcim_cabletermination" INNER JOIN "dcim_cable" ON ("dcim_cabletermination"."cable_id" = "dcim_cable"."id") WHERE "dcim_cabletermination"."termination_type_id" IN (166) ORDER BY "dcim_cable"."id" ASC, "dcim_cabletermination"."cable_end" ASC, "dcim_cabletermination"."id" ASC
2024-09-06 14:39:23.125 UTC [56] LOG:  statement: SELECT "extras_tag_object_types"."id", "extras_tag_object_types"."tag_id", "extras_tag_object_types"."objecttype_id" FROM "extras_tag_object_types" WHERE "extras_tag_object_types"."objecttype_id" IN (166)
2024-09-06 14:39:23.132 UTC [56] LOG:  statement: SELECT "extras_taggeditem"."id", "extras_taggeditem"."object_id", "extras_taggeditem"."content_type_id", "extras_taggeditem"."tag_id" FROM "extras_taggeditem" WHERE "extras_taggeditem"."content_type_id" IN (166)
2024-09-06 14:39:23.140 UTC [56] LOG:  statement: SELECT "extras_objectchange"."id", "extras_objectchange"."time", "extras_objectchange"."user_name", "extras_objectchange"."request_id", "extras_objectchange"."action", "extras_objectchange"."changed_object_id", "extras_objectchange"."related_object_id", "extras_objectchange"."object_repr", "extras_objectchange"."prechange_data", "extras_objectchange"."postchange_data", "extras_objectchange"."changed_object_type_id", "extras_objectchange"."related_object_type_id", "extras_objectchange"."user_id" FROM "extras_objectchange" WHERE "extras_objectchange"."changed_object_type_id" IN (166) ORDER BY "extras_objectchange"."time" DESC
2024-09-06 14:39:23.140 UTC [56] ERROR:  relation "extras_objectchange" does not exist at character 506
2024-09-06 14:39:23.140 UTC [56] STATEMENT:  SELECT "extras_objectchange"."id", "extras_objectchange"."time", "extras_objectchange"."user_name", "extras_objectchange"."request_id", "extras_objectchange"."action", "extras_objectchange"."changed_object_id", "extras_objectchange"."related_object_id", "extras_objectchange"."object_repr", "extras_objectchange"."prechange_data", "extras_objectchange"."postchange_data", "extras_objectchange"."changed_object_type_id", "extras_objectchange"."related_object_type_id", "extras_objectchange"."user_id" FROM "extras_objectchange" WHERE "extras_objectchange"."changed_object_type_id" IN (166) ORDER BY "extras_objectchange"."time" DESC
2024-09-06 14:39:23.143 UTC [56] LOG:  statement: ROLLBACK

Also, is it normal to first go renaming the table (ALTER TABLE "extras_objectchange" RENAME TO "core_objectchange") and then select from the old name (SELECT "extras_objectchange"."id", "extras_objectchange"."time", "extras_objectchange"."user_name", "extras_objectchange"."request_id", "extras_objectchange"."action", "extras_objectchange"."changed_object_id", "extras_objectchange"."related_object_id", "extras_objectchange"."object_repr", "extras_objectchange"."prechange_data", "extras_objectchange"."postchange_data", "extras_objectchange"."changed_object_type_id", "extras_objectchange"."related_object_type_id", "extras_objectchange"."user_id" FROM "extras_objectchange" WHERE "extras_objectchange"."changed_object_type_id" IN (166) ORDER BY "extras_objectchange"."time" DESC)?

jeremystretch commented 1 week ago

Trying to upgrade from 4.0.11 to 4.1.0 rise migration error.

Bug reports are not to be used for upgrade support. Per the bug report template:

Describe in detail the exact steps that someone else can take to reproduce this bug using the current stable release of NetBox.

We know that simply installing NetBox v4.0.11 and upgrading to v4.1.0 works as expected, so you'll need to identify steps that someone else can take to reproduce this issue if you want it to be investigated as a bug. These steps must not involve the Docker container, as that is out of scope for this project.

Once you've done that, please update the "Steps to Reproduce" section in your post above with the necessary detail.

avbor commented 1 week ago

Bug reports are not to be used for upgrade support

Very strange approach, migration procedures are kind of part of the product, aren't they?

so you'll need to identify steps that someone else can take to reproduce this issue if you want it to be investigated as a bug

I have provided several links to issues and discussions where other users have encountered the same problem.

These steps must not involve the Docker container, as that is out of scope for this project.

As I wrote above, issues in the netbox-docker repository are closed without review and indicate the need to create an issue in that repository. It all seems like a vicious circle. It turns out that users simply don't have the opportunity to create an issue in the "right place".

Steps to Reproduce

I and other users encountering the problem have no other steps to take other than trying to upgrade the product.

image image image

jeremystretch commented 1 week ago

Very strange approach, migration procedures are kind of part of the product, aren't they?

I never suggested otherwise. If you can point to a bug in the migration procedure, we'll happy to address it. But to do that, you first need to provide steps to reproduce the reported behavior, as I asked. Migrations can fail for any number of reasons, such as corrupt or missing data. This does not necessarily indicate a bug in NetBox.

avbor commented 1 week ago

steps to reproduce the reported behavior

I'd love to describe the specific steps, but there is only one step - trying to upgrade an existing netbox installation. Judging by the discussions, different users have tried different ways to reproduce the problem, but the result is the same. For example: I successfully exported the NetBox database version 4.0.7 with PostgreSQL version 16.3 and imported it onto a new fresh Netbox 4.1.0 install. Unfortunately, the following error occurs when running the upgrade script sudo /opt/netbox/upgrade.sh.

I tried the same, made new instance, made fresh install, the dropped netbox DB, created new and wrote a dump from last worked version. It didn't help at all.

Migrations can fail for any number of reasons, such as corrupt or missing data.

That's true, but without help from developers, we're unlikely to find the problem. What is required of us to find the problem? Database dumps, logs, something else?

And I want to pay attention to this point - judging by postgresql logs, during the migration procedure (0117_move_objectchange), first the table is renamed, and then an attempt to select by the old name is made:

2024-09-06 14:39:22.981 UTC [56] LOG:  statement: BEGIN
2024-09-06 14:39:22.981 UTC [56] LOG:  statement: ALTER TABLE "extras_objectchange" RENAME TO "core_objectchange"
...
2024-09-06 14:39:23.140 UTC [56] LOG:  statement: SELECT "extras_objectchange"."id", "extras_objectchange"."time", "extras_objectchange"."user_name", "extras_objectchange"."request_id", "extras_objectchange"."action", "extras_objectchange"."changed_object_id", "extras_objectchange"."related_object_id", "extras_objectchange"."object_repr", "extras_objectchange"."prechange_data", "extras_objectchange"."postchange_data", "extras_objectchange"."changed_object_type_id", "extras_objectchange"."related_object_type_id", "extras_objectchange"."user_id" FROM "extras_objectchange" WHERE "extras_objectchange"."changed_object_type_id" IN (166) ORDER BY "extras_objectchange"."time" DESC
2024-09-06 14:39:23.140 UTC [56] ERROR:  relation "extras_objectchange" does not exist at character 506
2024-09-06 14:39:23.140 UTC [56] STATEMENT:  SELECT "extras_objectchange"."id", "extras_objectchange"."time", "extras_objectchange"."user_name", "extras_objectchange"."request_id", "extras_objectchange"."action", "extras_objectchange"."changed_object_id", "extras_objectchange"."related_object_id", "extras_objectchange"."object_repr", "extras_objectchange"."prechange_data", "extras_objectchange"."postchange_data", "extras_objectchange"."changed_object_type_id", "extras_objectchange"."related_object_type_id", "extras_objectchange"."user_id" FROM "extras_objectchange" WHERE "extras_objectchange"."changed_object_type_id" IN (166) ORDER BY "extras_objectchange"."time" DESC
2024-09-06 14:39:23.143 UTC [56] LOG:  statement: ROLLBACK

This seems a bit odd, would this not help in finding the problem?

kovacs-andras commented 1 week ago

Similar problem here, in-place upgrade didn't work, now I can't revert.

jeremystretch commented 1 week ago

I'd love to describe the specific steps, but there is only one step - trying to upgrade an existing netbox installation.

Ok, let's do that "one step." You said:

Trying to upgrade from 4.0.11 to 4.1.0 rise migration error.

So let's do that:

  1. Check out NetBox v4.0.11
  2. Populate the v4.0 demo data
  3. Verify that NetBox runs as expected
  4. Check out the v4.1.0 release
  5. Run upgrade.sh successfully

All the migrations complete successfully, as expected:

Applying database migrations (python3 netbox/manage.py migrate)...
Operations to perform:
  Apply all migrations: account, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, vpn, wireless
Running migrations:
  Applying core.0011_move_objectchange... OK
  Applying extras.0117_move_objectchange... OK
  Applying extras.0118_customfield_uniqueness... OK
  Applying extras.0119_notifications... OK
  Applying circuits.0044_circuit_groups... OK
  Applying core.0012_job_object_type_optional... OK
  Applying extras.0120_eventrule_event_types... OK
  Applying extras.0121_customfield_related_object_filter... OK
  Applying dcim.0188_racktype... OK
  Applying dcim.0189_moduletype_rack_airflow... OK
  Applying dcim.0190_nested_modules... OK
  Applying ipam.0070_vlangroup_vlan_id_ranges... OK
  Applying virtualization.0039_virtualmachine_serial_number... OK
  Applying virtualization.0040_convert_disk_size... OK
  Applying wireless.0009_wirelesslink_distance... OK

So, what's different about what you're doing?

quantum909 commented 1 week ago

Thank you very much for NetBox and the described steps.

Since I have the same problem importing my NetBox 4.0.7 database (PostgreSQL 16.3) into NetBox 4.1.0 I have tested it with the v4.0 demo data and the same error appears as with my exported 4.0.7 database.

Installation of NetBox 4.1.0 on a newly installed Ubuntu 24.04.1 LTS. NetBox runs as expected

Loading the v4.0 demo data as described here or here into NetBox 4.1.0 then fix the permission problem with GRANT ALL ON SCHEMA public TO netbox.

Now when I run python netbox/manage.py migrate or run upgrade.sh I get the same error message with the v4.0 demo data in NetBox 4.1.0 as with my exported 4.0.7 database.

Operations to perform:
  Apply all migrations: account, auth, circuits, contenttypes, core, dcim, django_rq, extras, ipam, sessions, social_django, taggit, tenancy, users, virtualization, vpn, wireless
Running migrations:
  Applying extras.0117_move_objectchange...Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedTable: relation "extras_objectchange" does not exist
LINE 1: ...ct_type_id", "extras_objectchange"."user_id" FROM "extras_ob...
                                                             ^
candlerb commented 1 week ago

See https://github.com/netbox-community/netbox/discussions/17379#discussioncomment-10576206

From that, it appears to be that migration dependencies haven't been declared properly, so they are applied in a non-deterministic order. Hence it works for some people and not for others.

Making this something that could be reproduced would be difficult: I don't know what affects the order in which they're applied (e.g. the order in which the files were created in the directory?)

It might be a good idea if the migrations were sorted by name, so that any which don't explicitly depend on each other are still applied in a deterministic order, but that could involve changing the internals of the migrations library.

avbor commented 1 week ago

Not sure about this. Migration 0117_move_objectchange.py falls on https://github.com/netbox-community/netbox/blob/16f74f7b03092424554aa85e2043f434feeb807d/netbox/extras/migrations/0117_move_objectchange.py#L8

i.e. all code up to https://github.com/netbox-community/netbox/blob/16f74f7b03092424554aa85e2043f434feeb807d/netbox/extras/migrations/0117_move_objectchange.py#L116 is executed normally. We can see it in postgre log (ALTER TABLE, ALTER INDEX... etc). But, when it comes to the update_content_types function a SELECT from an already deleted table happens (SELECT "extras_objectchange"."id",...).

But, I understand absolutely nothing about django and very little about python =(

avbor commented 1 week ago

Great suggestion about 0011_move_objectchange - https://github.com/netbox-community/netbox/discussions/17388#discussioncomment-10582898 In my case, in addition to deleting this entry, it was also necessary to delete the redis caches - https://github.com/netbox-community/netbox/discussions/17388#discussioncomment-10583097

After that, the netbox successfully updated and started up!

amhn commented 1 week ago

As I said in the comment linked by @avbor the problem is that extras.0117_move_objectchange depends on core.0011_move_objectchange running in the same migration run. This would not be a problem, but core.0011_move_objectchange

If the second migration fails, the user is left with a state that is not recoverable without manual intervention. I do not know if it is possible to prevent the failure on the second run.

Reproduction:

candlerb commented 1 week ago

May or may not be related, but there is a reported dependency issue with core_configrevision in 0062_clear_secrets_changelog.py, see https://github.com/netbox-community/netbox/discussions/17412#discussioncomment-10588147

This is on a database being upgraded from v2.11.11 to v3.7.8 schema.

quantum909 commented 6 days ago

After the problem has been recognized and a manual fix is ready. What would be the best course of action now? Better to wait if it is possible to fix this problem with a future update. Or make the appropriate fixes yourself with the risk that future updates could cause problems again.

jeremystretch commented 6 days ago

What would be the best course of action now?

We still have not identified the source of the failure. As demonstrated above, upgrading from v4.0.11 to v4.1.0 does not reproduce the issue; at least not with Python 3.10 and PostgreSQL 14.

@amhn indicated in another thread that the initial failure was due to missing permissions:

On the first run there was a permission problem on the database, which caused extras.0117_move_objectchange to fail

It's not clear to me why this would be an issue if you're simply upgrading an existing database as suggested in the discussion above, as the necessary permissions should already be in place.

As for backing out core.0011_move_objectchange, you can run the management command manage.py migrate core 0010 to undo the application of the migration. However, it's also not clear why that would be necessary as each migration should be atomic AFAIK.

avbor commented 6 days ago

@amhn indicated in another thread that the initial failure was due to missing permissions:

I think it could be any of the problems in the migration process. For @amhn it was a permissions issue, for me I suspect it was a redis issue. As a result, it turns out that in case of any problem interfering with the migration, we first of all get a non-obvious error ("extras_objectchange" does not exist), and secondly, even after solving this problem, we cannot move on without manual rollback core.0011_move_objectchange (direct in DB or through manage.py migrate core 0010).

amhn commented 6 days ago

I think the problem here is using SeparateDatabaseAndState:

If the actual state of the database and Django’s view of the state get out of sync, this can break the migration framework, even leading to data loss.

https://docs.djangoproject.com/en/5.1/ref/migration-operations/#separatedatabaseandstate

I think after the core.0011_move_objectchange state and database are out of sync. Because this migration created a new Model but no corresponding database change.

I do not know if there is a better way to move a model between apps. This one [https://realpython.com/move-django-model/#the-django-way-rename-the-table] suggests deleting the old object and renaming the table in the first part of the migration.

Since this is triggered by a misconfiguration of the database or in the netbox-docker case of the redis/valkey service, it might not be worth the effort to fix it. Plain upgrades should just work.

@avbor I also hit the redis/valkey issue, although I could not reproduce it without netbox-docker, even after stopping the redis server I was using for the replication of this issue.

avbor commented 6 days ago

I also hit the redis/valkey issue, although I could not reproduce it without netbox-docker, even after stopping the redis server I was using for the replication of this issue.

Maybe the root of the problem here is replacing redis with valkey. But I still see some incorrectness here, no matter what kind of problem happened in the process, but even after solving it, it is impossible to continue the update\migration without manual intervention. And the errors that are generated make the situation even more confusing, at least for me =)

FrankPGH commented 6 days ago

@amhn indicated in another thread that the initial failure was due to missing permissions:

On the first run there was a permission problem on the database, which caused extras.0117_move_objectchange to fail

It's not clear to me why this would be an issue if you're simply upgrading an existing database as suggested in the discussion above, as the necessary permissions should already be in place.

Just want to point out that I experienced that permissions issue with the public schema and never had it before. 4.0.10 was working perfectly for me, the install is bog standard, and no permissions should have changed prior to performing the upgrade.

mikygee commented 6 days ago

Hello,

I don't understand what you did to solve the problem.

I did

delete from django_migrations WHERE name='0011_move_objectchange';

Flushed the redis cache

manage.py migrate core 0010

Nothing worked for me.

rfdrake commented 5 days ago

Here is what I did to get this working again:

# stop all processes
docker-compose down
# bring up postgres
docker-compose up postgres
docker volume ls | grep redis
# remove the redis volumes listed above.
docker volume rm beta_netbox-redis-cache-data beta_netbox-redis-data
# connect to psql client
docker-compose exec postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'
# run the following two commands in the shell
DELETE from django_migrations WHERE name='0011_move_objectchange';
GRANT ALL ON SCHEMA public TO netbox;
# quit the shell
\q
# start all processes in the foreground to watch the migration
docker-compose up

The redis changes are probably only needed if you're using docker. The switch from redis to valkey causes the database to be different. It might be good to check the status of redis connectivity before the migration as a way to defend against this.

Or something can watch for errors like these:

netbox-redis         | 1:M 08 Sep 2024 16:11:22.284 * Reading RDB base file on AOF loading...
netbox-redis         | 1:M 08 Sep 2024 16:11:22.284 # Can't handle RDB format version 12
netbox-redis         | 1:M 08 Sep 2024 16:11:22.284 # Error reading the RDB base file appendonly.aof.65.base.rdb, AOF loading aborted
mikygee commented 4 days ago

Hello Robert, It didn't work for me. Also I don't user docker. I made a database restauration and restarted the upgrarde process up to 4.0.11

KaleBrink commented 4 days ago

I have the same problem. Upgrade from 4.0.11 to 4.1.0 using the steps from the Netbox documentation, Below the error from the upgrade.

Running migrations:
  Applying core.0011_move_objectchange... OK
  Applying extras.0117_move_objectchange... OK
  Applying extras.0118_customfield_uniqueness... OK
  Applying extras.0119_notifications... OK
  Applying circuits.0044_circuit_groups... OK
  Applying core.0012_job_object_type_optional... OK
  Applying extras.0120_eventrule_event_types... OK
  Applying extras.0121_customfield_related_object_filter... OK
  Applying dcim.0188_racktype... OK
  Applying dcim.0189_moduletype_rack_airflow... OK
  Applying dcim.0190_nested_modules... OK
  Applying ipam.0070_vlangroup_vlan_id_ranges... OK
  Applying virtualization.0039_virtualmachine_serial_number... OK
  Applying virtualization.0040_convert_disk_size...Traceback (most recent call last):
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.errors.NumericValueOutOfRange: integer out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox-4.1.0/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox-4.1.0/netbox/virtualization/migrations/0040_convert_disk_size.py", line 7, in convert_disk_size
    VirtualMachine.objects.filter(disk__isnull=False).update(disk=F('disk') * 1000)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1253, in update
    rows = query.get_compiler(self.db).execute_sql(CURSOR)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1990, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
django.db.utils.DataError: integer out of range

And from the postgres log.

2024-09-12 16:34:03.219 UTC [3939] netbox@netbox ERROR:  integer out of range
2024-09-12 16:34:03.219 UTC [3939] netbox@netbox STATEMENT:  UPDATE "virtualization_virtualmachine" SET "disk" = ("virtualization_virtualmachine"."disk" * 1000) WHERE "virtualization_virtualmachine"."disk" IS NOT NULL

Only difference is the Python version used. I have python 3.10

mmahacek commented 4 days ago

@rfdrake Your steps worked perfectly to fix my netbox-docker upgrade from 4.0.11 -> 4.1.0. Thanks!

FrankPGH commented 3 days ago

I was finally able to successfully upgrade from 4.0.11 to 4.1.1 by fixing the broken permissions prior to running upgrade.sh. https://github.com/netbox-community/netbox/discussions/17388#discussioncomment-10638981

kovacs-andras commented 14 hours ago

Here is what I did to get this working again:

# stop all processes
docker-compose down
# bring up postgres
docker-compose up postgres
docker volume ls | grep redis
# remove the redis volumes listed above.
docker volume rm beta_netbox-redis-cache-data beta_netbox-redis-data
# connect to psql client
docker-compose exec postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'
# run the following two commands in the shell
DELETE from django_migrations WHERE name='0011_move_objectchange';
GRANT ALL ON SCHEMA public TO netbox;
# quit the shell
\q
# start all processes in the foreground to watch the migration
docker-compose up

The redis changes are probably only needed if you're using docker. The switch from redis to valkey causes the database to be different. It might be good to check the status of redis connectivity before the migration as a way to defend against this.

Or something can watch for errors like these:

netbox-redis         | 1:M 08 Sep 2024 16:11:22.284 * Reading RDB base file on AOF loading...
netbox-redis         | 1:M 08 Sep 2024 16:11:22.284 # Can't handle RDB format version 12
netbox-redis         | 1:M 08 Sep 2024 16:11:22.284 # Error reading the RDB base file appendonly.aof.65.base.rdb, AOF loading aborted

It worked! Very many thanks! :1st_place_medal: