alexisvl
commented
1 year ago I can confirm that the same build with the same docker config runs fine on a fresh database, however, it also fails to restore from one of the backups from before.
Closed alexisvl closed 1 year ago
alexisvl
commented
1 year ago I can confirm that the same build with the same docker config runs fine on a fresh database, however, it also fails to restore from one of the backups from before.
alexisvl
commented
1 year ago It seems like the failure to start might be connected to #3618. But once I delete those files, it comes up with a lot of errors. I'm curious if maybe it crashed during a database operation and corrupted something.
I moved on to trying to restore from a backup ... here's what happens if I invoke restore with one of the recent backup files, including one that is not the most recent (so any database corruption shouldn't exist in it):
Starting inventree-db ... done
Loading config file : /home/inventree/data/config.yaml
Restoring InvenTree database...
Finding latest backup
Restoring backup for database 'default' and server 'None'
Restoring: default-98386126a188-2023-03-20-165050.psql.bin.gz
Restore tempfile created: 609.6 KiB
Traceback (most recent call last):
File "/home/inventree/InvenTree/manage.py", line 23, in <module>
execute_from_command_line(sys.argv)
File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/root/.local/lib/python3.9/site-packages/dbbackup/management/commands/dbrestore.py", line 68, in handle
self._restore_backup()
File "/root/.local/lib/python3.9/site-packages/dbbackup/management/commands/dbrestore.py", line 118, in _restore_backup
self.connector.restore_dump(input_file)
File "/root/.local/lib/python3.9/site-packages/dbbackup/db/base.py", line 105, in restore_dump
return self._restore_dump(dump)
File "/root/.local/lib/python3.9/site-packages/dbbackup/db/postgresql.py", line 124, in _restore_dump
stdout, stderr = self.run_command(cmd, stdin=dump, env=self.restore_env)
File "/root/.local/lib/python3.9/site-packages/dbbackup/db/base.py", line 171, in run_command
raise exceptions.CommandConnectorError(
dbbackup.db.exceptions.CommandConnectorError: Error running: pg_restore --dbname=postgresql://pguser:pgpassword@inventree-db:5432/inventree --single-transaction --clean
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 3886; 1259 18090 INDEX django_session_session_key_c0390e0f_like pguser
pg_restore: error: could not execute query: ERROR: index "django_session_session_key_c0390e0f_like" does not exist
Command was: DROP INDEX public.django_session_session_key_c0390e0f_like;This has me pretty baffled. These are backups off a working system. They're probably slightly prior to the version I'm running now since this is latest off dockerhub — if this is indeed the problem, is there a way to identify the git hash that created a backup so I can restore into that?
matmair
commented
1 year ago @alexisvl
1) could you check if you can run inv migrate without errors? The database might be corrupt.
2) Do you run stable version or dev?
alexisvl
commented
1 year ago invoke migrate works without issue, other than the usual "fatal: not a git repository" I always get a pile of (attached just in case, but I'm pretty sure they're normal). Bear in mind I am currently trying to restore into a fresh install (though I've also tried on the old one, and have it backed up) and have tried nuking it and starting fresh several times.
This is on the latest tag from dockerhub, so dev.
Starting inventree-db ... done
Loading config file : /home/inventree/data/config.yaml
Running InvenTree database migrations...
========================================
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Migrations for 'part':
part/migrations/0104_alter_partpricing_currency.py
- Alter field currency on partpricing
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Operations to perform:
Apply all migrations: InvenTree, account, admin, auth, authtoken, build, common, company, contenttypes, django_q, error_report, exchange, label, order, otp_static, otp_totp, part, plugin, report, sites, socialaccount, stock, user_sessions, users
Running migrations:
No migrations to apply.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Operations to perform:
Synchronize unmigrated apps: allauth, allauth_2fa, corsheaders, crispy_forms, dbbackup, django_cleanup, django_filters, django_ical, django_otp, djmoney, formtools, import_export, maintenance_mode, markdownify, messages, mptt, rest_framework, staticfiles
Apply all migrations: InvenTree, account, admin, auth, authtoken, build, common, company, contenttypes, django_q, error_report, exchange, label, order, otp_static, otp_totp, part, plugin, report, sites, socialaccount, stock, user_sessions, users
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Running migrations:
No migrations to apply.
System check identified no issues (1 silenced).
========================================
InvenTree database migrations completed!
Rebuilding Part objects
Rebuilding PartCategory objects
Rebuilding StockItem objects
Rebuilding StockLocation objects
Rebuilding Build objects@alexisvl do you still have access to the broken state? I know from the CI that latest is working on a fresh install, I am more interested in how/why the update broke.
alexisvl
commented
1 year ago Oh sorry, yes, I do have the full state from directly after the upgrade. I will try invoke migrate on that with the current version of the docker image it's running. I don't immediately have access to the old state with the old inventree version since that appears to no longer be accessible on dockerhub, so I can try building that locally from an old commit if needed
alexisvl
commented
1 year ago Running it on a direct copy of the old state brings me back to ValueError: state value is not 0|1. If I run find . -name 'maintenance_mode_state*.txt -delete again as before, I get some pretty unhappy messages:
Starting inventree-db ... done
Loading config file : /home/inventree/data/config.yaml
Running InvenTree database migrations...
========================================
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
/root/.local/lib/python3.9/site-packages/django/core/management/commands/makemigrations.py:105: RuntimeWarning: Got an error checking a consistent migration history performed for database connection 'default': FATAL: database "inventree" does not exist
warnings.warn(
Migrations for 'part':
part/migrations/0104_alter_partpricing_currency.py
- Alter field currency on partpricing
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
File "/root/.local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
self.connect()
File "/root/.local/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "/root/.local/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: database "inventree" does not exist
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/inventree/InvenTree/manage.py", line 23, in <module>
execute_from_command_line(sys.argv)
File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 75, in handle
self.check(databases=[database])
File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 419, in check
all_issues = checks.run_checks(
File "/root/.local/lib/python3.9/site-packages/django/core/checks/registry.py", line 76, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
File "/root/.local/lib/python3.9/site-packages/django/core/checks/model_checks.py", line 34, in check_all_models
errors.extend(model.check(**kwargs))
File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 1303, in check
*cls._check_indexes(databases),
File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 1695, in _check_indexes
connection.features.supports_covering_indexes or
File "/root/.local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/postgresql/features.py", line 92, in is_postgresql_11
return self.connection.pg_version >= 110000
File "/root/.local/lib/python3.9/site-packages/django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 329, in pg_version
with self.temporary_connection():
File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 603, in temporary_connection
with self.cursor() as cursor:
File "/root/.local/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
return self._cursor()
File "/root/.local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
self.ensure_connection()
File "/root/.local/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
self.connect()
File "/root/.local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/root/.local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
self.connect()
File "/root/.local/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "/root/.local/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
return func(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL: database "inventree" does not existSo, probable corruption I'd say. But I'm much more concerned at this point with why backups from a few days before this point fail to restore...
alexisvl
commented
1 year ago I don't immediately have access to the old state with the old inventree version since that appears to no longer be accessible on dockerhub, so I can try building that locally from an old commit if needed
Just tried doing precisely this and the backups still don't restore:
dbbackup.db.exceptions.CommandConnectorError: Error running: pg_restore --dbname=postgresql://pguser:pgpassword@inventree-dev-db:5432/inventree --single-transaction --clean
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 3886; 1259 18090 INDEX django_session_session_key_c0390e0f_like pguser
pg_restore: error: could not execute query: ERROR: index "django_session_session_key_c0390e0f_like" does not exist
Command was: DROP INDEX public.django_session_session_key_c0390e0f_like;Am I just doing it wrong? My process is to nuke everything in the data volume, set up a new install with invoke update, drop the backups into backup in the data volume (making sure the one I want is the most recent), then invoke restore. The invoke update step seems to be necessary to get it to even try to do anything but maybe I'm just missing something?
alexisvl
commented
1 year ago Alright, ultimately I was able to get something up. I had to restore from a json backup instead (invoke import-records) and then copy the contents of the media directory over manually. It looks like my issue with restore from json before was just not knowing about the -c option to import-records.
I'm still really curious why the normal database backups don't restore. It'd be nice to resolve that in case I need to do this again in the future... I'm still not 100% sure if everything gets restored from json, but I haven't found anything missing yet.
I feel like I must be using all these things completely wrong.
alexisvl
commented
1 year ago I just tried running invoke backup followed by invoke restore just to make sure the restore mechanism worked at all, and it does. Which I guess would point to the backups themselves being damaged? But I'm very skeptical of that - they are from a time when the system was working fine, and the json dump is from the same time, and it imports ok.
alexisvl
commented
1 year ago At this point I'm just going to assume that there was a temporary issue with InvenTree (either my install specifically, or the thing as a whole) producing bad backups and I got stuck trying to restore them. It's probably not in anyone's best interest to keep poking this since I was able to resurrect a working system eventually and the backups it produces now seem to restore fine. I'm going to close this out, thanks for the help.
matmair
commented
1 year ago This is really a head-scratcher. Not sure what would corrupt the backups but I hope this does not resurface. An effective hatch against this could be regular exports as yaml - those are pretty hard to be corrupted and DBMS agnostic.
alexisvl
commented
1 year ago Is there a yaml-based mechanism I wasn't aware of? All I know is the json one, is there a difference between them or just different data formats? Curious if that's something I should be doing. I do have a cronjob doing nightly json exports already (thank god, as that's what saved my ass here)
matmair
commented
1 year ago A sorry, I thought of another project. InvenTree has the export-records function putting out json, the rest of the sentence is correct.
matmair
commented
1 year ago I have been thinking of moving to the json exports for records for a while but it adds quite a bit of compute and storage overhead compared to the dumps of the current backup solution.
alexisvl
commented
1 year ago Well, thank you very much for helping me troubleshoot my silly mystery issue :)
Deployment Method
Steps to ReproduceDescribe the problem*
After a recent update to a docker devel install (
docker-compose pull,docker-compose run inventree-server invoke update) InvenTree fails to start at all. I've got a pretty long stream of errors coming out of things here and not really sure what to make of them.Steps to Reproduce
docker-compose pull,docker-compose up -d,docker-compose run inventree-server invoke updatedocker-compose upand watch the log.Relevant log output