LibrePhotos / librephotos

A self-hosted open source photo management service. This is the repository of the backend.
MIT License
6.86k stars 299 forks source link

DB errors when installing #257

Closed paulmorabito closed 3 years ago

paulmorabito commented 3 years ago

Running on docker and previously had everything working but I didn't configure map box. I couldn't find a way to re-scan current photos so I dropped the database and deleted the config and other files. When re-installing, I started getting errors such as this:

auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
 [ ] 0010_alter_group_name_max_length
 [ ] 0011_update_proxy_permissions
 [ ] 0012_alter_user_first_name_max_length
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
database
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/code/api/management/commands/createadmin.py", line 40, in handle
    if not User.objects.filter(username=options['admin_username']).exists():
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 809, in exists
    return self.query.has_results(using=self.db)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 535, in has_results
    return compiler.has_results()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1126, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "api_user" does not exist
LINE 1: SELECT (1) AS "a" FROM "api_user" WHERE "api_user"."username...

In postgres, the logs are:

2021-05-28 16:11:25.217 BST [19275] ERROR:  relation "api_face" does not exist at character 336
2021-05-28 16:11:25.217 BST [19275] STATEMENT:  SELECT "api_face"."id", "api_face"."photo_id", "api_face"."image", "api_face"."image_path", "api_face"."person_id", "api_face"."person_label_is_inferred", "api_face"."person_label_probability", "api_face"."location_top", "api_face"."location_bottom", "api_face"."location_left", "api_face"."location_right", "api_face"."encoding" FROM "api_face" INNER JOIN "api_photo" ON ("api_face"."photo_id" = "api_photo"."image_hash") WHERE NOT "api_photo"."hidden" ORDER BY "api_face"."id" ASC
2021-05-28 16:11:25.220 BST [19275] ERROR:  relation "api_albumdate" does not exist at character 168
2021-05-28 16:11:25.220 BST [19275] STATEMENT:  SELECT "api_albumdate"."id", "api_albumdate"."title", "api_albumdate"."date", "api_albumdate"."favorited", "api_albumdate"."location", "api_albumdate"."owner_id" FROM "api_albumdate" ORDER BY "api_albumdate"."date" DESC
2021-05-28 16:11:25.223 BST [19275] ERROR:  relation "api_albumdate" does not exist at character 276
2021-05-28 16:11:25.223 BST [19275] STATEMENT:  SELECT "api_albumdate"."id", "api_albumdate"."title", "api_albumdate"."date", "api_albumdate"."favorited", "api_albumdate"."location", "api_albumdate"."owner_id", COUNT(DISTINCT "api_albumdate_photos"."photo_id") FILTER (WHERE NOT "api_photo"."hidden") AS "photo_count" FROM "api_albumdate" LEFT OUTER JOIN "api_albumdate_photos" ON ("api_albumdate"."id" = "api_albumdate_photos"."albumdate_id") LEFT OUTER JOIN "api_photo" ON ("api_albumdate_photos"."photo_id" = "api_photo"."image_hash") GROUP BY "api_albumdate"."id" HAVING COUNT(DISTINCT "api_albumdate_photos"."photo_id") FILTER (WHERE (NOT "api_photo"."hidden")) > 0 ORDER BY "api_albumdate"."date" DESC
2021-05-28 16:12:01.107 BST [19275] ERROR:  could not open file "base/41456/41981": Invalid argument
2021-05-28 16:12:01.107 BST [19275] CONTEXT:  SQL statement "SELECT fk."albumplace_id" FROM ONLY "public"."api_albumplace_photos" fk LEFT OUTER JOIN ONLY "public"."api_albumplace" pk ON ( pk."id" OPERATOR(pg_catalog.=) fk."albumplace_id") WHERE pk."id" IS NULL AND (fk."albumplace_id" IS NOT NULL)"

Above is just a snippet, there is several like this. I've definitely dropped the database and there are no other issues with the postgres server or anything that has changed in the past 24 hours that I can see. Any ideas on what this could be?

Thanks!

derneuere commented 3 years ago

The problem seems to be in the auth migration files which are supplied by django and not by me. Do you run any other django projects on the same db? I starts up fine when I use a new db.

paulmorabito commented 3 years ago

No, no other Django auth migration files. Other than dropping the DB and deleting the config, is there anything else I should clear?

derneuere commented 3 years ago

https://github.com/hooram/ownphotos/issues/99

He could resolve the error by stopping and starting the container. Does this work for you too?

paulmorabito commented 3 years ago

No, getting the same error despite several restarts.

derneuere commented 3 years ago

Can you send me your ownphotos.log? I feel like something is missing.

paulmorabito commented 3 years ago

it's empty actually.

-rw-r--r-- 1 root root  243 May 29 01:39 rqworker.log
-rw-r--r-- 1 root root  353 May 29 01:39 gunicorn_django.log
-rw-r--r-- 1 root root 2817 May 29 01:39 command_createadmin.log
-rw-r--r-- 1 root root 2885 May 29 01:39 command_build_similarity_index.log
-rw-r--r-- 1 root root  947 May 29 01:39 show_migrate.log
-rw-r--r-- 1 root root  148 May 29 01:38 command_migrate.log
-rw-r--r-- 1 root root  375 May 29 01:38 image_similarity.log
-rw-r--r-- 1 root root    0 May 29 01:38 gunicorn_image_similarity.log
-rw-r--r-- 1 root root    0 May 29 01:12 ownphotos.log
logan71f100 commented 3 years ago

im also affected by this, empty logs on fresh install, i cant even sign in for first time setup tho

derneuere commented 3 years ago

Alright, which CPU models are you both using?

derneuere commented 3 years ago

If you use an old cpu please build the image yourself: https://docs.librephotos.com/1/old_processors/

paulmorabito commented 3 years ago

I'm running a Ryzen 3600XT. Definitely not an old CPU. The docker is running in a Debian VM hosted on Unraid but the CPU is passed through:

cat /proc/cpuinfo
processor   : 0
vendor_id   : AuthenticAMD
cpu family  : 23
model       : 113
model name  : AMD Ryzen 5 3600XT 6-Core Processor
stepping    : 0
microcode   : 0x8701021
cpu MHz     : 3792.644
cache size  : 512 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 16
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
bugs        : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass
bogomips    : 7585.28
TLB size    : 3072 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : AuthenticAMD
cpu family  : 23
model       : 113
model name  : AMD Ryzen 5 3600XT 6-Core Processor
stepping    : 0
microcode   : 0x8701021
cpu MHz     : 3792.644
cache size  : 512 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 2
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 16
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
bugs        : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass
bogomips    : 7688.98
TLB size    : 3072 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : AuthenticAMD
cpu family  : 23
model       : 113
model name  : AMD Ryzen 5 3600XT 6-Core Processor
stepping    : 0
microcode   : 0x8701021
cpu MHz     : 3792.644
cache size  : 512 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 2
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 16
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
bugs        : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass
bogomips    : 7681.02
TLB size    : 3072 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : AuthenticAMD
cpu family  : 23
model       : 113
model name  : AMD Ryzen 5 3600XT 6-Core Processor
stepping    : 0
microcode   : 0x8701021
cpu MHz     : 3792.644
cache size  : 512 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 2
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 16
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
bugs        : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass
bogomips    : 7628.40
TLB size    : 3072 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:
jakev383 commented 3 years ago

I'm seeing some of the same/similar DB errors. Fresh install for me - I had to add a couple of items to the environment vars for postgres to come up (using https://raw.githubusercontent.com/LibrePhotos/librephotos-docker/main/docker-compose.yml): POSTGRES_USER POSTGRES_PASSWORD POSTGRES_DB

Once I added those, postgres could start and it began the DB population/install.

I created a new user, connected to my NextCloud account, then tried to import a folder of pictures.

I'm seeing similar errors:

16:25:31.404 UTC [35] ERROR:  relation "api_face" does not exist at character 336

2021-05-30 16:25:31.404 UTC [35] STATEMENT:  SELECT "api_face"."id", "api_face"."photo_id", "api_face"."image", "api_face"."image_path", "api_face"."person_id", "api_face"."person_label_is_inferred", "api_face"."person_label_probability", "api_face"."location_top", "api_face"."location_bottom", "api_face"."location_left", "api_face"."location_right", "api_face"."encoding" FROM "api_face" INNER JOIN "api_photo" ON ("api_face"."photo_id" = "api_photo"."image_hash") WHERE NOT "api_photo"."hidden" ORDER BY "api_face"."id" ASC

2021-05-30 16:25:31.407 UTC [35] ERROR:  relation "api_albumdate" does not exist at character 168

2021-05-30 16:25:31.407 UTC [35] STATEMENT:  SELECT "api_albumdate"."id", "api_albumdate"."title", "api_albumdate"."date", "api_albumdate"."favorited", "api_albumdate"."location", "api_albumdate"."owner_id" FROM "api_albumdate" ORDER BY "api_albumdate"."date" DESC

2021-05-30 16:25:31.411 UTC [35] ERROR:  relation "api_albumdate" does not exist at character 276

2021-05-30 16:25:31.411 UTC [35] STATEMENT:  SELECT "api_albumdate"."id", "api_albumdate"."title", "api_albumdate"."date", "api_albumdate"."favorited", "api_albumdate"."location", "api_albumdate"."owner_id", COUNT(DISTINCT "api_albumdate_photos"."photo_id") FILTER (WHERE NOT "api_photo"."hidden") AS "photo_count" FROM "api_albumdate" LEFT OUTER JOIN "api_albumdate_photos" ON ("api_albumdate"."id" = "api_albumdate_photos"."albumdate_id") LEFT OUTER JOIN "api_photo" ON ("api_albumdate_photos"."photo_id" = "api_photo"."image_hash") GROUP BY "api_albumdate"."id" HAVING COUNT(DISTINCT "api_albumdate_photos"."photo_id") FILTER (WHERE (NOT "api_photo"."hidden")) > 0 ORDER BY "api_albumdate"."date" DESC

2021-05-30 16:25:37.565 UTC [36] ERROR:  relation "api_face" does not exist at character 336

2021-05-30 16:25:37.565 UTC [36] STATEMENT:  SELECT "api_face"."id", "api_face"."photo_id", "api_face"."image", "api_face"."image_path", "api_face"."person_id", "api_face"."person_label_is_inferred", "api_face"."person_label_probability", "api_face"."location_top", "api_face"."location_bottom", "api_face"."location_left", "api_face"."location_right", "api_face"."encoding" FROM "api_face" INNER JOIN "api_photo" ON ("api_face"."photo_id" = "api_photo"."image_hash") WHERE NOT "api_photo"."hidden" ORDER BY "api_face"."id" ASC

2021-05-30 16:25:37.568 UTC [36] ERROR:  relation "api_albumdate" does not exist at character 168

2021-05-30 16:25:37.568 UTC [36] STATEMENT:  SELECT "api_albumdate"."id", "api_albumdate"."title", "api_albumdate"."date", "api_albumdate"."favorited", "api_albumdate"."location", "api_albumdate"."owner_id" FROM "api_albumdate" ORDER BY "api_albumdate"."date" DESC

2021-05-30 16:25:37.571 UTC [36] ERROR:  relation "api_albumdate" does not exist at character 276

From ownphotos.log:

2021-05-30 21:02:42,339 : directory_watcher.py : scan_photos : 67 : INFO : Downloaded photo from nextcloud to /data/nextcloud_media/jake/Pictures/nose-burn.jpg
2021-05-30 21:02:42,339 : directory_watcher.py : scan_photos : 67 : INFO : Downloaded photo from nextcloud to /data/nextcloud_media/jake/Pictures/river.gif
2021-05-30 21:02:42,339 : directory_watcher.py : scan_photos : 67 : INFO : Downloaded photo from nextcloud to /data/nextcloud_media/jake/Pictures/sam_dino1.jpg
2021-05-30 21:02:42,343 : directory_watcher.py : scan_photos : 105 : ERROR : Cannot resolve keyword 'image_path' into field. Choices are: added_on, albumauto, albumdate, albumplace, albumthing, albumuser, captions_json, encoding, exif_gps_lat, exif_gps_lon, exif_json, exif_timestamp, faces, favorited, geolocation_json, hidden, image, image_hash, image_paths, owner, owner_id, public, search_captions, search_location, shared_to, square_thumbnail, square_thumbnail_small, thumbnail_big, video
Traceback (most recent call last):
  File "/code/nextcloud/directory_watcher.py", line 76, in scan_photos
    if not Photo.objects.filter(image_path=image_path).exists():
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 942, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 962, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 969, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1358, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1377, in _add_q
    child_clause, needed_inner = self.build_filter(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1258, in build_filter
    lookups, parts, reffed_expression = self.solve_lookup_type(arg)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1084, in solve_lookup_type
    _, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 1481, in names_to_path
    raise FieldError("Cannot resolve keyword '%s' into field. "
django.core.exceptions.FieldError: Cannot resolve keyword 'image_path' into field. Choices are: added_on, albumauto, albumdate, albumplace, albumthing, albumuser, captions_json, encoding, exif_gps_lat, exif_gps_lon, exif_json, exif_timestamp, faces, favorited, geolocation_json, hidden, image, image_hash, image_paths, owner, owner_id, public, search_captions, search_location, shared_to, square_thumbnail, square_thumbnail_small, thumbnail_big, video
2021-05-30 21:02:42,347 : views.py : get : 44 : ERROR : An Error occured
Traceback (most recent call last):
  File "/code/nextcloud/views.py", line 41, in get
    scan_photos(request.user, job_id)
  File "/code/nextcloud/directory_watcher.py", line 115, in scan_photos
    return {"new_photo_count": added_photo_count, "status": True}
UnboundLocalError: local variable 'added_photo_count' referenced before assignment
derneuere commented 3 years ago

@jakev383 This seems to be a different error. I will look into it.

alyssonee1010 commented 6 months ago

The problem seems to be in the auth migration files which are supplied by django and not by me. Do you run any other django projects on the same db? I starts up fine when I use a new db.

that solve to me, my case was I was running postgres sql database on google cloud, as I'd connected another django db on the same DB i was using (I was testing), it stated to give me this error. I deleted the entire database and created a new one, same name, did the migrations and everything worked just fine