search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
52.03k stars 2.76k forks source link

[BUG] Android app only showing some local photos and limited remote assets #2063

Closed kevinanderson1 closed 1 year ago

kevinanderson1 commented 1 year ago

The bug

I recently upgraded Immich from a previous version (I unfortunately do not recall the old version) and the Android application only shows local assets on my phone and a small number of assets that I had in a shared album in Immich.

If I access Immich through the Web URL, I see all assets (around 17K in my setup) as expected.

I am unsure of where to get more logs from the Android application side, but looking in the Immich logs shows the error below:

Failed to upsert 17582 assets into the DB: IsarError: Unique index violated

I have tried clearing the application cache and logging out and back in unsuccessfully. I also don't see any obvious errors in the container logs.

The OS that Immich Server is running on

Fedora 37

Version of Immich Server

1.51.2

Version of Immich Mobile App

1.51.0 build.74

Platform with the issue

Your docker-compose.yml content

I am running Immich in Podman containers using systemd to manage the stack. I use the Podman Quadlet generator to generate the systemd unit files which I included below.

Your .env content

# /run/systemd/generator/immich-db.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
# 
[Unit]
Description=Immich Database
After=local-fs.target
SourcePath=/etc/containers/systemd/immich-db.container
RequiresMountsFor=%t/containers
Requires=immich-net-network.service
After=immich-net-network.service
RequiresMountsFor=/srv/immich/db

[X-Container]
Image=docker.io/postgres:14
ContainerName=immich-db
Network=immich-net.network
Volume=/srv/immich/db:/var/lib/postgresql/data:Z
Environment=POSTGRES_PASSWORD=<redacted>
Environment=POSTGRES_USER=postgres
Environment=POSTGRES_DB=immich

[Service]
Restart=on-failure
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-rm -f %t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=immich-db --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --network=systemd-immich-net --sdnotify=conmon -d -v /srv/immich/db:/var/lib/postgresql/data:Z --env POSTGRES_DB=immich --env POSTGRES_PASSWORD=<redacted> --env POSTGRES_USER=postgres docker.io/postgres:14

[Install]
WantedBy=multi-user.target

# /run/systemd/generator/immich-machine-learning.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
# 
[Unit]
Description=Immich Machine Learning
After=local-fs.target
Wants=immich-typesense.server
SourcePath=/etc/containers/systemd/immich-machine-learning.container
RequiresMountsFor=%t/containers
Requires=immich-net-network.service
After=immich-net-network.service
RequiresMountsFor=/srv/immich/uploads
RequiresMountsFor=/srv/immich/model-cache

[X-Container]
Image=ghcr.io/immich-app/immich-machine-learning:release
ContainerName=immich-machine-learning
Network=immich-net.network
Environment=NODE_ENV=production
Environment=TYPESENSE_API_KEY=<redacted>
Volume=/srv/immich/uploads:/usr/src/app/upload:z
Volume=/srv/immich/model-cache:/cache:z

[Service]
Restart=on-failure
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-rm -f %t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=immich-machine-learning --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --network=systemd-immich-net --sdnotify=conmon -d -v /srv/immich/uploads:/usr/src/app/upload:z -v /srv/immich/model-cache:/cache:z --env NODE_ENV=production --env TYPESENSE_API_KEY=<redacted> ghcr.io/immich-app/immich-machine-learning:release

[Install]
WantedBy=multi-user.target

# /run/systemd/generator/immich-microservices.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
# 
[Unit]
Description=Immich Microservices
After=local-fs.target
Wants=immich-db.service
Wants=immich-redis.service
SourcePath=/etc/containers/systemd/immich-microservices.container
RequiresMountsFor=%t/containers
Requires=immich-net-network.service
After=immich-net-network.service
RequiresMountsFor=/srv/immich/uploads

[X-Container]
Image=ghcr.io/immich-app/immich-server:release
ContainerName=immich-microservices
Network=immich-net.network
Exec="/bin/sh" "./start-microservices.sh"
Environment=NODE_ENV=production
Environment=REDIS_HOSTNAME=immich-redis
Environment=DB_PASSWORD=<redacted>
Environment=DB_USERNAME=postgres
Environment=DB_HOSTNAME=immich-db
Environment=DB_DATABASE_NAME=immich
Environment=TYPESENSE_API_KEY=<redacted>
Volume=/srv/immich/uploads:/usr/src/app/upload:z

[Service]
Restart=on-failure
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-rm -f %t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=immich-microservices --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --network=systemd-immich-net --sdnotify=conmon -d -v /srv/immich/uploads:/usr/src/app/upload:z --env DB_DATABASE_NAME=immich --env DB_HOSTNAME=immich-db --env DB_PASSWORD=<redacted> --env DB_USERNAME=postgres --env NODE_ENV=production --env REDIS_HOSTNAME=immich-redis --env TYPESENSE_API_KEY=<redacted> ghcr.io/immich-app/immich-server:release /bin/sh ./start-microservices.sh

[Install]
WantedBy=multi-user.target

# /run/systemd/generator/immich-net-network.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
# 
[X-Network]

[Unit]
RequiresMountsFor=%t/containers

[Service]
ExecStart=/usr/bin/podman network create --ignore systemd-immich-net
Type=oneshot
RemainAfterExit=yes
SyslogIdentifier=%N

# /run/systemd/generator/immich-redis.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
# 
[Unit]
Description=Immich Redis
After=local-fs.target
SourcePath=/etc/containers/systemd/immich-redis.container
RequiresMountsFor=%t/containers
Requires=immich-net-network.service
After=immich-net-network.service

[X-Container]
Image=docker.io/redis:6.2
ContainerName=immich-redis
Network=immich-net.network

[Service]
Restart=on-failure
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-rm -f %t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=immich-redis --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --network=systemd-immich-net --sdnotify=conmon -d docker.io/redis:6.2

[Install]
WantedBy=multi-user.target

# /run/systemd/generator/immich-server.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
# 
[Unit]
Description=Immich Server
After=local-fs.target
Wants=immich-db.service
Wants=immich-redis.service
Wants=immich-typesense.service
SourcePath=/etc/containers/systemd/immich-server.container
RequiresMountsFor=%t/containers
Requires=immich-net-network.service
After=immich-net-network.service
RequiresMountsFor=/srv/immich/uploads

[X-Container]
Image=ghcr.io/immich-app/immich-server:release
ContainerName=immich-server
Network=immich-net.network
Exec="/bin/sh" "./start-server.sh"
Environment=NODE_ENV=production
Environment=REDIS_HOSTNAME=immich-redis
Environment=DB_PASSWORD=<redacted>
Environment=DB_USERNAME=postgres
Environment=DB_HOSTNAME=immich-db
Environment=DB_DATABASE_NAME=immich
Environment=TYPESENSE_API_KEY=<redacted>
Environment=LOG_LEVEL=verbose
Volume=/srv/immich/uploads:/usr/src/app/upload:z
Label="traefik.http.middlewares.immich.stripprefix.prefixes=/api"
Label="traefik.http.routers.immich.rule=Host(`immich.lan.<redacted>.dev`) && PathPrefix(`/api`)"
Label="traefik.http.routers.immich.tls.certresolver=letsencrypt"
Label="traefik.http.routers.immich.entrypoints=https"
Label="traefik.http.routers.immich.middlewares=lan@file,immich"
Label="traefik.enable=true"

[Service]
Restart=on-failure
TimeoutSec=15
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-rm -f %t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=immich-server --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --network=systemd-immich-net --sdnotify=conmon -d -v /srv/immich/uploads:/usr/src/app/upload:z --env DB_DATABASE_NAME=immich --env DB_HOSTNAME=immich-db --env DB_PASSWORD=<redacted> --env DB_USERNAME=postgres --env LOG_LEVEL=verbose --env NODE_ENV=production --env REDIS_HOSTNAME=immich-redis --env TYPESENSE_API_KEY=<redacted> --label traefik.enable=true --label traefik.http.middlewares.immich.stripprefix.prefixes=/api --label traefik.http.routers.immich.entrypoints=https --label traefik.http.routers.immich.middlewares=lan@file,immich --label "traefik.http.routers.immich.rule=Host(`immich.lan.<redacted>.dev`) && PathPrefix(`/api`)" --label traefik.http.routers.immich.tls.certresolver=letsencrypt ghcr.io/immich-app/immich-server:release /bin/sh ./start-server.sh

[Install]
WantedBy=multi-user.target

# /run/systemd/generator/immich-typesense.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
# 
[Unit]
Description=Immich Typesense
After=local-fs.target
SourcePath=/etc/containers/systemd/immich-typesense.container
RequiresMountsFor=%t/containers
Requires=immich-net-network.service
After=immich-net-network.service
RequiresMountsFor=/srv/immich/typesense

[X-Container]
Image=docker.io/typesense/typesense:0.24.0
ContainerName=typesense
Network=immich-net.network
Environment=NODE_ENV=production
Environment=TYPESENSE_API_KEY=<redacted>
Environment=TYPESENSE_DATA_DIR=/data
Volume=/srv/immich/typesense:/data:z

[Service]
Restart=on-failure
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-rm -f %t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=typesense --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --network=systemd-immich-net --sdnotify=conmon -d -v /srv/immich/typesense:/data:z --env NODE_ENV=production --env TYPESENSE_API_KEY=<redacted> --env TYPESENSE_DATA_DIR=/data docker.io/typesense/typesense:0.24.0

[Install]
WantedBy=multi-user.target

# /run/systemd/generator/immich-web.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
# 
[Unit]
Description=Immich Web
After=local-fs.target
SourcePath=/etc/containers/systemd/immich-web.container
RequiresMountsFor=%t/containers
Requires=immich-net-network.service
After=immich-net-network.service

[X-Container]
Image=ghcr.io/immich-app/immich-web:release
ContainerName=immich-web
Network=immich-net.network
Exec="/bin/sh" "./entrypoint.sh"
Label="traefik.http.routers.immich-web.rule=Host(`immich.lan.<redacted>.dev`)"
Label="traefik.http.routers.immich-web.tls.certresolver=letsencrypt"
Label="traefik.http.routers.immich-web.entrypoints=https"
Label="traefik.http.routers.immich-web.middlewares=lan@file"
Label="traefik.enable=true"

[Service]
Restart=on-failure
TimeoutSec=15
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=%t/%N.cid
ExecStopPost=-rm -f %t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run --name=immich-web --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --network=systemd-immich-net --sdnotify=conmon -d --label traefik.enable=true --label traefik.http.routers.immich-web.entrypoints=https --label traefik.http.routers.immich-web.middlewares=lan@file --label traefik.http.routers.immich-web.rule=Host(`immich.lan.<redacted>.dev`) --label traefik.http.routers.immich-web.tls.certresolver=letsencrypt ghcr.io/immich-app/immich-web:release /bin/sh ./entrypoint.sh

[Install]
WantedBy=multi-user.target

### Reproduction steps

```bash
1. Import assets in to Immich
2. Add a handful of assets to a shared album
3. Open the Android application and see that only local assets and the shared album are showed instead of all images on the timeline

Additional information

No response

alextran1502 commented 1 year ago

@fyfrey we have another sync issue I think.

@kevinanderson1 can you help uninstall and reinstall the all to see if it helps with the issue?

alextran1502 commented 1 year ago

@kevinanderson1 can you help exporting the log on the mobile app and share with us here?

kevinanderson1 commented 1 year ago

@alextran1502 I logged out, uninstalled, and reinstalled the Immich Android application with the same error. I've pasted the log from the latest run below.

created_at  level   context message stacktrace
2023-03-23 19:27:45 INFO    AssetNotifier   setting new asset state 
2023-03-23 19:27:45 INFO    AssetNotifier   Load assets: 5300ms 
2023-03-23 19:27:45 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-23 19:27:45 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-23 19:27:45 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-23 19:27:45 INFO    AlbumService    Found 2 device albums   
2023-03-23 19:27:45 WARNING SyncService Failed to upsert 17582 assets into the DB: IsarError: Unique index violated.    
2023-03-23 19:27:42 INFO    SyncService Upserted 9 assets into the DB   
2023-03-23 19:27:42 INFO    SyncService Upserted 1 assets into the DB   
2023-03-23 19:27:40 INFO    AlbumService    Syncing completed. Changes: true    
2023-03-23 19:27:40 INFO    SyncService Added a new local album to DB: AssetPathEntity(id: 76699306, name: DogHealt)    
2023-03-23 19:27:40 INFO    SyncService Upserted 1 assets into the DB   
2023-03-23 19:27:40 INFO    SyncService 0 assets already existed in DB, to upsert 1 
2023-03-23 19:27:40 INFO    SyncService Syncing a new local album to DB: AssetPathEntity(id: 76699306, name: DogHealt)  
2023-03-23 19:27:40 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-23 19:27:40 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-23 19:27:40 INFO    AlbumService    Found 2 device albums   
2023-03-22 18:14:40 INFO    AssetNotifier   Load assets: 10096ms    
2023-03-22 18:14:40 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-22 18:14:40 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-22 18:14:40 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-22 18:14:40 INFO    AlbumService    Found 2 device albums   
2023-03-22 18:14:40 WARNING SyncService Failed to upsert 17582 assets into the DB: IsarError: Unique index violated.    
2023-03-22 18:10:08 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-22 18:10:08 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-22 18:10:08 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-22 18:10:08 INFO    AlbumService    Found 2 device albums   
2023-03-22 17:59:31 INFO    AssetNotifier   Load assets: 5851ms 
2023-03-22 17:59:31 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-22 17:59:31 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-22 17:59:31 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-22 17:59:31 INFO    AlbumService    Found 2 device albums   
2023-03-22 17:59:31 WARNING SyncService Failed to upsert 17582 assets into the DB: IsarError: Unique index violated.    
2023-03-22 17:59:18 INFO    BackupNotifier  Not found albums or assets on the device to backup  
2023-03-22 17:59:18 INFO    BackupNotifier  Found 2 local albums    
2023-03-22 17:59:14 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-22 17:59:14 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-22 17:59:14 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-22 17:59:14 INFO    AlbumService    Found 2 device albums   
2023-03-22 17:59:10 INFO    AssetNotifier   Load assets: 6564ms 
2023-03-22 17:59:10 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-22 17:59:10 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-22 17:59:10 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-22 17:59:10 INFO    AlbumService    Found 2 device albums   
2023-03-22 17:59:10 WARNING SyncService Failed to upsert 17582 assets into the DB: IsarError: Unique index violated.    
2023-03-22 17:59:03 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-22 17:59:03 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-22 17:59:03 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-22 17:59:03 INFO    AlbumService    Found 2 device albums   
2023-03-22 17:59:03 INFO    AssetNotifier   Reading assets 11 from DB: 6ms  
2023-03-21 21:34:52 INFO    AssetNotifier   Load assets: 5385ms 
2023-03-21 21:34:52 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-21 21:34:52 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-21 21:34:52 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-21 21:34:52 INFO    AlbumService    Found 2 device albums   
2023-03-21 21:34:52 WARNING SyncService Failed to upsert 17582 assets into the DB: IsarError: Unique index violated.    
2023-03-21 21:34:47 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-21 21:34:47 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-21 21:34:47 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-21 21:34:47 INFO    AlbumService    Found 2 device albums   
2023-03-21 21:34:47 INFO    AssetNotifier   Reading assets 11 from DB: 8ms  
2023-03-21 21:34:25 INFO    SyncService Upserted 1 assets into the DB   
2023-03-21 21:34:24 INFO    AlbumService    Syncing completed. Changes: false   
2023-03-21 21:34:24 INFO    SyncService Syncing 1 albums from device: [AssetPathEntity(id: 76699306, name: DogHealt)]   
2023-03-21 21:34:24 INFO    AlbumService    Recents' is selected, keeping all individual albums 
2023-03-21 21:34:24 INFO    AlbumService    Found 2 device albums   
2023-03-21 21:34:17 INFO    BackupNotifier  Found 2 local albums
alextran1502 commented 1 year ago

If you can use the share function from the log page it would be helpful since it would contain the stack trace as well

kevinanderson1 commented 1 year ago

@alextran1502 I shared those logs to my email and attached them here. Unfortunately it does not appear to contain a stack trace. Just to be sure though I've attached the raw CSV that was generated by the share function on the log page. Immich_log_2023-03-23T19 33 51.839022.csv

alextran1502 commented 1 year ago

Ah crap we put the cannot upsert to Isar database as a warning log instead of an error log, that is why the stack trace doesn't show.

stefano99 commented 1 year ago

I'm having something similar. I upgraded both server and client from the last point release, and as soon as I upgraded the mobile app, it slowed down. Now it takes seconds or even minutes to load the previews and the images, and only on the Android app on my phone.

I have other people with their own account on the server and, on the same smartphone model (Galaxy S21) and with a similar number of assets (roughly 6k each), they have no issue. The only difference is that i use HEIF and HEIC formats instead of AVC and JPEG.

I already uninstalled and reinstalled the app with no luck (and manually cleared the app data because after the reinstallation it was still logged in. I believe it can be the Google backup feature). I tried also running the jobs manually, but still no difference.

Basically, it looks like it's slow to reach the server.

I'll add the logs asap.

fyfrey commented 1 year ago

@kevinanderson1 I have a suspicion: There might be kind of duplicate assets in the server database. Could you please run the following query on your postgres Immich db: SELECT * FROM (SELECT Count(id), "deviceId", "deviceAssetId" FROM assets GROUP BY "deviceId", "deviceAssetId") as q WHERE q.Count > 1 This should return 0 results. If it does return some assets, these are the cause of the error in the mobile app. In this case, please paste some of the results here. Thank you!

We would need to discuss how to deal with such a situation.

@alextran1502 no worries regarding the missing stack trace in this case. I know exactly where it is: SyncService, _syncRemoteAssetsToDb, _upsertAssetsWithExif. I've reproduced the error with a unit test: When the response from the server contains duplicates of deviceId, deviceAssetId, the unique index violation occurs.. because on the mobile app these are (and must) be unique. So my guess is, either due to previous bugs duplicate IDs where uploaded or upload via WEB might create these..

kevinanderson1 commented 1 year ago

@fyfrey It appears there are duplicates which probably explains the issue. 

immich=# SELECT * FROM (SELECT Count(id), "deviceId", "deviceAssetId" FROM assets GROUP BY "deviceId", "deviceAssetId") as q WHERE q.Count > 1;
 count | deviceId |          deviceAssetId          
-------+----------+---------------------------------
     2 | CLI      | IMG_20190323_135609.jpg-3959945
     2 | CLI      | IMG_1340.JPG-88875
     2 | CLI      | DSC00371.JPG-4824723
     2 | CLI      | IMG_20190323_135610.jpg-3920803
     2 | CLI      | IMG_0964.JPG-104850
     2 | CLI      | IMG_0961.JPG-109247
     2 | CLI      | IMG_0015.JPG-2770528
     2 | CLI      | IMG_0963.JPG-102178
     2 | CLI      | IMG_20190323_135611.jpg-3832364
     2 | CLI      | IMG_0960.JPG-117821
     2 | CLI      | IMG_0956.JPG-92164
     2 | CLI      | IMG_0962.JPG-104651
fyfrey commented 1 year ago

Thank you! This is very helpful. We'll need to discuss what to do about it.

Can you check whether some of these are actual duplicates (same checksum)? You could use something like: SELECT "id","deviceAssetId", "ownerId", "deviceId", "fileCreatedAt","fileModifiedAt", encode("checksum", 'hex') as checksum FROM assets WHERE "deviceId"='replaceme' AND "deviceAssetId"='replaceme'; and look at the checksum column. Maybe you imported the same assets for different users or there were some bugs in the CLI

kevinanderson1 commented 1 year ago

@fyfrey Just to share some background information, I installed Immich on my phone to backup my phone photos but it was a very small number of photos as I usually keep them stored in Google Photos. I then decided to use the Immich cli to upload the rest of my photo library to the server. While I have another user in Immich, they haven't uploaded any photos and so only my account has any assets right now.

Is it a reasonable expectation that the deviceAssetId would match the filename on the server filesystem under the uploads directory?

If I search for those assets in the file system I get no results.

[root@redacted immich]# find -name 'IMG_20190323_135611.jpg-3832364' -o -name 'IMG_1340.JPG-88875' -o -name 'DSC00371.JPG-4824723' -o -name 'IMG_20190323_135610.jpg-3920803' -o -name 'IMG_0964.JPG-104850' -o -name 'IMG_0961.JPG-109247' -o -name 'IMG_0015.JPG-2770528' -o -name 'IMG_0963.JPG-102178' -o -name 'IMG_20190323_135611.jpg-3832364' -o -name 'IMG_0960.JPG-117821' -o -name 'IMG_0956.JPG-92164' -o -name 'IMG_0962.JPG-104651'
[root@redacted immich]#

To answer your other question, the checksums for some of the duplicate entries appear to be unique.

SELECT "id","deviceAssetId","ownerId","deviceId","fileCreatedAt","fileModifiedAt",encode("checksum",'hex') as checksum FROM assets WHERE "deviceAssetId"='IMG_20190323_135609.jpg-3959945' or "deviceAssetId"='IMG_1340.JPG-88875' or "deviceAssetId"='DSC00371.JPG-4824723' or "deviceAssetId"='IMG_20190323_135610.jpg-3920803' or "deviceAssetId"='IMG_0964.JPG-104850' or "deviceAssetId"='IMG_0961.JPG-109247' or "deviceAssetId"='IMG_0015.JPG-2770528' or "deviceAssetId"='IMG_0963.JPG-102178' or "deviceAssetId"='IMG_20190323_135611.jpg-3832364' or "deviceAssetId"='IMG_0960.JPG-117821' or "deviceAssetId"='IMG_0956.JPG-92164' or "deviceAssetId"='IMG_0962.JPG-104651' ORDER BY "deviceAssetId";
                  id                  |          deviceAssetId          |               ownerId                | deviceId |     fileCreatedAt      |       fileModifiedAt       |            checksum                 
--------------------------------------+---------------------------------+--------------------------------------+----------+------------------------+----------------------------+------------------------------------------
 080add70-275e-4647-a99f-9ef078720bb9 | DSC00371.JPG-4824723            | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2010-07-16 08:40:53+00 | 2010-07-16 19:39:57+00     | 94283cde2acc2aca693cf0353992f1c6ba58bee0
 78be3bb4-8118-4617-8660-708f308bcb04 | DSC00371.JPG-4824723            | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2010-07-16 08:40:53+00 | 2010-07-16 18:31:45+00     | ecf12d8bc39adb1eff536be1aa8c6c2dccee076b
 5d001c88-1fbc-42d3-b643-3b6f94d5dad8 | IMG_0015.JPG-2770528            | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2010-02-08 04:07:48+00 | 2010-02-28 19:21:06+00     | e1b135844e17eee93bf8da6197256aa0e5997629
 50b9033f-fc91-4d49-bd8e-98a79edebb79 | IMG_0015.JPG-2770528            | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2010-02-08 04:07:48+00 | 2010-02-21 00:11:00.196+00 | acc46bebad220c490077be05a752ada7b0e56f2d
 0ebfcbee-b0de-498b-8e2c-adf86a92e124 | IMG_0956.JPG-92164              | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 12:35:54+00 | 2012-01-17 01:21:21+00     | 7ba0012c8d2ae61e0de4c177ae553691fe6d8997
 23359f12-7f2e-4837-bfa1-d1a99c405de4 | IMG_0956.JPG-92164              | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 12:35:54+00 | 2012-01-16 21:53:44.839+00 | e1706ac8d997ca52b22314da858d4eb6a04ecce4
 3a0c1720-4840-4a45-a4cd-6b0b7c0ac192 | IMG_0960.JPG-117821             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 13:12:56+00 | 2012-01-16 21:53:41.577+00 | 10e5642c8b0e307640c5eb2244a0c86c359257ae
 12cb9315-d8b3-4bfb-8f17-92153936af39 | IMG_0960.JPG-117821             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 13:12:56+00 | 2012-01-17 01:21:32+00     | 87378168571af6699f57366a26839dd3ce632f23
 61d013cc-60d8-43c3-9357-b77a38468c1a | IMG_0961.JPG-109247             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 15:28:19+00 | 2012-01-17 01:21:36+00     | dac8c95319d5e30fa66d53f33e912c74507b8204
 bbac68f6-f870-4360-b41d-d7ce295a7cb2 | IMG_0961.JPG-109247             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 15:28:19+00 | 2012-01-16 21:53:53.832+00 | 1c859652d25f6ea5a8e7d0f0fb597af79dc81b7b
 e6aecfcc-c2ca-41e6-b8d7-d96ff7def7ce | IMG_0962.JPG-104651             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 15:28:32+00 | 2012-01-17 01:21:39+00     | 94cfd08915fa56485d5c50f6036c754371056035
 70429f69-323b-4edd-a773-f942f7b05f4a | IMG_0962.JPG-104651             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 15:28:32+00 | 2012-01-16 21:53:56.02+00  | aa5046cd3ce410d3704b657b37947871f2fce294
 82451673-b3c8-4017-88f4-854a6b360d22 | IMG_0963.JPG-102178             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 15:28:41+00 | 2012-01-17 01:21:51+00     | 2c1cd32d46fbf2f5599d819d05a2a5ac71707364
 a798b3a8-839d-4de1-b168-7521120eb790 | IMG_0963.JPG-102178             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 15:28:41+00 | 2012-01-16 21:53:50.122+00 | ae204560627f22418b36b283c7d436690f41f4ae
 19cace70-f2ac-481d-b403-d106a64808be | IMG_0964.JPG-104850             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 15:28:57+00 | 2012-01-16 21:54:23.654+00 | c5de144c9a9cdbec672636e2df4ba019b8893acb
 7307b82f-e9aa-4a37-b302-08fe15738046 | IMG_0964.JPG-104850             | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-01-16 15:28:57+00 | 2012-01-17 01:21:53+00     | 722691ac28b76307ef0369147eee995bdc1f7744
 91b40a0e-b11e-4c1b-a625-81fd55123baf | IMG_1340.JPG-88875              | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-05-18 18:43:07+00 | 2012-05-20 00:15:48.563+00 | d907c77e81a21bd07f699772d556e7bb148df64a
 c1a88848-5ce9-456b-b3a1-f596d3ee1553 | IMG_1340.JPG-88875              | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2012-05-18 18:43:07+00 | 2012-05-20 23:11:35+00     | 71568274926e6114529f02ad8b377b6f56142624
 be03ec4e-252c-47f4-91df-da2fccc46e3c | IMG_20190323_135609.jpg-3959945 | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2019-03-23 17:56:09+00 | 2019-03-23 17:56:09+00     | c33ae6c0ba419ba2d74f9590ebbe17db2efa79de
 f44617dd-f196-4e42-a1d6-66e65ee7fa9a | IMG_20190323_135609.jpg-3959945 | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2019-03-23 17:56:09+00 | 2020-08-31 23:04:42+00     | f21d363f91384fc08c3e34f4701cec633385b9f6
 0c5ac06b-473d-4b04-9580-b9a1172f0a50 | IMG_20190323_135610.jpg-3920803 | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2019-03-23 17:56:10+00 | 2019-03-23 17:56:10+00     | 2ccba786936a5137d01391c608848fe3cbdc4788
 5ce890e3-25c2-4a1b-b4e8-73d2b494f58c | IMG_20190323_135610.jpg-3920803 | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2019-03-23 17:56:10+00 | 2020-08-31 23:04:42+00     | ba9d2148c11c386fbd5c6a1aa8797b4b49146a00
 b5368040-dba3-4c58-ab2b-3724aa9016be | IMG_20190323_135611.jpg-3832364 | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2019-03-23 17:56:11+00 | 2020-08-31 23:04:42+00     | c5c6b12e2ef014b922cf394480d6bbdaf5c4d67f
 ee4af6cf-5bd0-44c5-a81a-3a1c9956ce3a | IMG_20190323_135611.jpg-3832364 | 0c8319a8-5662-47dd-9fb6-febef6e029c0 | CLI      | 2019-03-23 17:56:11+00 | 2019-03-23 17:56:11+00     | 8489410eefb7b47cc1a77e1e7f6e7f532ea873cb
(24 rows)
fyfrey commented 1 year ago

Alright, thanks for the information. I guess I need to somehow modify the app to support this case. The problematic assets seem to be files that have the same name and creation date but one of them was modified later and also uploaded (which is totally fine). Sadly, the CLI gave both of them the same deviceAssetId.

Is it a reasonable expectation that the deviceAssetId would match the filename on the server filesystem under the uploads directory?

You can find the assets using the originalPath DB column.

alextran1502 commented 1 year ago

@fyfrey I think this is an issue with the CLI, so it should be fixed from that side instead of fixing the implementation in the mobile app. However, I don't know the best path for this yet.

PeterBasista commented 1 year ago

@fyfrey @alextran1502 Hi, I just came across this same problem. After I tried everything you wrote here I found the problem. LivePhotos :) Same name, same time, same device, it came to me while I was reading it here. It's not visible in DB because it probably translates to some fancy name but this fact that livephotos have the same name but different type is still there. Then DB just confirmed it to me. Here is my DB dump.

id deviceAssetId ownerId deviceId fileCreatedAt fileModifiedAt originalPath checksum
1f6db386-049d-4d39-bceb-3bcb61e39af4 033695C9-AB49-4479-A618-CC9F8DA80576/L0/001 d8c4c8e3-a6ee-4889-8148-4e7db21660f4 fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937 2023-02-26 16:35:03.000 +0100 2023-02-26 17:35:06.000 +0100 upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/2023/February/26/IMG_0965.heic 0019cd23e69d6991fe386c1a716d706fe5485fc7
057af2ed-590c-4d79-b1a3-399822d96e25 033695C9-AB49-4479-A618-CC9F8DA80576/L0/001 d8c4c8e3-a6ee-4889-8148-4e7db21660f4 fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937 2023-02-26 17:35:03.000 +0100 2023-02-26 17:35:06.000 +0100 upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/2023/February/26/IMG_0965.mov 8a65a7999bdbb86ab2f7c29fea781d53ed0155ce
b61c6da5-736c-4a1e-aefa-c60328618ba7 03D8C528-D3CE-49BF-B8C8-71227DA21713/L0/001 d8c4c8e3-a6ee-4889-8148-4e7db21660f4 fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937 2023-02-12 14:19:27.000 +0100 2023-02-13 11:21:15.000 +0100 upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/2023/February/12/IMG_0818.mov 4f9b017b339dd6e64f12f44ea54c840b37db14f6
51c9d2fc-2bcc-459f-84f2-bbdb08ac0a4f 03D8C528-D3CE-49BF-B8C8-71227DA21713/L0/001 d8c4c8e3-a6ee-4889-8148-4e7db21660f4 fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937 2023-02-12 13:19:27.000 +0100 2023-02-13 11:21:15.000 +0100 upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/2023/February/12/IMG_0818.heic e2dae813cb2d52a608b146e20acddd513ff3e7b0

assets_202303242025.csv

PS: I have 800+ of them

alextran1502 commented 1 year ago

@PeterBasista Thank you, this is very useful info

alextran1502 commented 1 year ago

Fixed in #2085

kevinanderson1 commented 1 year ago

@fyfrey @alextran1502 I just wanted to say thank you for the great work! I can confirm the updated app fixes the problem I saw. :smiley:

fyfrey commented 1 year ago

@fyfrey @alextran1502 I just wanted to say thank you for the great work! I can confirm the updated app fixes the problem I saw. :smiley:

Excellent! Thanks for confirming it's working as expected now for you as well