search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
44.98k stars 2.18k forks source link

Duplicate photos on mobile #2129

Closed dsippel closed 1 year ago

dsippel commented 1 year ago

The bug

Since a couple of releases all photos uploaded from my iPhone, that are still on my phone, show up as duplicates in the photos view on my iPhone. See screenshot attached.

It probably is an issue with the recognition of what’s already uploaded and what not. In the web view or on an android tablet no duplicates are shown.

What I tried so far: logging out and back in on the mobile iOS app. Uninstalling and reinstalling the iOS app. Restarting the immich server. All without any luck.

Any suggestions? IMG_3616

The OS that Immich Server is running on

Unraid

Version of Immich Server

v.1.52.1

Version of Immich Mobile App

v.1.52.0

Platform with the issue

Your docker-compose.yml content

Not using docker compose. Using the all in one docker image: https://github.com/imagegenius/docker-immich

Your .env content

-

Reproduction steps

Open the mobile iOS app with photos that are still on the device and already uploaded.

Additional information

No response

alextran1502 commented 1 year ago

Thank you, we are also received similar reports. So we will take a look at this. I just want to confirm that all of those shown as duplicated photos have been uploaded, correct? Can you help me navigate to the backup page and take a screenshot of that page as well?

dsippel commented 1 year ago

Thanks for the fast reply and this amazing piece of software!

Yes, all of the photos are uploaded correctly. Can view them on the web and on other devices.

Screenshots of my backup page attached. IMG_3617 IMG_3618

alextran1502 commented 1 year ago

Ok, the current workaround is to choose an album that has 1 or 2 assets, so it won't show the local assets on your timeline.

alextran1502 commented 1 year ago

@fyfrey Tagging you in case you can come up with other ideas of how this could happen

dsippel commented 1 year ago

Ok, the current workaround is to choose an album that has 1 or 2 assets, so it won't show the local assets on your timeline.

Ok, thanks. That works as a workaround. But then automatic upload is not working…

If you need any more information from my side, let me know!

fyfrey commented 1 year ago

@dsippel could you export the logs: profile drawer -> logs -> share (top right) or if this fails take a screenshot of any yellow or red entries in the log (or else simply the topmost logs after logout+login). Thanks!

dsippel commented 1 year ago

Exporting the log file didn’t work. Please find the screenshots of the log after a fresh sign in attached. Thanks for helping!

IMG_3625 IMG_3626 IMG_3627 IMG_3628 IMG_3629 IMG_3630 IMG_3631 IMG_3632 IMG_3633

Isti6github commented 1 year ago

Hello,

I have the same issue after the new relase (mobile iOS and server)imageimage

alextran1502 commented 1 year ago

Hello,

I have the same issue after the new relase (mobile iOS and server)image![image](https://user-images.githubusercontent.com/127151924/230789542-28b29878-4265-4e19-8522-78a79689d10c.

Does it happen on all photos? And did you upload those photos on from any other client, Web or CLI?

bart268 commented 1 year ago

I also got the same issue starting about two weeks ago. I only use the web interface and IOS client on iPhone, all photo's are uploaded using the IOS client. Sometimes the app seems to think all the photo's are new and tries to upload them (sort of). My remainder photo's are now on -328. When I see this happening top right in the app I quickly close the app to not have all the photo's as duplicate. The timeline has now 328 (I guess) duplicates where one of the photo is uploaded and one isn't uploaded, the web UI doens't have the duplicate photo's.

I think this happened after the new folder structure was introduced, I migrated from the old to the new structure but not sure if this has anything to do with it.

This is the logging from last time it happened, I noticed the issue at 8:21 on april 11th. Also added logging from the day before, maybe this is related to the issue.

created_at,level,context,message,stacktrace
2023-04-11 08:31:35.239391,LogLevel.INFO,"AssetNotifier","state is already up-to-date",""
2023-04-11 08:31:35.236382,LogLevel.INFO,"AssetNotifier","Load assets: 1836ms",""
2023-04-11 08:31:35.236347,LogLevel.INFO,"AlbumService","Syncing completed. Changes: false",""
2023-04-11 08:31:34.704940,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-11 08:31:33.423056,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-11 08:31:33.401727,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-04-11 08:31:09.891857,LogLevel.INFO,"AssetNotifier","state is already up-to-date",""
2023-04-11 08:31:09.891337,LogLevel.INFO,"AssetNotifier","Load assets: 2065ms",""
2023-04-11 08:31:09.891305,LogLevel.INFO,"AlbumService","Syncing completed. Changes: false",""
2023-04-11 08:31:09.392588,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-11 08:31:07.846762,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-11 08:31:07.827505,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-04-11 08:22:31.464884,LogLevel.INFO,"AssetNotifier","state is already up-to-date",""
2023-04-11 08:22:31.464297,LogLevel.INFO,"AssetNotifier","Load assets: 1566ms",""
2023-04-11 08:22:31.464270,LogLevel.INFO,"AlbumService","Syncing completed. Changes: false",""
2023-04-11 08:22:31.051190,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-11 08:22:27.514539,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-11 08:31:09.891857,LogLevel.INFO,"AssetNotifier","state is already up-to-date",""
2023-04-11 08:31:09.891337,LogLevel.INFO,"AssetNotifier","Load assets: 2065ms",""
2023-04-11 08:31:09.891305,LogLevel.INFO,"AlbumService","Syncing completed. Changes: false",""
2023-04-11 08:31:09.392588,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-11 08:31:07.846762,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-11 08:31:07.827505,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-04-11 08:22:31.464884,LogLevel.INFO,"AssetNotifier","state is already up-to-date",""
2023-04-11 08:22:31.464297,LogLevel.INFO,"AssetNotifier","Load assets: 1566ms",""
2023-04-11 08:22:31.464270,LogLevel.INFO,"AlbumService","Syncing completed. Changes: false",""
2023-04-11 08:22:31.051190,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-11 08:22:27.514539,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-11 08:22:19.848796,LogLevel.INFO,"AssetNotifier","setting new asset state",""
2023-04-11 08:22:19.819641,LogLevel.INFO,"AssetNotifier","Load assets: 1389ms",""
2023-04-11 08:22:19.819559,LogLevel.INFO,"AlbumService","Syncing completed. Changes: true",""
2023-04-11 08:22:19.675790,LogLevel.INFO,"SyncService","Synced changes of local album AssetPathEntity(id: 7054C870-C326-40DF-933B-35759EC4E3D0/L0/040, name: Recent) to DB",""
2023-04-11 08:22:19.313763,LogLevel.INFO,"AlbumService","refreshDeviceAlbums is already in progress",""
2023-04-11 08:22:18.788620,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-11 08:22:18.434034,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-11 08:22:18.429840,LogLevel.INFO,"AssetNotifier","Reading assets 14713 from DB: 50ms",""
2023-04-11 08:22:18.367613,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-04-11 08:22:01.138378,LogLevel.SEVERE,"ImmichErrorLogger","PathNotFoundException: Cannot delete file, path = '/private/var/mobile/Containers/Data/Application/E469E05F-4D60-4D32-A94D-2D638B82416E/tmp/.image/1681161904.923239_o_IMG_3107.HEIC' (OS Error: No such file or directory, errno = 2)","#0      _File.throwIfError (dart:io/file_impl.dart:629)
#1      _File._deleteSync (dart:io/file_impl.dart:312)
#2      FileSystemEntity.deleteSync (dart:io/file_system_entity.dart:424)
#3      BackupService.backupAsset (package:immich_mobile/modules/backup/services/backup.service.dart:319)
<asynchronous suspension>
#4      BackupNotifier.startBackupProcess (package:immich_mobile/modules/backup/providers/backup.provider.dart:448)
<asynchronous suspension>
#5      BackupNotifier._resumeBackup (package:immich_mobile/modules/backup/providers/backup.provider.dart:573)
<asynchronous suspension>
"
2023-04-11 08:22:01.069906,LogLevel.INFO,"AssetNotifier","setting new asset state",""
2023-04-11 08:22:01.035569,LogLevel.INFO,"AssetNotifier","Load assets: 3450ms",""
2023-04-11 08:22:01.035508,LogLevel.INFO,"AlbumService","Syncing completed. Changes: true",""
2023-04-11 08:22:00.919719,LogLevel.INFO,"SyncService","Synced changes of local album AssetPathEntity(id: 7054C870-C326-40DF-933B-35759EC4E3D0/L0/040, name: Recent) to DB",""
2023-04-11 08:21:59.727618,LogLevel.INFO,"AlbumService","refreshDeviceAlbums is already in progress",""
2023-04-11 08:21:59.727584,LogLevel.INFO,"SyncService","Upserted 2 assets into the DB",""
2023-04-11 08:21:57.987896,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-11 08:21:57.825795,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-11 08:21:57.811949,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-04-11 08:21:57.585380,LogLevel.INFO,"AssetNotifier","Reading assets 7554 from DB: 25224ms",""
2023-04-11 08:21:57.521727,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-11 08:21:32.345925,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-04-10 14:20:06.524588,LogLevel.INFO,"BackupNotifier","No Asset On Device - Abort Backup Process",""
2023-04-10 14:20:05.242409,LogLevel.INFO,"AssetNotifier","state is already up-to-date",""
2023-04-10 14:20:05.240362,LogLevel.INFO,"AssetNotifier","Load assets: 828ms",""
2023-04-10 14:20:05.240331,LogLevel.INFO,"AlbumService","Syncing completed. Changes: false",""
2023-04-10 14:20:04.925762,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-10 14:20:04.465390,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-10 14:20:04.464453,LogLevel.SEVERE,"AssetService","Error while getting remote assets","#0      ApiClient.invokeAPI (package:openapi/api_client.dart:105)
<asynchronous suspension>
#1      WithETag.getAllAssetsWithETag (package:immich_mobile/utils/openapi_extensions.dart:20)
<asynchronous suspension>
#2      AssetService._getRemoteAssets (package:immich_mobile/shared/services/asset.service.dart:64)
<asynchronous suspension>
#3      AssetService.refreshRemoteAssets.<anonymous closure> (package:immich_mobile/shared/services/asset.service.dart:49)
<asynchronous suspension>
#4      SyncService._syncRemoteAssetsToDb (package:immich_mobile/shared/services/sync.service.dart:154)
<asynchronous suspension>
#5      AssetService.refreshRemoteAssets (package:immich_mobile/shared/services/asset.service.dart:48)
<asynchronous suspension>
#6      AssetNotifier.getAllAsset (package:immich_mobile/shared/providers/asset.provider.dart:113)
<asynchronous suspension>
"
2023-04-10 14:20:04.414635,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-04-10 05:27:00.008112,LogLevel.INFO,"BackupNotifier","No Asset On Device - Abort Backup Process",""
2023-04-10 05:26:46.969242,LogLevel.INFO,"BackupNotifier","Found 65 local albums",""
2023-04-10 05:26:46.929239,LogLevel.INFO,"AssetNotifier","state is already up-to-date",""
2023-04-10 05:26:46.928843,LogLevel.INFO,"BackupNotifier","[_resumeBackup] Start back up",""
2023-04-10 05:26:46.753153,LogLevel.INFO,"AssetNotifier","Load assets: 3489ms",""
2023-04-10 05:26:46.752894,LogLevel.INFO,"AlbumService","Syncing completed. Changes: false",""
2023-04-10 05:26:44.814657,LogLevel.INFO,"AlbumService","Found 59 device albums",""
2023-04-10 05:26:43.319299,LogLevel.INFO,"AlbumService","refreshDeviceAlbums is already in progress",""
2023-04-10 05:26:43.319011,LogLevel.SEVERE,"AssetService","Error while getting remote assets","#0      ApiClient.invokeAPI (package:openapi/api_client.dart:105)
<asynchronous suspension>
#1      WithETag.getAllAssetsWithETag (package:immich_mobile/utils/openapi_extensions.dart:20)
<asynchronous suspension>
#2      AssetService._getRemoteAssets (package:immich_mobile/shared/services/asset.service.dart:64)
<asynchronous suspension>
#3      AssetService.refreshRemoteAssets.<anonymous closure> (package:immich_mobile/shared/services/asset.service.dart:49)
<asynchronous suspension>
#4      SyncService._syncRemoteAssetsToDb (package:immich_mobile/shared/services/sync.service.dart:154)
<asynchronous suspension>
#5      AssetService.refreshRemoteAssets (package:immich_mobile/shared/services/asset.service.dart:48)
<asynchronous suspension>
#6      AssetNotifier.getAllAsset (package:immich_mobile/shared/providers/asset.provider.dart:113)
<asynchronous suspension>
"
2023-04-10 05:26:43.264046,LogLevel.INFO,"AssetNotifier","Reading assets 7554 from DB: 212ms",""
2023-04-10 05:26:43.211549,LogLevel.SEVERE,"ImmichErrorLogger","ApiException 400: Socket operation failed: GET /album (Inner exception: SocketException: Failed host lookup: '<dns name>' (OS Error: nodename nor servname provided, or not known, errno = 8))

#0      _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:694)
#1      _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1855)
#2      RawSocket.startConnect (dart:io-patch/socket_patch.dart:27)
#3      RawSecureSocket.startConnect (dart:io/secure_socket.dart:297)
#4      SecureSocket.startConnect (dart:io/secure_socket.dart:77)
#5      _ConnectionTarget.connect (dart:_http/http_impl.dart:2462)
#6      _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2906)
#7      _HttpClient._getConnection (dart:_http/http_impl.dart:2911)
#8      _HttpClient._openUrl (dart:_http/http_impl.dart:2766)
#9      _HttpClient.openUrl (dart:_http/http_impl.dart:2604)
#10     IOClient.send (package:http/src/io_client.dart:35)
#11     BaseClient._sendUnstreamed (package:http/src/base_client.dart:93)
#12     BaseClient.get (package:http/src/base_client.dart:27)
#13     ApiClient.invokeAPI (package:openapi/api_client.dart:102)
<asynchronous suspension>
#14     AlbumApi.getAllAlbums (package:openapi/api/album_api.dart:514)
<asynchronous suspension>
#15     AlbumService.refreshRemoteAlbums (package:immich_mobile/modules/album/services/album.service.dart:149)
<asynchronous suspension>
#16     SharedAlbumNotifier.getAllSharedAlbums (package:immich_mobile/modules/album/providers/shared_album.provider.dart:48)
<asynchronous suspension>
","#0      ApiClient.invokeAPI (package:openapi/api_client.dart:105)
<asynchronous suspension>
#1      AlbumApi.getAllAlbums (package:openapi/api/album_api.dart:514)
<asynchronous suspension>
#2      AlbumService.refreshRemoteAlbums (package:immich_mobile/modules/album/services/album.service.dart:149)
<asynchronous suspension>
#3      SharedAlbumNotifier.getAllSharedAlbums (package:immich_mobile/modules/album/providers/shared_album.provider.dart:48)
<asynchronous suspension>
dsippel commented 1 year ago

The problem still persists for me with the newest server and iOS version (1.53.0). I tried uninstalling and reinstalling the iOS app and restarting the server. With the newest iOS app, the log export works now. Please find it attached. If you need any more information, let me know.

And as always: thanks again for this great piece of software! Amazing work!

dsippel commented 1 year ago

Immich_log_2023-04-11T09:18:45.942815.csv

alextran1502 commented 1 year ago

The problem still persists for me with the newest server and iOS version (1.53.0). I tried uninstalling and reinstalling the iOS app and restarting the server. With the newest iOS app, the log export works now. Please find it attached. If you need any more information, let me know.

And as always: thanks again for this great piece of software! Amazing work!

Thank you for the log, we still haven't found the root cause yet and in the process of building more debug info to help find the bug

fyfrey commented 1 year ago

@dsippel thank you for providing the log. Do the duplicates also show in the album view? Scroll down on the library page to "Albums on Device" and open the local/device albums. I'd would be very interesting to know whether the duplicates from the main timeline are also shown as duplicates here!

Isti6github commented 1 year ago

@dsippel thank you for providing the log. Do the duplicates also show in the album view? Scroll down on the library page to "Albums on Device" and open the local/device albums. I'd would be very interesting to know whether the duplicates from the main timeline are also shown as duplicates here!

Hi, I see the duplicates pictures in the "Recents" folder view too.

dsippel commented 1 year ago

@dsippel thank you for providing the log. Do the duplicates also show in the album view? Scroll down on the library page to "Albums on Device" and open the local/device albums. I'd would be very interesting to know whether the duplicates from the main timeline are also shown as duplicates here!

Nope, I don't see the duplicates there.

fyfrey commented 1 year ago

Thank you both! That is certainly an ambivalent picture; maybe these are two separate issues. The next release will allow to get more debug info. Hopefully this will help us to resolve this issue.

dsippel commented 1 year ago

Hey immich team,

great new release 1.54.0. I updated my server and the mobile app. The issue with duplicates shown still persists. Please see attached screenshot and log file. Finest logging level selected. Hope this helps track down the issue.

Best

Daniel

IMG_3724 Immich_log_2023-04-19T17:31:46.925624.csv

alextran1502 commented 1 year ago

@dsippel Thank you for the follow-up, and we didn't have a fix to this issue because we didn't know what to fix. Can you turn on the Advanced troubleshooting in the setting, go to a set of duplicate photos, open the info panel, take a screenshot, or add the info here?

Thanks!

dsippel commented 1 year ago

Please find attached the two screenshots. The first one is from the already uploaded file (cloud with check mark). The second from the local file (cloud struck through). Hope this helps.

alextran1502 commented 1 year ago

@dsippel As you can see the localId of the two files are different as well as the fileModifiedAt property is different. The width and height of the two files are also different, which indicates two different files. Do you know if they ever get modified, or download from different sources? What is the album are you selecting for update?

@fyfrey FYI

dsippel commented 1 year ago

This is very strange. The photos were taken on my iPhone and were uploaded through the immich app. I did not modify them in any way before or after upload! I didn't change the resolution and they don't get downloaded from different sources. iCloud Photos is disabled (photostream is on, but I don't think that's a problem). I selected the "Recents (All)" album als backup album.

alextran1502 commented 1 year ago

Do you use the Share Album feature of Apple Photos app?

alextran1502 commented 1 year ago

Can you help take a screenshot of your photos app settings?

dsippel commented 1 year ago

here are my settings:

IMG_3727 IMG_3728

I have one shared Album I was invited to. I disabled it now in settings, logged out of immich in the app and back in. No change. Still duplicates.

alextran1502 commented 1 year ago

hmm very very strange. What is your iPhone model and iOS version?

dsippel commented 1 year ago

iPhone 13 Pro, iOS 16.4.1.

I tried to uninstall and reinstall the app now, select a different album for backup and then agains the recents album. All that doesn't change anything...

dsippel commented 1 year ago

OK, I think I found the problem! Disabling the PhotoStream made the duplicates disappear! 🙂 Thanks for all the help and effort! If you need any more information from me, let me know!

alextran1502 commented 1 year ago

OHHHH very good data point. Thank you!!!

alextran1502 commented 1 year ago

Do you know if those duplicated photos can be uploaded to Immich?

dsippel commented 1 year ago

They were not uploaded. Somehow the photostream was the reason that the local photos were displayed as duplicates. None of these with the lower resolution were uploaded. Or what do you mean?

alextran1502 commented 1 year ago

I mean can they be uploaded at all?

dsippel commented 1 year ago

Nope

yoyotueur commented 1 year ago

I have the same issue. But I don't have the photostream setting in Setting>Photos

jeff15110168 commented 5 months ago

I have the same issue. But I don't have the photostream setting in Setting>Photos

I also have the same issue. I don't have photostream option but i do use shared photo libraries a lot so turning it off is not an option.

I currently have every single "local" folder selected for backup. Is that a bad idea and should i just select "recent" for backup that then captures all the files on my local iphone (i don't want to backup any shared albums)