Android app confuses fresh pictures/videos for old non-local pictures/videos

[aphymi]

The bug

This is a weird one, but bear with me.

Since I re-installed the app after a factory-reset about a week ago, whenever a new photo (or video, but I'll just say photos here for brevity) is taken (added to the Camera album, or any other backup albums), the Immich app mistakes that photo for some other photo on the timeline (which does not exist on the phone), seemingly believing that it now has a copy of that old photo on the local device, and that that copy is this new photo file.

The photo does not get uploaded to the server or added to its proper place in the timeline. The new photo takes the old photo's place on the app's timeline. If the new photo replaces a video, the preview shows correctly, but with the old video's length preview on it, and expanding the 'video' results in an infinite loading wheel.

I don't know by what process it's deciding which old photo to mistake the new one for, but they've all been from the last year so far, and photos taken close together seem to replace photos on the timeline that were taken close together.

---

That description may be unclear, so here's an example. Say I take a photo of a candle on my desk with the default camera app:

Finding it through my phone's file browser, the name of the photo file is PXL_20231109_224242285.MP.jpg

Loading the Immich app and finding the photo through the camera album, it's apparently mistaken the photo for a video (trying and failing to play it), and the file name it believes it has is PXL_20221120_182501591.mp4

Loading up immich through the browser and looking at that date, there is indeed a completely separate video with that filename, which the candle photo has replaced in the app's timeline.

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

1.85.0

Version of Immich Mobile App

1.85.0 build.109

Platform with the issue

• [ ] Server
• [ ] Web
• [X] Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2001:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=<redacted>
DB_PASSWORD=<redacted>
DB_DATABASE_NAME=<redacted>

REDIS_HOSTNAME=immich_redis

UPLOAD_LOCATION=<redacted>/immich/

TYPESENSE_API_KEY=<redacted>

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=false
IMMICH_MACHINE_LEARNING_ENABLED=false

Reproduction steps

As described above.

Additional information

This issue persists even without access to the internet, so it's probably contained to the mobile app.

This issue persists through an uninstall/reinstall from the play store, and cache clear.

Loading up a completely separate instance of the app from source (37ab37b) through USB debugging with Flutter and VSCode, this issue does not occur on the debug instance. All local photos that have been affected by this issue on the main instance get recognised and uploaded as they originally should have been. Closing down the debug instance, the issue persists on the main instance with new photos.

The app log has nothing notable. Just variations on:

2023-11-09 17:43:20.248175,LogLevel.INFO,"AlbumService","Syncing completed. Changes: false",""
2023-11-09 17:43:20.246320,LogLevel.INFO,"AlbumService","'Recents' is not selected, keeping only selected albums",""
2023-11-09 17:43:20.246156,LogLevel.INFO,"AlbumService","Found 8 device albums",""

[alextran1502]

Can you go into the app settings >advanced > turn on debug mode and go look at the details of the weird asset again?

[aphymi]

For the same candle photo, enabling Troubleshooting in the advanced settings gives the following:

{
  "id": 1083,
  "remoteId": "5d68fecc-6e26-4bed-884b-ebb50ed92f70",
  "localId": "1000000293",
  "checksum": "TEgquisHQbxebs2GowZsX7zxD38=",
  "ownerId": 9130815009914457114, 
  "livePhotoVideoId": "N/A",
  "stackCount": "0",
  "stackParentId": "N/A",
  "fileCreatedAt": "2022-11-20 13:25:10.000",
  "fileModifiedAt": "2022-11-20 13:25:10.000", 
  "updatedAt": "2023-08-12 22:59:40.351", 
  "durationInSeconds": 5, 
  "type": "AssetType.video",
  "fileName": "PXL_20221120_182501591.mp4", 
  "isFavorite": false, 
  "isRemote: true,
  "storage": "AssetState.merged",
  "width": 1920,
  "height": 1080,
  "isArchived": false,
  "isTrashed": false,
}

[sic] on the missing quote after "isRemote

[aphymi]

Opening the app (same version) on another phone to check, all of that JSON information is from the actual video that the candle photo is replacing; the only fields that are different on the other phone are "id" and "localId".

[jrasm91]

It looks like the new photo is being incorrectly matched against an already uploaded file, so they presumably have the same hash or there is a bug in the matching logic.

I have heard that isolates don't run in debug more, but do in production, so there could be a race condition related to the hash/matching implementation.

Thoughts @fyfrey @shenlong-tanwen ?

[jrasm91]

Not sure if it is important or not, but the original picture is named .MP.jpg,, indicating that it is an android motion photo, so it has a photo with an embedded video also included in the file itself.

[aphymi]

The single candle photo here is just an example; this same thing happens for every photo and video I've taken for the last week, so it's definitely not coincidentally matched hashes.

[fyfrey]

I have a hunch that Google saves and restores Immich data.

The app keeps a mapping from the system's asset ID to the file checksum. Without this, it'd need to recompute the expensive file checksums for all assets on the device every time again.

Due to the factory reset, the systems asset IDs are reset to start at 0 etc. If the old mapping is consulted, a wrong checksum is used for new assets because the system's IDs are re-used for entirely different assets.

Please try clearing the app data (and not only the cache).

[aphymi]

Fully clearing the app's storage resolved the issue; opening the app after, new photos and videos get recognised as normal. So that's a workaround, at least.

That does sound like a plausible etiology.

[fyfrey]

I think properly clearing the data is a solution and not only a workaround. The app can't really know whether someone (in this case the Google backup restore) messed with its data. Or can it?

[jrasm91]

It sounds like we basically have a "cache" of hashes, based on local asset ids. That cache becomes invalid/stale when a factory reset happens.

The real question should be - is there anyway to be notified or become aware that the cache is not invalid and should be evicted.

[fyfrey]

Did some research. It seems some device id is reset after a factory reset. If this is true (and we can get that ID in Dart), we could get this ID on every app start and compare it to the ID saved in our database. One change, invalidate all data (maybe ask user first)