search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
47.36k stars 2.41k forks source link

„Empty“ people / faces detected from Live Photo Video #8597

Closed raisinbear closed 5 months ago

raisinbear commented 6 months ago

The bug

Hi, Thanks once more for the fantastic work you guys are doing with Immich! It took me a while to figure this one out, but is has been there for several versions. Under the people category, empty people keep showing up for me. Empty by means of no assets appearing when clicking on the person. By now, I’m pretty much sure, those are created when analyzing Live Photo Video frame(s) and a face is visible on that that is not visible in the static image itself or is different enough that it isn’t assigned to the same person. Sorry if that has been brought up elsewhere, but I couldn’t find a matching issue. Also, this may be intentional, but it can be quite hard to assign the correct person from the small preview image, not being able to see the whole image / frame as context.

The OS that Immich Server is running on

Raspbian

Version of Immich Server

v1.101.0

Version of Immich Mobile App

v1.101.0

Platform with the issue

Your docker-compose.yml content

IMO not relevant, please let me know if required.

Your .env content

IMO not relevant, please let me know if required.

Reproduction steps

1. Upload Live Photo with person in the (first frame of the?) video part that is not visible in the still image.
2. Observe empty person being created that has no assets linked to it.

Additional information

No response

aviv926 commented 6 months ago

Can you run the query?

http://yourimmich.com/api/person/56304fd0-62ba-4b0c-a95e-8512f21ab0b0/assets

When the value 56304fd0-62ba-4b0c-a95e-8512f21ab0b0 is the person's ID. You can enter the empty face created for him, above the URL will be the ID as in the format I sent above.

raisinbear commented 6 months ago

Sure, this is the reply: []

raisinbear commented 6 months ago

Sure, this is the reply: []

Just to clarify: This is one of the persons only showing up in a Live Photo accompanying video file as the face is obscured in the still image.

aviv926 commented 6 months ago

Sure, this is the reply: []

Just to clarify: This is one of the persons only showing up in a Live Photo accompanying video file as the face is obscured in the still image.

The reply you sent is empty, I guess that's how it appears or is it a miss?

raisinbear commented 6 months ago

Sure, this is the reply: []

Just to clarify: This is one of the persons only showing up in a Live Photo accompanying video file as the face is obscured in the still image.

The reply you sent is empty, I guess that's how it appears or is it a miss?

I don’t know what should be in the reply for a „valid“ person, but I guess it should be a list of assets, where for this person it is an empty list, correct? So yeah, would make sense that the reply is empty since no assets appear when clicking on the person‘s thumbnail either.

aviv926 commented 6 months ago

Sure, this is the reply: []

Just to clarify: This is one of the persons only showing up in a Live Photo accompanying video file as the face is obscured in the still image.

The reply you sent is empty, I guess that's how it appears or is it a miss?

I don’t know what should be in the reply for a „valid“ person, but I guess it should be a list of assets, where for this person it is an empty list, correct? So yeah, would make sense that the reply is empty since no assets appear when clicking on the person‘s thumbnail either.

It should retrieve the person's assets. The purpose of this was to retrieve the assets and check what the original asset was where the face was identified.

Since it returned an empty result, it is not clear whether it is a bug.

I'll add a label so we can find out what the development team thinks about it.

raisinbear commented 6 months ago

Since it returned an empty result, it is not clear whether it is a bug.

I'll add a label so we can find out what the development team thinks about it.

Got it, thanks. As I said this one face (as many others by now) is only visible in the video part and not the image. In the meantime, I’ll tinker a bit and see what I can find in the db.

raisinbear commented 5 months ago

Ok, so for completeness sake I checked with the person which led to the empty response above. In the db, this person belongs to a face which, as an asset, has the .mov file of a Live Photo, as suspected. I believe, the url response is empty, as Live Photo videos are not assets to be supplied (shown, listed) independently from the still image, by the framework, but that is pure guesswork.

aviv926 commented 5 months ago

Ok, so for completeness sake I checked with the person which led to the empty response above. In the db, this person belongs to a face which, as an asset, has the .mov file of a Live Photo, as suspected. I believe, the url response is empty, as Live Photo videos are not assets to be supplied (shown, listed) independently from the still image, by the framework, but that is pure guesswork.

There is a possibility that it was fixed with the merge of #8724, could you upload a sample file so that it can be checked? If you don't want to reveal a face you can take a picture of an ad face or something.

raisinbear commented 5 months ago

There is a possibility that it was fixed with the merge of #8724, could you upload a sample file so that it can be checked? If you don't want to reveal a face you can take a picture of an ad face or something.

Thanks for the info! I’m afraid I can’t share any of the original photos, there are too many people plus infants in them to ask for permission. I’ll try and capture one from an ad or something as you suggested. But getting the scenario right may be challenging. If I can’t produce one, I’ll just test with the next release and report back.

raisinbear commented 5 months ago

There is a possibility that it was fixed with the merge of #8724, could you upload a sample file so that it can be checked? If you don't want to reveal a face you can take a picture of an ad face or something.

Ok, took me a while to get it right, but I got one that is working. However, I don't know how else to share it but as the .heic + .mov files. Not sure how you would combine them back into a live photo and if the effect persists then. Please let me know if there is another way to properly get this to you.

IMG_4304.zip

raisinbear commented 5 months ago

And maybe one other thing. I stumbled upon another issue while trying to produce a sample live photo, which might make it hard / impossible for anyone else to reproduce the issue: Upon uploading a single live photo, I would regularly get this kind of error signature.

Error ``` immich_microservices | [Nest] 7 - 04/13/2024, 10:03:58 AM ERROR [JobService] Unable to run job handler (thumbnailGene ration/generate-preview): Error: ffprobe exited with code 1 immich_microservices | ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers immich_microservices | built with gcc 12 (Debian 12.2.0-14) immich_microservices | configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --d isable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enab le-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --en able-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libm p3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --ena ble-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=a rm64 --cross-prefix=/usr/bin/aarch64-linux-gnu- --toolchain=hardened --enable-cross-compile --enable-rkmpp --enable-rkrga immich_microservices | libavutil 58. 2.100 / 58. 2.100 immich_microservices | libavcodec 60. 3.100 / 60. 3.100 immich_microservices | libavformat 60. 3.100 / 60. 3.100 immich_microservices | libavdevice 60. 1.100 / 60. 1.100 immich_microservices | libavfilter 9. 3.100 / 9. 3.100 immich_microservices | libswscale 7. 1.100 / 7. 1.100 immich_microservices | libswresample 4. 10.100 / 4. 10.100 immich_microservices | libpostproc 57. 1.100 / 57. 1.100 immich_microservices | upload/library/ea8f7f01-33a1-4b8a-85d4-15ab6689bc27/2024/04/IMG_4309.MOV.mov: No such file or dir ectory immich_microservices | immich_microservices | [Nest] 7 - 04/13/2024, 10:03:58 AM ERROR [JobService] Error: ffprobe exited with code 1 immich_microservices | ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers immich_microservices | built with gcc 12 (Debian 12.2.0-14) immich_microservices | configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=arm64 --cross-prefix=/usr/bin/aarch64-linux-gnu- --toolchain=hardened --enable-cross-compile --enable-rkmpp --enable-rkrga immich_microservices | libavutil 58. 2.100 / 58. 2.100 immich_microservices | libavcodec 60. 3.100 / 60. 3.100 immich_microservices | libavformat 60. 3.100 / 60. 3.100 immich_microservices | libavdevice 60. 1.100 / 60. 1.100 immich_microservices | libavfilter 9. 3.100 / 9. 3.100 immich_microservices | libswscale 7. 1.100 / 7. 1.100 immich_microservices | libswresample 4. 10.100 / 4. 10.100 immich_microservices | libpostproc 57. 1.100 / 57. 1.100 immich_microservices | upload/library/ea8f7f01-33a1-4b8a-85d4-15ab6689bc27/2024/04/IMG_4309.MOV.mov: No such file or directory immich_microservices | immich_microservices | at ChildProcess. (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22) immich_microservices | at ChildProcess.emit (node:events:518:28) immich_microservices | at ChildProcess._handle.onexit (node:internal/child_process:294:12) immich_microservices | [Nest] 7 - 04/13/2024, 10:03:58 AM ERROR [JobService] Object: immich_microservices | { immich_microservices | "id": "5e86106d-a115-4a82-b946-ff3cd2c10cc2", immich_microservices | "source": "upload" immich_microservices | } immich_microservices | immich_microservices | [Nest] 7 - 04/13/2024, 10:04:02 AM LOG [MediaService] Successfully generated JPEG image thumbnail for asset ffb84916-803d-4e25-8ee3-ee8169a479fd immich_microservices | [Nest] 7 - 04/13/2024, 10:04:02 AM LOG [MediaService] Started encoding video 5e86106d-a115-4a82-b946-ff3cd2c10cc2 {"inputOptions":[],"outputOptions":["-c:v h264","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-v verbose","-vf scale=720:-2,format=yuv420p","-preset veryfast","-crf 28"],"twoPass":false} immich_machine_learning | [04/13/24 08:04:02] INFO Setting 'buffalo_l' execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [04/13/24 08:04:02] INFO Loading facial recognition model 'buffalo_l' to immich_machine_learning | memory immich_microservices | [Nest] 7 - 04/13/2024, 10:04:09 AM LOG [MediaService] Successfully generated WEBP image thumbnail for asset ffb84916-803d-4e25-8ee3-ee8169a479fd immich_microservices | [Nest] 7 - 04/13/2024, 10:04:14 AM LOG [MediaService] Successfully encoded 5e86106d-a115-4a82-b946-ff3cd2c10cc2 ```

While the thumbnail and video would show up in the ui eventually, facial recognition was not performed, at least for the video part of the assets that produced this error. What I did was capture a couple of other images that were eventually uploaded first and produced the above error. The Live Photo in question was then processed correctly, meaning the empty face was detected. I assume this error would be a separate issue. Given that it appears sometimes, but not always and only for the first image or so when uploading a couple of images gives me the impression that there is a race condition somewhere..

aviv926 commented 5 months ago

There is a possibility that it was fixed with the merge of #8724, could you upload a sample file so that it can be checked? If you don't want to reveal a face you can take a picture of an ad face or something.

Ok, took me a while to get it right, but I got one that is working. However, I don't know how else to share it but as the .heic + .mov files. Not sure how you would combine them back into a live photo and if the effect persists then. Please let me know if there is another way to properly get this to you.

IMG_4304.zip

I was unable to restore the file back to a live image Neither with ffempg nor with https://github.com/RhetTbull/makelive (it requires mac os)

raisinbear commented 5 months ago

I was unable to restore the file back to a live image Neither with ffempg nor with https://github.com/RhetTbull/makelive (it requires mac os)

Thanks a lot for trying! I’ll just wait for the new release then and see if it was fixed.

aviv926 commented 5 months ago

I was unable to restore the file back to a live image Neither with ffempg nor with https://github.com/RhetTbull/makelive (it requires mac os)

Thanks a lot for trying! I’ll just wait for the new release then and see if it was fixed.

No problem, let me know if it worked for you so we know whether to close this issue