search

ente-io / ente

Fully open source, End to End Encrypted alternative to Google Photos and Apple Photos
https://ente.io
GNU Affero General Public License v3.0
16.01k stars 823 forks source link

Machine learning indexing crawls to nearly stopped on Android #2877

Closed AntsyBoi closed 1 month ago

AntsyBoi commented 2 months ago

Description

On latest GrapheneOS patch level for Android 14 on a Pixel 7, the indexing goes pretty fast for the first 700 photos or so (I only have around 750 photos and videos backed up in ente) but for the last bit, especially the last 2 the processing rate slows way down. The 2nd to last file took around an hour to process and the last file has taken over 2 hours so far and has not completed. I don't have any huge photos or videos stored in ente, just regular photos taken from my phone camera and the occasional short video. I don't believe I have anything backed up that should be causing it to take so long to process especially considering I could've reprocessed all the other photos in the time it is currently taking to process the last one.

Version

v0.9.27

What product are you using?

Ente Photos

What platform are you using?

Mobile - Android

laurenspriem commented 2 months ago

Thanks for reporting! Could you please send logs to laurens@ente.io? That gives me a bit more information on what's happening.

One possible cause I can think of is that the last 2 files have a rare format that's hard for the app to decode.

AntsyBoi commented 2 months ago

I sent the logs as of yesterday and since then, the app now says I have 0 photos remaining but when I go into the search section of the app where's its meant to display the faces from the photos it's indexed it still says "People will be shown here once indexing is done". This feels like it's resulting from the same underlying issue so I'm not sure if i should create a new issue as I'm not sure if it ever actually finished the last photo as it didn't finish processing the photo after having the whole previous day to work on it but it now says its done after one reboot and a short amount of time

laurenspriem commented 2 months ago

Thanks for sending the logs! I took a look at them, and it seems that the issue was that the app was not able to download these last two files in order to analyze them. Hard to say what caused it exactly, but eventually it seems to have obtained it, so aside from the wait it doesn't seem like a major issue.

Not showing anything in the search section is altogether a different problem. One important detail is that after indexing the app first has to cluster results before any faces are shown. So most likely the reason no faces were shown was because clustering was still running. If you still don't have any faces, could you leave the app open on the ML settings page (the one with the indexing status)? It should display there when clustering is running.

AntsyBoi commented 2 months ago

The app doesn't seem to say anything about clustering on the machine learning page, and it still isn't displaying any faces and also saying that I'll have to wait for indexing to finish to see them. Screenshot_20240827-054359.png Should I refile this under a new issue? Thanks for all your help on this!

laurenspriem commented 2 months ago

No need to file a new issue, let's just use this one.

Very strange behavior indeed. Thinking it through more, I don't think the issue is with clustering itself.

To check, could you look at some of your images with people in them, go into file info (bottom left icon) and check if it displays the faces there? And if faces are there, could you go press on any familiar ones and see if they take you to a grouping?

AntsyBoi commented 2 months ago

So far, any photo that I've found that face groupings only ever have 1 other photo in them if any and it's usually only if there's 2 photos taken from like the exact same angle in the same setting as eachother. There are also a lot of cases where even when a face is very clearly present in the photo (taking up a good chunk of the frame, not obscured, facing the camera) where it doesn't even recognize the face at all saying there's no faces found in the photo. When I manually name faces and group them together they do show up in the faces menu and the message about waiting for indexing to be completed disappears but then it still only shows the faces that I manually named.

laurenspriem commented 2 months ago

Thanks for the information, and for your patience in debugging the issue!

Based on your report and the logs, we think a considerable amount of photos in your library were not decoded correctly during indexing and hence were not properly analyzed. We've updated our logging to better detect this next time. To help further debug this issue, it would be great if you could help us with the following two things.

Firstly, I have a bunch of file IDs from your library that I suspect (from the logs) were not decoded correctly. Ente being encrypted I don't have any more information to work out why it failed for these files. If you have the time, could you use the latest version of our desktop app and search for these file IDs? The desktop app should show the file when you enter the ID in the search bar. Best thing for us would be if you could share those images with us via an Ente Album link you can email me, but if you don't want to share the images (totally fair) it would already help a lot if you could send me information about them (file name, format, size, EXIF).

Here are the file IDs:

Secondly, we will release a new version of the mobile app with better logging. I'll provide a dev option in the app for you to locally reset your ML data, and then it would be great if you could try to re-index and afterwards send logs. I'll ping you once the release is out!

AntsyBoi commented 2 months ago

I sent in a photo album with as many of the photos as I could submit. My guess (included in the email) for what caused the decoding errors was due to the photos being saved as RAW from my phone's camera in the .dng format. The camera app I use is just the regular Pixel Camera app and they were saved just by telling the app to save RAW copies of the photos alongside the regular photos.

laurenspriem commented 2 months ago

Indeed that seems to be the issue. Dng format is not supported and hence decoding for ML is failing. We're working on a potential fix for the decoding where we first convert them to jpg. I will let you know once that fix is in the next release!

One other thing: I am curious how files are counted with this many raw formats. When you long press on an image from the gallery view, and then press "All", how many files does it say are selected in the bottom left corner?

AntsyBoi commented 2 months ago

Just says 4 like if they were regular photos, no weird behavior seems to be happening there. The photos properly preview and all that in the app too. When I click on them they go sideways but that's a whole other issue with them not being supported. For a non-supported filetype they do seem to work pretty well though.

laurenspriem commented 2 months ago

Got it, thanks! Unfortunately until we support these RAW formats I cannot guarantee that they will play nicely with the ML features. However, let's see if the proposed fix works for you.

It will go out in a release this week, will ping you here when that happens!

AntsyBoi commented 2 months ago

Thank you! This is great to hear you guys do great work!

laurenspriem commented 2 months ago

@AntsyBoi The release is out on Play Store, Github and F-droid following soon (not sure which you're on).

Once you have updated to version 0.9.35, you could try the following:

  1. Go to ML settings page (Settings > General > Advanced > Machine Learning)
  2. Open a special ML debug options page by pressing the "Machine learning" title 7 times
  3. On this new page, press "Purge empty indices"
  4. Wait for indexing to complete automatically

What this does is it deletes all results where decoding has failed, and will automatically retry them using the new fix that might actually decode this time.

After these four steps, please check if it has improved your situation :)

AntsyBoi commented 2 months ago

After purging empty indices and also resetting all local ML, the photos do process faster but the faces section still says that I have to wait for them to be done indexing for them to show up there, which they all are. All 875 photos are shown as complete in the Machine Learning settings.

laurenspriem commented 1 month ago

Unfortunately that means the potential fix for ML on raw images did not work, and instead we will have to wait for full RAW support.

In the meantime all I can do is fix the incorrect "waiting for indexing to complete" UI. I will probably add a check for this case, on which the People section won't be surfaced at all.

AntsyBoi commented 1 month ago

Would it be possible to just skip RAW photos so then it can do the indexing as normal? When I take a RAW photo it also takes a regular photo alongside it and so if I were to just have all the RAW copies skipped, I wouldnt really be missing any photos, and this would be much favorable app behaviour over just all facial recognition and other machine learning just fail. There could be a disclaimer or something, maybe a "skipped photos" counter for unsupported files types or something.

laurenspriem commented 1 month ago

There are a lot of things we need to do to handle RAW photos better, all of which will come when we offer official RAW support.

That said, I suspect the actual issue is that your image library is too small for the clustering in ML to work well. You stated 750 files, but given the RAW files this means only 375 normal, unique photos. Right now we only surface big clusters on the UI, and if no big clusters are found we display "waiting for indexing to complete". Which for most users is correct, but for the ones with a small library indexing might already be done.

We will change the UI to better handle this case, and ideally display smaller clusters. For now I am closing this issue since the original complaint is resolved. Feel free to re-open if you disagree.

AntsyBoi commented 1 month ago

Only 23 of my 973 photos backed up to Ente are in the raw .dng format. Meaning I have 950 photos that it should be able to use to cluster things together. How many photos would it take to actually cluster the photos together?

laurenspriem commented 1 month ago

Well it actually depends on how many people are in your photos, and even more specifically how often the same people are in your photos. 950 photos should actually be enough, assuming those are not all scenery photos and you have an average amount of (mostly the same) people in them.

Are you sure you have 950 photos in standard formats (jpg, png)?

Could you try the following:

  1. Go to ML settings page (Settings > General > Advanced > Machine Learning)
  2. Open the special ML debug options page by pressing the "Machine learning" title 7 times
  3. On this new page, press "Purge empty indices"

When you press "purge empty indices" it should give a message saying how many empty results were purged. What is that number? Or stated differently, how many items are pending again in the Machine Learning screen after doing this?

AntsyBoi commented 1 month ago

It says "Deleted 0 items"

About half of my photos on Ente have faces in them, so I doubt that it's too few for it to catalogue them.

The majority of the photos with faces in them are of either my brother or a few friends, I could pick out probably 50 photos that just have one of my friends face in them. Is that enough to catalogue?

laurenspriem commented 1 month ago

Yes 50 photos of the same person should definitely be more than enough.

In all of these image, what does it show in the file info? Below the EXIF section, does it show the detected faces?

AntsyBoi commented 1 month ago

It shows their face but it doesn't group it with many other photos of their face, the only ones it can find are ones that are taken from the exact same angle within seconds of eachother. A lot of times the groupings don't even get all the photos of them from one day. Often times requiring me to manually group faces together for them to be all in one collection.

My guess for why it's still saying it then is just that it can't find enough of one person to put it in the faces area, which there are enough photos of multiple people to be able to get their face shown in there, it's just that it doesn't group all those photos together. It feels like it needs to do another set of linking where it connects faces to ones already grouped and then group those that are close enough without manual user intervention.

laurenspriem commented 1 month ago

This is very strange behavior. It sounds like the faces get analyzed fine, but then something goes wrong with clustering. Either clustering is not running at all, or the face embeddings it works with are corrupted somehow.

We have not seen this issue anywhere else, so I'm really not sure what the cause is. So to properly debug this, I will have to make a few changes to the app (more logging, debug options, etc.). Once that is done and gone out with the next release I will ping here again and ask you to re-index.

Hopefully at that point we'll be able to find out from the logs what is happening. Until then, sorry for the wait and thank you so much for your patience while we sort this out!