after applying group_modified.sql, pictures disappear from albums

[erkcan]

Applying the group_modified.sql postmortem causes some albums to loose items. For example an album with 148 items in it, started to show only 4 of them afterwards.

Running sqlite on my ~/Pictures/digikam4.db, and dumping the list of all Images using select * from Images;, I can see that they are still all there (as expected).

I scanned for one of the missing images in the Images table:

select * from Images where name='104_0436_IMG.JPG';
2273|130|104_0436_IMG.JPG|1|1|2003-01-12T04:33:42|875718|0063e8523c695e070dfc05206ac85a2e

Then I looked for 2273 in the ImageRelations table:

select * from ImageRelations where subject=2273 or object=2273;
2273|2273|1
2273|2273|2

I don't know what this means, or if it helps to debug the problem.

[erkcan]

I should add that I am running Digikam 4.13 (latest version available as an OSX package at kde.org), and the above mentioned missing photo does not have a modified version.

One more possible clue: The missing items are from an album with merged iPhoto events. Two sets of photos had been imported separately, but then merged into a single event with iPhoto, and the photos in the second set are all missing.

And one more possible clue: If I remember correctly, I might have imported the missing photos into iPhoto one time, and then trashed them, and reimported them again.

[erkcan]

The following observation might or might not be relevant, but here is one more interesting fact: My missing.log has lots of items. This had not bothered me before, because I had checked that all my photos had been properly handled by iphoto2xmp.

Now, when I grep for the above mentioned item in my missing.log, its "Previews" version is indeed listed:

$ grep 104_0436_IMG.JPG missing.log 
/Users/erkcan/Pictures/iPhoto Library/Previews/2016/04/22/20160422-220905/pwoutEqsS2q1B3kJBDBPXw/104_0436_IMG.JPG

It is worth noting that iphoto2xmp is correct in saying that no file at the listed path exists. But this is not an indication of a problem, that file does not exist as the 104_0436_IMG.JPG is present only in its original form in the relevant Masters directory.

[jensb]

I have experienced this too. Thank you for your analysis.

The problem with group_modified.sql is that (currently) Digikam does not use the UUID (universally unique identifier) that I create in the XMP files to populate the database. Thus when creating the postmortem scripts, I can only access the right files by event (= folder/album) and file name. This isn't perfect since iPhoto allows duplicate file names (titles) in the same event, but it works 99%. So in your case my group_modified.sql might reference the wrong image since you surely have duplicate file names in your iPhoto events. Tracking that would be a lot of additional work but I guess it'll have to be done.

So after each import run I create new Digikam DB entries and relations for each image, which Digikam (apparently) doesn't like in the long run. The perfect solution would be a cleanup command which purges Digikam's database of any references to nonexisting image files - but I haven't dared to do that yet because I don't really know what internal cleanup routines Digikam uses itself.

[erkcan]

I have very few edited versions of my photos; I don't like spending extra disk space, I only keep the originals. So as a temporary work around, I decided not to apply group_modified.sqlat all. Instead, here is what I did:

1. Go to the Fuzzy tab in digikam and create fingerprints of my photos.
2. After increasing the similarity to 99%, I did "Find duplicates".
3. That gave me the list of my edited files, and for each pair of copies of every photo, I manually did "Group Selected".

I assume this procedure could actually be scripted to be run from the digikam. I will look at the digikam api for this purpose.

[jensb]

This works too. Since Digikam 5.6 (I think) there is a database cleanup option in the Digikam menu, so this issue will not appear any more with current Digikam versions.

Since my script is supposed to be run once (to export iPhoto data) and not incrementally except for testing, I think we can close this. Reopen if you disagree.