Albums and Keywords not exporting correctly?

[mlevin77]

I was able to export my iPhoto library without error, and add it to digikam as a Collection. I see the Tags page, but the tags I see are not a good match to the albums and keywords I had. There are tags here for words that appear in photos' descriptions but are not keywords or albums, and most of my albums/keywords are not in these tags. Is it possible to see (a debug mode?) what keywords and albums it's finding during export?

[jensb]

Albums are exported as Tags collections. See README. Smart Albums are not exported except in raw rule format which I cannot read (yet). Keywords are also Tags. Starred, Hidden, etc. photos are also exported as Tags (see README). At the beginning of the output when using DEBUG=3, before the photos are being read, there should be an output list of the found keywords, etc. You can press CTRL-C as soon as it starts exporting photos.

[mlevin77]

hmmm I don't see any of my keywords or albums in the Tags (and the Tags are not collections, they look like a flat list). So I'm trying to convert again and see what it says. But this time, there are very many messages like W, [2018-09-16T06:56:06.407944 #88655] WARN -- : Bad date/time value "0000:00:00 00:00:00": mon out of range and it's scrolling too fast, so I'm afraid I'll miss the messages. What's the right way to capture all the output to more or to a text file so I can see what's going on? I tried ... |& more but bash doesn't seem to like it (and I can't tell if these debug messages are coming on stdout or stderr).

[jensb]

You can use

 DEBUG=3 ruby .... 2>&1 | tee -a iphoto2xmp.log

tee (think of it as a "T" style pipe, dividing a stream into two) saves output into a file and displays it on screen. "2>&1" just means "redirect STDERR(2) to STDOUT(1)".

[mlevin77]

It's running now. So far, I see a bunch of lines like

W, [2018-09-18T21:48:31.926257 #31936] WARN -- : Bad date/time value "0000:00:00 00:00:00": mon out of range

and

W, [2018-09-18T21:52:46.061816 #31936] WARN -- : Badly formed IFD: Infinity

and

One or more files were missing from your iPhoto library! See 'missing.log' in output directory.

at the end, which I didn't see the last time I converted the same library (this was before you fixed the file name extension issue) - any idea what this means?

[jensb]

Hm, strange. Can you make your iPhoto metadata .zip available to me again? I already deleted it ... I'll have another look.

[mlevin77]

Also:

At the beginning of the output when using DEBUG=3, before the photos are being read, there should be an output list of the found keywords this is what I was looking for, to see why my Keywords and Albums didn't make it, but I don't see them in the output. I did as you said:

DEBUG=3 sudo /opt/local/bin/ruby2.5 iphoto2xmp.rb "/Volumes/LaCieScratchDisk/iPhotoLibrary/" "/Volumes/LaCieScratchDisk/test3/" 2>&1 | tee -a iphoto2xmp.log

and the log file is attached - nothing there about keywords or albums. What did I do wrong?

iphoto2xmp.txt

[jensb]

if you use sudo with debug, export it:

 export DEBUG=3
 sudo ruby ...

otherwise the debug output is not created. then re-upload please. zip or bzip2 the file before upload, it will be large.

[mlevin77]

Sorry, I don't understand - I thought I did that:

DEBUG=3 sudo /opt/local/bin/ruby2.5 iphoto2xmp.rb "/Volumes/LaCieScratchDisk/iPhotoLibrary/" "/Volumes/LaCieScratchDisk/test3/" 2>&1 | tee -a iphoto2xmp.log

was I supposed to do it differently? I attached the log file to the previous post, and emailed you the zipped metadata directly to your email - is that what you need, or are you talking about sending the whole exported library (which is like 300 GB)?

[jensb]

No, you didn't. :-)

The current problem has nothing to do with photo2xmp, it is a generic shell issue. If you set environment variables as a prefix to a command they will be only valid that that command, not for any child processes. So if you use DEBUG=3 ruby iphoto2xmp.rb ...., Ruby will see the DEBUG variable, which is what we need. But if you use DEBUG=3 **sudo** ruby iphoto2xmp.rb ..., only sudo will see the DEBUG variable, which doesn't help. To make ruby inside sudo also see the DEBUG variable, you need to "export" it (make it permanent), in a separate command before executing sudo:

 export DEBUG=3
 sudo ruby iphoto2xmp.rb ....

Does this help?

[mlevin77]

aha. Ok. I did export DEBUG=3 but the output looked exactly the same as before - didn't seem like it was working even though printenv shows that the shell knows DEBUG=3. So, I re-ran it with DEBUG=3 ruby ... (no sudo) and now I'm seeing the verbose debug output. I'm not sure why I had to do sudo before and now it works without it, but it seems to be outputting a ton of stuff. I will upload the log file as soon as it's done.

[jensb]

I just tested it with your metadata. When digikam has imported all the files (scanned the directories) you should be able to see all tags, and a tag hierarchy "TopLevelAlbums" which contains tags for your iPhoto albums, and some more "TopLevel..." tag hierarchies which contain tags for iPhoto hidden, starred, etc. photos. However, this might take some time, you'll see a progress bar at the bottom of the screen.

Assuming the script output was piped to iphoto2xmp.log: You can use grep Tags iphoto2xmp.log to see all tags that were found, grep albumdata iphoto2xmp.log to see all albums, and grep folderlist iphoto2xmp.log to see all folders, etc. Folders should be visible as tag hierarchy parts in Digikam, e.g. "TopLevelAlbums/MyFolder/MyAlbum". Almost all freely editable image properties and some special flags are recreated as generic tags, because this is the simplest and most unversal way to do it.

[mlevin77]

Here's what I see in Digikam - no TopLevelAlbums.

when I do the grep, here's what I see:

% grep Tags iphoto2xmp.txt | head Tags: Historical Family(18) Tags: Historical Family(18), Myself(51) Tags: Historical Family(18) Tags: Historical Family(18), Myself(51) AlbumTags: ["TopLevelAlbums/Present DVD for Jay and Leonard"] Tags: Historical Family(18), Myself(51) Tags: Historical Family(18) Tags: Historical Family(18) Tags: Historical Family(18)

% grep Albumdata iphoto2xmp.txt Albumdata: Best ones, Nostalgic, Dima, JK, Historic family photos, Jay, Lee, Asya, Best photos of both kids,...

so, yes some Tags seem to be in there (although they don't make it into DigiKam tags) and yes the keywords seem to appear in Album data,but they don't make it to Digikam Tags list.

[jensb]

Did you ever solve this, can I close this issue?

[mlevin77]

Hmm no, but now that DigiKam is a much newer version, I suppose I should try it again and see. As soon as the conversion works, I'll try it!

-----Original Message----- From: jensb @.> Reply-To: jensb/iphoto2xmp @.> Date: Sunday, September 24, 2023 at 3:50 PM To: jensb/iphoto2xmp @.> Cc: Michael Levin @.>, Author @.***> Subject: [External] Re: [jensb/iphoto2xmp] Albums and Keywords not exporting correctly? (#18)

Did you ever solve this, can I close this issue?
—
Reply to this email directly, view it on GitHub <https://github.com/jensb/iphoto2xmp/issues/18#issuecomment-1732654697>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIS7OAR5TU7X7H6POIAAINDX4CFGJANCNFSM4FVJH2WA>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
[
{
***@***.***": "http://schema.org",
***@***.***": "EmailMessage",
"potentialAction": {
***@***.***": "ViewAction",
"target": "https://github.com/jensb/iphoto2xmp/issues/18#issuecomment-1732654697",
"url": "https://github.com/jensb/iphoto2xmp/issues/18#issuecomment-1732654697",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
***@***.***": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
Caution: This message originated from outside of the Tufts University organization. Please exercise caution when clicking links or opening attachments. When in doubt, email the TTS Service Desk at ***@***.*** or call them directly at 617-627-3376. 

[mlevin77]

Hi, so I tried your new version with DEBU3 and I saw:

DEBUG=3; it converts, and at some point I see

… ============W, [2023-09-24T06:49:29.326007 #80827] WARN -- : Bad date/time value "0000:00:00 00:00:00": mon out of range W, [2023-09-24T06:49:29.326055 #80827] WARN -- : Bad date/time value "0000:00:00 00:00:00": mon out of range W, [2023-09-24T06:49:29.326067 #80827] WARN -- : Bad date/time value "0000:00:00 00:00:00": mon out of range ================W, [2023-09-24T06:49:46.861895 #80827] WARN -- : malformed GPS degrees: expected [degrees, minutes, seconds]; got [NaN, NaN, NaN] W, [2023-09-24T06:49:46.861924 #80827] WARN -- : malformed GPS degrees: expected [degrees, minutes, seconds]; got [NaN, NaN, NaN] W, [2023-09-24T06:49:46.864357 #80827] WARN -- : malformed GPS degrees: expected [degrees, minutes, seconds]; got [NaN, NaN, NaN] W, [2023-09-24T06:49:46.864383 #80827] WARN -- : malformed GPS degrees: expected [degrees, minutes, seconds]; got [NaN, NaN, NaN] W, [2023-09-24T06:49:46.882677 #80827] WARN -- : malformed GPS degrees: expected [degrees, minutes, seconds]; got [NaN, NaN, NaN] W, [2023-09-24T06:49:46.882706 #80827] WARN -- : malformed GPS degrees: expected [degrees, minutes, seconds]; got [NaN, NaN, NaN] =====W, [2023-09-24T06:49:50.793211 #80827] WARN -- : Badly formed IFD: Infinity W, [2023-09-24T06:49:50.796622 #80827] WARN -- : Badly formed IFD: Infinity W, [2023-09-24T06:49:50.797049 #80827] WARN -- : Badly formed IFD: Infinity ===================================================================================================================/Library/Ruby/Gems/2.6.0/gems/exifr-1.4.0/lib/exifr/jpeg.rb:102:in examine': no start of image marker found (EXIFR::MalformedJPEG) from /Library/Ruby/Gems/2.6.0/gems/exifr-1.4.0/lib/exifr/jpeg.rb:34:in block in initialize’ from /Library/Ruby/Gems/2.6.0/gems/exifr-1.4.0/lib/exifr/jpeg.rb:34:in open' from /Library/Ruby/Gems/2.6.0/gems/exifr-1.4.0/lib/exifr/jpeg.rb:34:in initialize’ from iphoto2xmp.rb:579:in new' from iphoto2xmp.rb:579:in block in ‘ from iphoto2xmp.rb:438:in each' from iphoto2xmp.rb:438:in ‘

 So then I tried modifying the code to just show me each file it was 

working on, by adding 1 line in the beginning of the main loop:

masters.each do |photo| debug 1, "Started to work on: #{photo['id']}. #{photo['imagepath’]}”

this one

 I thought it wouldn’t change anything, but now instead of crashing, 

it goes to the end, but gives me a huge list of "One or more files were missing from your iPhoto library! See 'missing.log' in output directory.”

Any idea why this simple debug statement would change whether it crashes or not? And, I do see these photos in the original iPhoto library, so why can’t it find it?

Thank you!!!

------ Original Message ------ From "jensb" @.> To "jensb/iphoto2xmp" @.> Cc "Levin, Michael" @.>; "Author" @.> Date 9/24/23, 3:49:56 PM Subject [External] Re: [jensb/iphoto2xmp] Albums and Keywords not exporting correctly? (#18)

Did you ever solve this, can I close this issue?

— Reply to this email directly, view it on GitHub https://github.com/jensb/iphoto2xmp/issues/18#issuecomment-1732654697, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIS7OAR5TU7X7H6POIAAINDX4CFGJANCNFSM4FVJH2WA. You are receiving this because you authored the thread.Message ID: @.***>

Caution: This message originated from outside of the Tufts University organization. Please exercise caution when clicking links or opening attachments. When in doubt, email the TTS Service Desk at @.*** or call them directly at 617-627-3376.

[RhetTbull]

@mlevin77 FYI, I'm working on a port of iphoto2xmp to osxphotos so that osxphotos can export iPhoto libraries. I'm about 60% done. Will likely need another week or so to finish (I do this part time as hobby) but when done, that might work for you.

[mlevin77]

Amazing!! Very hopeful for this.

-----Original Message----- From: Rhet Turnbull @.> Reply-To: jensb/iphoto2xmp @.> Date: Wednesday, September 27, 2023 at 1:12 PM To: jensb/iphoto2xmp @.> Cc: Michael Levin @.>, Mention @.***> Subject: [External] Re: [jensb/iphoto2xmp] Albums and Keywords not exporting correctly? (#18)

@mlevin77 <https://github.com/mlevin77> FYI, I'm working on a port of iphoto2xmp to osxphotos so that osxphotos can export iPhoto libraries. I'm about 60% done. Will likely need another week or so to finish (I do this part time as hobby) but when done, that might work for you.
—
Reply to this email directly, view it on GitHub <https://github.com/jensb/iphoto2xmp/issues/18#issuecomment-1737787112>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIS7OAUXH74AAGFMNSMICNTX4RM7NANCNFSM4FVJH2WA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
[
{
***@***.***": "http://schema.org",
***@***.***": "EmailMessage",
"potentialAction": {
***@***.***": "ViewAction",
"target": "https://github.com/jensb/iphoto2xmp/issues/18#issuecomment-1737787112",
"url": "https://github.com/jensb/iphoto2xmp/issues/18#issuecomment-1737787112",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
***@***.***": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
Caution: This message originated from outside of the Tufts University organization. Please exercise caution when clicking links or opening attachments. When in doubt, email the TTS Service Desk at ***@***.*** or call them directly at 617-627-3376. 

[jensb]

@RhetTbull that's great, I'm happy iphoto2xmp is useful to people! If you need any support or help reading my code, just ask. I admit it wasn't created at first with the intention of publishing it anywhere :-)

[RhetTbull]

@jensb thanks! I've got all the queries working and added a few fields that osxphotos needs that iphoto2xmp wasn't pulling. I've got a bit more tooling to do to make the internal iPhoto database object plug & play compatible with the Photos database object that osxphotos uses then I'll run it through the exporter code and see what breaks. Will let you know if I have any questions! The face rectangles transform part is where I suspect I'll run into issues as your code is very different than how I did it in osxphotos.