Add support for Projects (cards, calendars, slideshows) (was albums in TopLevelKeepsakes folder are not handled correctly)

[thomashelders]

Hi,

yesterday I was starting to use osxphotos. I am still plying with the options, which are the best for my export. But during execution I see that it is tried for some pictures to not export to the given export directory path, but to a path starting with /TopLevelKeepsakes As this is not existing, the script is complaining.

I am executing as a dry-run: osxphotos export /Volumes/Temp/Bilder_Photo_Export/ --from-date "2011-10-02" --to-date "2011-10-04" --jpeg-ext JPG --convert-to-jpeg --directory "{folder_album}" --exiftool --dry-run --verbose

and no-dry-run: osxphotos export /Volumes/Temp/Bilder_Photo_Export/ --from-date "2011-10-02" --to-date "2011-10-04" --jpeg-ext JPG --convert-to-jpeg --directory "{folder_album}" --exiftool

in the output with the dry-run there are good cases and bad cases:

good case: Exporting IMG_2304.JPG (IMG_2304.JPG) as IMG_2304.JPG Writing metadata with exiftool for /Volumes/Temp/Bilder_Photo_Export/iPhoto-Ereignisse/2011-10-03 Erntemarkt in Hörde und Phönixsee/IMG_2304.JPG Exported /Volumes/Temp/Bilder_Photo_Export/iPhoto-Ereignisse/2011-10-03 Erntemarkt in Hörde und Phönixsee/IMG_2304.JPG

bad case: Exporting IMG_2330.JPG (IMG_2330.JPG) as IMG_2330.JPG Writing metadata with exiftool for /Volumes/Temp/Bilder_Photo_Export/iPhoto-Ereignisse/2011-10-03 Erntemarkt in Hörde und Phönixsee/IMG_2330.JPG Exported /Volumes/Temp/Bilder_Photo_Export/iPhoto-Ereignisse/2011-10-03 Erntemarkt in Hörde und Phönixsee/IMG_2330.JPG Exporting IMG_2330.JPG (IMG_2330.JPG) as IMG_2330.JPG Error exporting photo (o6FalzTsQiC6JSqs7Z+dLw: IMG_2330.JPG) as IMG_2330.JPG: '/TopLevelKeepsakes/Neuer Kalender/IMG_2330.JPG' does not start with '/Volumes/Temp/Bilder_Photo_Export'

I was checking these pictures in the Photos app. But I could not find anything specific for what could be the reason why they are treated differently.

I attach output for a dry-run case and a without dry-run.

Could you please have a look or do you have an idea what might be the reason for this behavior ?

Thanks a lot.

output_error_dryrun.txt output_error.txt

[thomashelders]

I forgot to mention I am executing the script on photo library on a macOS 10.13.6 "High Sierra".

[RhetTbull]

This definitely should not be happening. Please try the following command and post the output here. (This command will produce additional debug data regarding the photo in question):

osxphotos debug-dump --dump photos --uuid 'o6FalzTsQiC6JSqs7Z+dLw' > debug.txt

[thomashelders]

Output file debug.txt attached.

debug.txt

[RhetTbull]

Thanks. I suspect there's an album with a name that's causing issues. Could you try the following command and post the output here? Note: this will include the name and creation date of all albums in your library but no other private information. If there's any information such as an album name you'd prefer not to publicly post, you can search through the text document and delete it.

osxphotos debug-dump --dump albums > album.txt

[thomashelders]

OK, here is the complete album.txt content album.txt

[thomashelders]

I was now trying see how many pictures are affected and wether the special characters ä,ö,ü,ß are related to this. I put the stdout and stderr output to a file test_äöüß.txt. osxphotos --db /Volumes/Seagate\ Portable\ Drive/Fotos\ Library.photoslibrary export /tmp/ --from-date "1970-01-01" --to-date "2021-12-27" --jpeg-ext JPG --convert-to-jpeg --directory "{folder_album}" --exiftool --dry-run --verbose > test_äöüß.txt 2>&1

On this I am using a "grep -B 3" to check for TopLevelKeepsakes and output 3 lines before the match to see the affected album names. cat test_äöüß.txt |grep -B3 TopLevelKeepsakes > test_äöüß_TopLevelKeepsakes.txt

At least I can see that using of ä, ö, ü, ß is not a problem. Pictures in albums without using these characters are also affected.

In total 63 photos do have this behaviour. grep TopLevelKeepsakes test_äöüß.txt|wc -l 63

With the attached files, you have now also the information which of the photos are affected in the database.

test_äöüß_TopLevelKeepsakes.txt test_äöüß.txt

[RhetTbull]

OK, here is the complete album.txt content

Thank-you for the additional data. I believe I've found the problem using the data you provided. There is a "Keepsakes" folder (I believe left over from iPhoto) which is used to store calendars, cards, etc. that were printed from photos in the library. The code doesn't properly account for the presence of albums inside this folder. (It may be called something different in your library due to language localization). There are "Neuer Kalendar" and "Neue Karte" albums (and possibly others) which are part of this top-level Keepsakes folder and they're causing the problem as the logic which constructs the folder/album path doesn't properly account for the presence of the Keepsakes folder. I think I know how to fix this but won't be able to work on this until later in the week.

For now, you could temporarily work around this by finding the albums in the Keepsakes folder and copying all the photos contained in these albums to another "regular" album in the library. osxphotos would still generate the errors for the Keepsakes albums but the photos would get exported correctly as part of the new album.

At least I can see that using of ä, ö, ü, ß is not a problem

That's a logical guess about where the problem could have been and indeed, in the early days, osxphotos did have problems with unicode characters, but I've since tried to be very thorough to properly handle all cases of accented and special characters. (There may still be some bugs but no one has reported issues with unicode in a long time).

[RhetTbull]

Note: This error likely exists with 'TopLevelSlideshows' as well. Check that when implementing fix.

[thomashelders]

Yes, you are right. In 2012 I was ordering one Calendar. At that time it was still iPhoto. This Calendar is still visible. It is in a folder structure at the bottom ("Meine Projekte"). See picture. And this is using many of the pictures which are throwing the TopLevelKeepsakes error.

I made now a copy of the Photo Library to play around. When I delete the "Neuer Kalender" and "Neue Karte" entries, then I do not see the TopLevelKeepsakes entries during export dry-run. Removing "Neuer Kalendar" was not enough, also "Neue Karte" had to be removed, as it had links to some pictures.

[RhetTbull]

I still have a machine running 10.12.6 with iPhoto so I was able to create a test library and recreate this. I've confirmed the problem also exists for slideshows created in iPhoto. The test library will allow me to start working on a fix.

Error exporting photo (lmFQY5k+RYup5Xpox1oEtg: Tulips.jpg) as Tulips.jpg: '/TopLevelKeepsakes/Test Card/Tulips.jpg' is not in the subpath of '/Users/rhet/Desktop/export' OR one path is relative and the other is absolute.
Exporting Tulips.jpg (Tulips.jpg) as Tulips.jpg
Error exporting photo (lmFQY5k+RYup5Xpox1oEtg: Tulips.jpg) as Tulips.jpg: '/TopLevelKeepsakes/Test Calendar/Tulips.jpg' is not in the subpath of '/Users/rhet/Desktop/export' OR one path is relative and the other is absolute.
Exporting Pumkins2.jpg (Pumkins2.jpg) as Pumkins2.jpg
Exported /Users/rhet/Desktop/export/iPhoto Events/Sep 28, 2018/Pumkins2.jpg
Exporting Pumkins2.jpg (Pumkins2.jpg) as Pumkins2.jpg
Exported /Users/rhet/Desktop/export/Pumpkin Farm/Pumkins2.jpg
Exporting Pumkins2.jpg (Pumkins2.jpg) as Pumkins2.jpg
Error exporting photo (FO2q5khAS9y4O9CkjpuYaw: Pumkins2.jpg) as Pumkins2.jpg: '/TopLevelSlideshows/Slideshow1/Pumkins2.jpg' is not in the subpath of '/Users/rhet/Desktop/export' OR one path is relative and the other is absolute.
Exporting Pumpkins3.jpg (Pumpkins3.jpg) as Pumpkins3.jpg
Exported /Users/rhet/Desktop/export/iPhoto Events/Sep 28, 2018/Pumpkins3.jpg
Exporting Pumpkins3.jpg (Pumpkins3.jpg) as Pumpkins3.jpg
Exported /Users/rhet/Desktop/export/Pumpkin Farm/Pumpkins3.jpg
Exporting Pumpkins3.jpg (Pumpkins3.jpg) as Pumpkins3.jpg
Error exporting photo (47wXm7h6RfGRACCdcbLiCA: Pumpkins3.jpg) as Pumpkins3.jpg: '/TopLevelSlideshows/Slideshow1/Pumpkins3.jpg' is not in the subpath of '/Users/rhet/Desktop/export' OR one path is relative and the other is absolute.

[RhetTbull]

@thomashelders I'm curious for any input you have as an osxphotos user on how the fix should be implemented.

1. The simplest solution is to treat any cards/calendars/slideshows as top-level albums and export them as such. E.g. in your example, it would be /Volumes/Temp/Bilder_Photo_Export/Neuer Kalender etc. The downside to this approach is users might be surprised to find these albums in the top-level folder as they don't appear as regular albums in Photos.

2. Another possibility would be to treat "My Projects" as a top-level folder (slideshows, calendars, and cards all appear in My Projects in Photos) and thus in your example, the export path would be /Volumes/Temp/Bilder_Photo_Export/My Projects/Neuer Kalender. The downside to this approach is that osxphotos has no way to access the localized names (for example, Meine Projekte in your case) as this information is not stored directly in the Photos database.

3. A third possibility is to just exclude these "project albums" from the export altogether. Alternatively, there could be a flag to either include or exclude the project albums and use approach 1 or 2 above.

[RhetTbull]

I have #1 above implemented -- was very easy. As a bonus, this now exposes the iPhoto Events albums which had previously been inaccessible. Not sure if people will also want photos exported to iPhoto Events albums.

[thomashelders]

If these iPhoto Event albums are these Calendars and Cards which can be printed, I would say that the use case is not so important, because the content from these iPhoto Event albums are pictures which are stored in or derived from other albums of the database. So it is somehow redundant. Only if somebody wants to know which pictures were used in a specific Calendar or card it might be of use. But this is an edge case, I think.

[RhetTbull]

iPhoto grouper photos automatically into "events" based on date/time. These are similar to Photos "Moments". When converting from iPhoto, Photos creates a folder called "iPhoto Events" and puts all the events as separate albums there. Every photo in an event would also be in an album or in no album and in either case would get exported twice so probably best to hide the events so users don't get duplicates unknowingly

[RhetTbull]

I was wrong about the Events -- they're normal albums created by Photos when migrating from iPhoto. My personal library and test libraries were all created in Photos so I hadn't seen these Events albums before.

[RhetTbull]

For now, I've decided to implement a separate type for projects so they don't get treated by osxphotos as albums. This solves a number of problems and makes the implementation consistent across all versions of Photos (the schema for Photos 5+ is very different in how it handles albums). I'll also expose a {project} template type to provide access to the project name(s) if desired and maybe a --projects-as-albums flags that would instruct osxphotos to include projects in things like {folder_album} by treating them as ordinary albums. This adds some complexity but allows users to include projects if they want or to exclude them easily. As they're not ordinary albums and don't appear in "My albums" in Photos, I think excluding projects by default is the "best" behavior but also provide a way for users to access the projects if desired.

[RhetTbull]

@thomashelders this should be fixed in v0.44.0. By default, the albums in "My Projects" will not be exported; they'll be ignored as if the photos contained in them aren't actually in those albums thus the photos will still be exported but not as part of the project albums. If you want to include them in the export, you can use the new {folder_album_project} template which will treat them like regular albums.

Assuming you installed with pipx, you can do a pipx upgrade osxphotos to upgrade

[thomashelders]

I have tried the new version and is not exporting the pictures without failure. Thanks a lot !!!

One note I have - regarding the created folder structure. Not sure whether you have seen it in your environment, as well. I tried now both options {folder_album} and {folder_album_project}

option{folder_album} is creating: "iPhoto-Event" ->album

The new option is now putting "iPhoto-Event/album"

If you see the attached pictures, you know what I mean.

The dry-run verbose output excerpt is showing a difference in the output - a colon

{folder_album}: Exporting IMG_2330.JPG (IMG_2330.JPG) as IMG_2330.JPG Writing metadata with exiftool for /Volumes/WD Book/Photos_Export/iPhoto-Ereignisse/2011-10-03 Erntemarkt in Hörde und Phönixsee/IMG_2330.JPG Exported /Volumes/WD Book/Photos_Export/iPhoto-Ereignisse/2011-10-03 Erntemarkt in Hörde und Phönixsee/IMG_2330.JPG

=> here no colon between the folder but a / Exported /Volumes/WD Book/Photos_Export/iPhoto-Ereignisse/2011-10-03 Erntemarkt... ^

{folder_album_project}: Exporting IMG_2330.JPG (IMG_2330.JPG) as IMG_2330.JPG Writing metadata with exiftool for /Volumes/WD Book/Photos_Export/iPhoto-Ereignisse:2011-10-03 Erntemarkt in Hörde und Phönixsee/IMG_2330.JPG Exported /Volumes/WD Book/Photos_Export/iPhoto-Ereignisse:2011-10-03 Erntemarkt in Hörde und Phönixsee/IMG_2330.JPG Exporting IMG_2330.JPG (IMG_2330.JPG) as IMG_2330.JPG Writing metadata with exiftool for /Volumes/WD Book/Photos_Export/Neuer Kalender/IMG_2330.JPG Exported /Volumes/WD Book/Photos_Export/Neuer Kalender/IMG_2330.JPG

=> here is a colon between the folders Exported /Volumes/WD Book/Photos_Export/iPhoto-Ereignisse:2011-10-03 Erntemarkt... ^

Can this be the reason for the different folder structure ?

[RhetTbull]

I hadn't seen this but thanks for reporting. This is a bug as the output of both {folder_album} and {folder_album_project} should be the same with exception of adding project albums to the latter. I'll take a look at the code.

[RhetTbull]

Found the problem and fixed in in v0.44.2

[thomashelders]

Works now perfectly. Thanks :-) Happy new year !!!