Errors: Skipping missing original photo

[jrmgx]

osxphotos, version 0.58.0 macOS Ventura: 13.2.1 (22D68)

While using the command:

osxphotos export \
    --verbose \
    --export-by-date \
    --download-missing \
    --update \
    --force-update \
    --update-errors \
    --overwrite \
    --exiftool \
    --only-photos \
    --year 2007 \
    --exportdb /Users/jerome/.local/.osxphotos_export.db \
    --db /Users/jerome/Pictures/Photos.photoslibrary \
    "/Volumes/Toshiba Slave/Photos/"

I see plenty of messages like this:

Exporting 2007-04-09_01-57-02.JPG (D504E44B-DF31-45E8-8807-283A67177E16.jpeg) (3900/6130)
Skipping missing original photo 2007-04-09_01-57-02.JPG (D504E44B-DF31-45E8-8807-283A67177E16)
Exporting 2007-09-22_03-44-56.JPG (ADD0C0DA-63BE-4E74-AF89-5D5EF21DD2E6.jpeg) (3901/6130)
Skipping missing original photo 2007-09-22_03-44-56.JPG (ADD0C0DA-63BE-4E74-AF89-5D5EF21DD2E6)
Exporting 2007-02-19_13-51-38.jpg (4D2C793E-802F-49B0-9074-C2B68BE704AC.jpeg) (3902/6130)
Skipping missing original photo 2007-02-19_13-51-38.jpg (4D2C793E-802F-49B0-9074-C2B68BE704AC)
Exporting 2007-04-30_23-17-03.JPG (73AE0E10-14E1-4255-BD5E-C97188BA24E5.jpeg) (3903/6130)
Skipping missing original photo 2007-04-30_23-17-03.JPG (73AE0E10-14E1-4255-BD5E-C97188BA24E5)
Exporting 2007-04-22_12-02-21.JPG (CAC8D8CE-9B48-455C-AFC7-A745058D93F9.jpeg) (3904/6130)
Skipping missing original photo 2007-04-22_12-02-21.JPG (CAC8D8CE-9B48-455C-AFC7-A745058D93F9)
Exporting 2007-09-24_11-19-08.JPG (9AF9CCD3-0AE6-47E0-985B-8149B2219CC9.jpeg) (3905/6130)
Skipping missing original photo 2007-09-24_11-19-08.JPG (9AF9CCD3-0AE6-47E0-985B-8149B2219CC9)
Exporting 2007-08-28_14-08-37.JPG (5A3436CF-3821-4091-A171-0DA372546764.jpeg) (3906/6130)
Skipping missing original photo 2007-08-28_14-08-37.JPG (5A3436CF-3821-4091-A171-0DA372546764)
Exporting 2007-04-07_18-21-08.JPG (789C4D95-F72A-472A-8BAE-2C25CA1414B6.jpeg) (3907/6130)
Skipping missing original photo 2007-04-07_18-21-08.JPG (789C4D95-F72A-472A-8BAE-2C25CA1414B6)
Exporting CIMG1750.JPG (208A41CC-1E62-455F-9AE3-BF73AFF34D37.jpeg) (3908/6130)
Skipping missing original photo CIMG1750.JPG (208A41CC-1E62-455F-9AE3-BF73AFF34D37)
Exporting DSC03018.JPG (4BC47C8B-26F5-46C2-BB70-455649243405.jpeg) (3909/6130)
Skipping missing original photo DSC03018.JPG (4BC47C8B-26F5-46C2-BB70-455649243405)

Saying Skipping missing original photo, I don't understand why?

This is the head of the logs, I guess it contains useful info so I add it:

osxphotos version: 0.58.0
Python version: 3.10.8 (main, Oct 13 2022, 10:17:43) [Clang 14.0.0 (clang-1400.0.29.102)]
Platform: macOS-13.2.1-x86_64-i386-64bit, 13.2.1
Verbose level: 1
exiftool path: /usr/local/bin/exiftool
Warning: export database '/Users/jerome/.local/.osxphotos_export.db' is in a different directory than export destination '/Volumes/Toshiba Slave/Photos'
Using export database /Users/jerome/.local/.osxphotos_export.db
Processing database /Users/jerome/Pictures/Photos.photoslibrary/database/photos.db
Processing database /Users/jerome/Pictures/Photos.photoslibrary/database/Photos.sqlite
Processing database.
Database version: 6000, 8.
Processing persons in photos.
Processing detected faces in photos.
Processing albums.
Processing keywords.
Processing photo details.
Processing import sessions.
Processing additional photo details.
Processing face details.
Processing photo labels.
Processing EXIF details.
Processing computed aesthetic scores.
Processing comments and likes for shared photos.
Processing moments.
Done processing details from Photos library.
Exporting 6130 photos to /Volumes/Toshiba Slave/Photos...

[oPromessa]

Hi @jrmgx

1. RhetTbull will most probably ask you to test with a couple of these pics and check if you're able to Export the original Pics from within the Photos app itself. Can you confirm that you're able to do it?

2. Are your pics on iCloud or on the local library. Or event are they referenced files? I see you use the download-missing so there shouldn't be a problem in case they're on iCloud.

3. Try also the following to dump info on one specific uuid that is failing. Like: 9AF9CCD3-0AE6-47E0-985B-8149B2219CC9...

osxphotos debug-dump --db /Users/jerome/Pictures/Photos.photoslibrary/database/Photos.sqlite --dump photos --uuid 9AF9CCD3-0AE6-47E0-985B-8149B2219CC9 > debug.txt

And share the debug.txt file. DO REMOVE/MASK ANY PRIVATE INFO LIKE GPS LOCATIOND AND/OR ANY NAMES, etc.

[RhetTbull]

Thank you @oPromessa

@jrmgx -- the debug steps @oPromessa recommended will be helpful to figure out what is going on.

Additionally, the --download-missing option requires osxphotos be able to interact with Photos using AppleScript. Your terminal app should have requested permission to do this when you first ran the command. If permission wasn't granted, then the download missing option won't be able to function. Please check System Settings > Privacy & Security > Automation > Terminal and verify that "Photos" is enabled. See screenshot below.

One other step you can try is to re-run the same command but to add --use-photokit to your options. This uses a different code pathway to download the missing photos.

[jrmgx]

I think I have more clues for you. When using the script for the first time I use --download-missing and I have been asked to give permission to control the Photos App, which I accepted (because it's well explained in the docs, kudos for that).

And now, I have retried with --use-photokit and it re-asked me for the permission, weird?

Then I remembered that, when I started the script for the first time, it went quite well for a very long time (about 3 hours with thousands of pictures), it is only when I came back from a break that the problem started to occur.

My theory is, when I went for a break my screen locked, and from that moment it may have lost the ability to control Photos App (or even the whole permission).

Anyway, for now it's working well with photo-kit, and to be honest I prefer this way than having the Photo App started.

When my first full export will be done, I'm willing to take more time to investigate, be it with your instructions and my idea too.

Thank you for your work!

[barnabyc]

Apologies for hijacking but I have 57 "missing original" errors every time I run an --update on a library of 10,343 photos. I've reviewed the suggestions here and ensured Automation permissions are enabled, I've also tested with and without --use-photokit but receive the same results.

Below is a debug-dump output of one of the example missing items, I hope it helps in some way. Thanks for the awesome tool!

debug.txt

[RhetTbull]

@barnabyc The sample debug data you sent indicates this photo is a syndicated photo that has not been saved to the library. That is, it is a photo someone shared with you via Messages or another app but hasn't been saved to the library. In recent versions of macOS these appear in the library in the "shared with you" album. Until these are imported to the library, osxphotos cannot export them.

The clues:

    'syndicated': True,
    'saved_to_library': False,

The preview image for this photo is here in case you want to see which photo it is:

/Users/barnabyc/Pictures/Photos Library.photoslibrary/scopes/syndication/resources/derivatives/masters/F/FF795CE4-0B82-46DF-BF67-4DF05AD7DBEB_4_5005_c.jpeg

It might make sense for osxphotos export to ignore photos that are: 1. missing, 2. syndicated, and 3. not yet saved to the library as these will not be exportable and could cause confusion as in your situation.

You can run a query to find the photos that meet this criteria using:

osxphotos query --missing --syndicated --not-saved-to-library

I'm going to guess the results match the 57 missing photos in your export.

[barnabyc]

Thank you @RhetTbull that does help clarify. The query returned 42 photos so a large portion of the 57. I'll explore query syntax to see if I can find examples of the remaining 15 and post back here. Thanks for the help.