search

RhetTbull / osxphotos

Python app to work with pictures and associated metadata from Apple Photos on macOS. Also includes a package to provide programmatic access to the Photos library, pictures, and metadata.
MIT License
2.17k stars 100 forks source link

Advice on dealing with high number of export errors with --download-missing flag #555

Open hyfen opened 2 years ago

hyfen commented 2 years ago

I tried to do a full export today (about 40k items) with the --download-missing option and a few thousand failed. A typical error: 

Error exporting photo (12CB438D-D650-4C4A-92CE-1C2898697E39: IMG_8367.JPG) as /Users/alouis/Pictures/Phone/2015/03/17/IMG_8367.JPG: Could not export photo 12CB438D-D650-4C4A-92CE-1C2898697E39 (_photoinfo_export.py: 1343)

This suggests it's the AppleScript export that's failing. I tried using --retry (hence #553) and that didn't seem to improve things. I did notice that opening a failed photo in Photos.app often leads to a successful export the next time. Also, using --uuid and running the same command for just one particular failed photo has been successful each time I tried. I'm using --skip-original-if-edited and it seems there's a higher chance of failure for images that have been edited (usually just basic stuff like rotations).

Could this be fixed with a custom --timeout flag for the CLI? Is there anything else I should be trying for a successful large export?

Two ideas I haven't tried yet:

  1. Capture the output of the export, parse out the UUIDs of the failures, and run osxphotos --uuid for each one.
  2. Navigate through every photo in Photos.app to somehow trigger local caching from iCloud.

The full command I've been using:

osxphotos export ~/Pictures/Phone --from-date "2010-05-20" --to-date "2021-12-15" --not-shared --export-by-date --exiftool --download-missing --finder-tag-template "{label}" --xattr-template comment "{detected_text:0.8}" --xattr-template participants "{person}" --xattr-template version "{uuid}" --update --ignore-signature --skip-bursts --skip-live --skip-original-if-edited --overwrite --touch-file
RhetTbull commented 2 years ago

This could be a problem with AppleScript not running at all (maybe due to security permissions). Here's a way to test that:

Select a photo in Photos.

Type osxphotos repl in Terminal

Then in the REPL, try:

>>> export = selected[0].export2(dest="/tmp",use_photos_export=True)
>>> str(export)
"ExportResults(exported=['/tmp/IMG_7077 (1).png'],new=[],updated=[],skipped=[],exif_updated=[],touched=[],converted_to_jpeg=[],sidecar_json_written=[],sidecar_json_skipped=[],sidecar_exiftool_written=[],sidecar_exiftool_skipped=[],sidecar_xmp_written=[],sidecar_xmp_skipped=[],missing=[],error=[],exiftool_warning=[],exiftool_error=[],deleted_files=[],deleted_directories=[],exported_album=[],skipped_album=[],missing_album=[])"

You should see the file got exported to /tmp and that errors=[] (no errors).

Also, you can try to add --use-photokit flag to your command line which will use a native PhotoKit framework call instead of AppleScript to do the export of missing files. This is usually more reliable but I haven't made it the default because it sometimes causes more permissions issues with System Integrity Protection (SIP).

RhetTbull commented 2 years ago

Could this be fixed with a custom --timeout flag for the CLI?

The AppleScript code has a 120 second timeout -- maybe that's enough to download large edits (but seems unlikely) and it does retry 3 times automatically. If we can determine AppleScript is timing out I'll either increase the timeout out add a user option. I suspect it's a SIP issue and AppleScript isn't even running. I also suspect --use-photokit will fix these errors.

hyfen commented 2 years ago

You should see the file got exported to /tmp and that errors=[] (no errors).

I got errors (Could not export photo) for each random one I tried. Ok, so it's not a timeout issue.

Thanks for pointing out the --use-photokit option. It seems to have filled in most images that errored. I did see this a few times: Could not get authorizaton to use Photos: auth_status = 0 which is strange because after authorizing Terminal.app to access Photos the first time, it should stay authorized, no?

RhetTbull commented 2 years ago

I got errors (Could not export photo) for each random one I tried. Ok, so it's not a timeout issue.

Sounds like a problem calling AppleScript. It may have to do with a permission issue -- will see if I can devise a test to get some more granular data on the error. Glad that --use-photokit seems to be working.

I did see this a few times: Could not get authorizaton to use Photos: auth_status = 0

That is weird -- you should only have to authorize the calling app (e.g. Terminal) once. I'll take a look and see if anything jumps out.