Error running AppleScript with --download-missing

[narensankar0529]

Trying to export from library only photos that are not in shared albums. But using --not-shared with export seems to export all media include both non-shared and shared. Using osxphotos on Catalina with Photos 5.0.

Is this a known issue and is there a way to export photos only from non-shared albums? Or from Smart Albums?

[RhetTbull]

This isn't a known issue. You're correct that --not-shared should exclude shared albums. I'll take a look and see if I can replicate this. Could you provide the full command line you're using with osxphotos?

[narensankar0529]

See the commands below -

There are 43501 non shared pictures in the library. But I get total exported as 57733

silvialin@ns-MacBookPro16 Pictures % osxphotos query --not-hidden --not-shared --only-photos |wc -l Using last opened Photos library: /Users/silvialin/Pictures/Photos Library.photoslibrary 43501 silvialin@ns-MacBookPro16 Pictures % osxphotos export --dry-run --not-hidden --not-shared --only-photos . Using last opened Photos library: /Users/silvialin/Pictures/Photos Library.photoslibrary Exporting 43519 photos to .... [####################################] 100% Exported: 57733 photos Elapsed time: 60.435 seconds silvialin@ns-MacBookPro16 Pictures % osxphotos query --not-hidden --shared --only-photos |wc -l Using last opened Photos library: /Users/silvialin/Pictures/Photos Library.photoslibrary 21156

[narensankar0529]

Log including photos and videos -

Instead of 49921 total items almost 62743 items are exported

silvialin@ns-MacBookPro16 Fotos1 % osxphotos export --update --not-shared --skip-bursts --current-name --export-as-hardlink --skip-original-if-edited --directory '{created.year}/{created.month}' /Users/silvialin/Pictures/Fotos1 Using last opened Photos library: /Users/silvialin/Pictures/Photos Library.photoslibrary Exporting 49921 photos to /Users/silvialin/Pictures/Fotos1... [####################################] 100% Exported: 62742 photos, updated: 0 photos, skipped: 0 photos, updated EXIF data: 0 photos Elapsed time: 304.681 seconds silvialin@ns-MacBookPro16 Fotos1 % find . -type f | wc -l 62743

[RhetTbull]

@narensankar0529 It's not really clear in the documentation but I don't think this is exporting shared images. The "Exported" count is often different than the initially estimated "Exporting X photos" count. This is because a single photo in the library could actually have more than one image file associated with it: original, edited version, bursts, RAW+jpeg complement, live photo. Since you've used --skip-bursts and --skip-original-if-edited you'll exclude those. Do you have a lot of RAW images and/or live images? (These have an associated .mov file for the "live" part) Try again with --skip-live and --skip-raw.

I'll work on the documentation that gets printed: e.g. maybe instead of: Exported: 62742 photos, updated: 0 photos, skipped: 0 photos, updated EXIF data: 0 photos

Something like: Exported: 49921 photos (62742 files), updated: 0 photos, skipped: 0 photos, updated EXIF data: 0 photos

UPDATE: By the way, the "Exported: x" count could also be less because some photos might be missing from the library -- e.g. they're in the database but not physically present on disk because they've not been downloaded from iCloud or they weren't copied to the library and the path can't be found.

Thoughts?

[RhetTbull]

You could test to verify this isn't exporting shared images by exporting a small date range for which you know you have shared images and non-shared images then look at the output. I did this just now and confirmed no shared images were exported.

  --from-date DATETIME            Search by start item date, e.g.
                                  2000-01-12T12:00:00,
                                  2001-01-12T12:00:00-07:00, or 2000-12-31
                                  (ISO 8601).
  --to-date DATETIME              Search by end item date, e.g.
                                  2000-01-12T12:00:00,
                                  2001-01-12T12:00:00-07:00, or 2000-12-31
                                  (ISO 8601).

[narensankar0529]

Ok. So it does seem shared are not being exported. But I am still confused on the number of exports since they don't seem to add up. So it would be great if the output actually indicated what were the various media types exported and their counts -

The photos app says I have 43500 pictures and 6421 videos. osxphotos says it is exporting 49921 as expected. but then the final counts do not match up and I am not sure what is missing or not exported. There are 106 items that I don't know what they are and are not exported. And it is very difficult to track this down. Is it possible to print details similar to the media types Photos uses? like video, slo-mo, screenshots etc to know what was actually exported?

silvialin@ns-MacBookPro16 Fotos1 % osxphotos export --update --not-shared --skip-bursts --current-name --export-as-hardlink --skip-live --skip-original-if-edited --directory '{created.year}/{created.month}' /Users/silvialin/Pictures/Fotos1 Using last opened Photos library: /Users/silvialin/Pictures/Photos Library.photoslibrary Exporting 49921 photos to /Users/silvialin/Pictures/Fotos1... [####################################] 100% Exported: 49815 photos, updated: 0 photos, skipped: 0 photos, updated EXIF data: 0 photos Elapsed time: 285.754 seconds silvialin@ns-MacBookPro16 Fotos1 % osxphotos export --update --not-shared --skip-bursts --current-name --export-as-hardlink --skip-live --directory '{created.year}/{created.month}' /Users/silvialin/Pictures/Fotos1 Using last opened Photos library: /Users/silvialin/Pictures/Photos Library.photoslibrary Exporting 49921 photos to /Users/silvialin/Pictures/Fotos1... [####################################] 100% Exported: 1362 photos, updated: 0 photos, skipped: 49815 photos, updated EXIF data: 0 photos

[RhetTbull]

I'll rename this issue and keep it open to add better description of what actually got exported. That'll take a bit of work as the code doesn't know ahead of time what is actually going to get exported, just the total number of photos to try to export.

You can run any of the export commands with --verbose or -V and it'll print name of every file exported or skipped. In your case the missing photos may be ones that aren't in the local library but are on iCloud. These can be downloaded with --download-missing

Note: --verbose prints a lot of info so you might want to redirect to a file, e.g. using >export.txt

[narensankar0529]

I have the library set to download all originals so in theory there should be nothing missing. And the “PhotosExporter” swift based app does export the right number.

But I would like to script the exports and so trying to get osxphotos to work. On Nov 6, 2020, 11:41 PM +0800, Rhet Turnbull notifications@github.com, wrote:

I'll rename this issue and keep it open to add better description of what actually got exported. That'll take a bit of work as the code doesn't know ahead of time what is actually going to get exported, just the total number of photos to try to export. You can run any of the export commands with --verbose or -V and it'll print name of every file exported or skipped. In your case the missing photos may be ones that aren't in the local library but are on iCloud. These can be downloaded with --download-missing — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[RhetTbull]

I've seen Photos not download some photos even when set to download all originals. PhotosExporter works in a very different way than osxphotos -- it gets the photos directly via an interface provided by Apple. This has some limitations in that much of the metadata is not accessible this way. osxphotos extracts the data directly from the Photos database, bypassing the native interface.

It's certainly possible osxphotos is incorrectly missing some photos. If you run osxphotos with --verbose and search for "Skipping" in the output it will show you which photos were skipped. If you identify any and can post some of the output here that would help with debugging as I could see why the photos are skipped.

[narensankar0529]

So running into a bunch of issues with that option - Logs of various runs attached. First log is for a run without the option. With the option turned on it seems to get into some form of loop that just runs forever and I have to CTRL-C

Without the download option I see some files skipped - Skipping missing edited photo for 9FD88C63-4719-49EF-B2EA-E9520475B9F3.mov Not sure what this refers to? A live pic or an actual movie shot.

Then when turn on the download option - Exporting edited version of 9FD88C63-4719-49EF-B2EA-E9520475B9F3.mov as 9FD88C63-4719-49EF-B2EA-E9520475B9F3_edited.mov 2020-11-07 11:04:12,073 - WARNING - _photoinfo_export.py - 691 - Error exporting photo 9FD88C63-4719-49EF-B2EA-E9520475B9F3 to /Users/silvialin/Pictures/Fotos1/2019/December/9FD88C63-4719-49EF-B2EA-E9520475B9F3_edited.mov with use_photos_export

But there are lots of warnings in the log.

osxphotoslog.zip

[RhetTbull]

The log data is helpful. From the first log I can see that the difference between "Exporting 49928 photos" and "Exported: 49849 photos" is 79 and there were 79 photos skipped edited videos. All 79 were .mov files and they appear to be actual movies not live photos.

Exporting IMG_9842.MOV (927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov) as 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov
Skipping original version of IMG_9842.MOV
Skipping missing edited photo for 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov

If you search for IMG_9842.MOV in Photos, what do you find? Does it show the movie file has been edited (e.g. if you right click is there a Revert to Original option? If not, it's possible osxphotos thinks these images are edited by they actually are not.

Regarding the warning and errors in the other log files, that's more difficult to debug. Those errors are generated by the --download-missing option which uses AppleScript to do the download.

Exporting IMG_9842.MOV (927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov) as 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov
Skipping original version of IMG_9842.MOV
Exporting edited version of 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov as 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84_edited.mov
2020-11-07 10:44:46,317 - WARNING - _photoinfo_export.py - 691 - Error exporting photo 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84 to /Users/silvialin/Pictures/Fotos1/2019/December/927D1A07-9F23-4FDA-8D2E-53D3DD12FE84_edited.mov with use_photos_export

Some of these are the same as the 79 in the first log file but there are others too.

The output of the following two commands would be helpful:

osxphotos debug-dump --dump photos --uuid 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84

osxphotos debug-dump --dump photos --uuid 6AF224FC-FF10-433A-A7C2-DEA668036900

The first will dump information related to one of the mystery .mov files and the latter will dump info related to one of the other errors.

[narensankar0529]

I figured out what those 79 are. They are slo-mo videos. They have been downloaded and are in my photos library but osxphotos seems to fail on them. Is this something that has been tested?

Right now download missing just barfs with a python error all the time. I will get you more logs from that error.

Thanks! —————

Naren Sankar On Nov 7, 2020, 1:14 PM +0800, Rhet Turnbull notifications@github.com, wrote:

The log data is helpful. From the first log I can see that the difference between "Exporting 49928 photos" and "Exported: 49849 photos" is 79 and there were 79 photos skipped edited videos. All 79 were .mov files and they appear to be actual movies not live photos. Exporting IMG_9842.MOV (927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov) as 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov Skipping original version of IMG_9842.MOV Skipping missing edited photo for 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov If you search for IMG_9842.MOV in Photos, what do you find? Does it show the movie file has been edited (e.g. if you right click is there a Revert to Original option? If not, it's possible osxphotos thinks these images are edited by they actually are not. Regarding the warning and errors in the other log files, that's more difficult to debug. Those errors are generated by the --download-missing option which uses AppleScript to do the download. Exporting IMG_9842.MOV (927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov) as 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov Skipping original version of IMG_9842.MOV Exporting edited version of 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84.mov as 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84_edited.mov 2020-11-07 10:44:46,317 - WARNING - _photoinfo_export.py - 691 - Error exporting photo 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84 to /Users/silvialin/Pictures/Fotos1/2019/December/927D1A07-9F23-4FDA-8D2E-53D3DD12FE84_edited.mov with use_photos_export Some of these are the same as the 79 in the first log file but there are others too. The output of the following two commands would be helpful: osxphotos debug-dump --dump photos --uuid 927D1A07-9F23-4FDA-8D2E-53D3DD12FE84 osxphotos debug-dump --dump photos --uuid 6AF224FC-FF10-433A-A7C2-DEA668036900 The first will dump information related to one of the mystery .mov files and the latter will dump info related to one of the other errors. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[narensankar0529]

Attached is the log from download missing with photos only. So no slo-mo included. And the dump from one of UUID which generates the AppleScript/Python failure.

osxphotos7.log B5B1FB72-CFEF-48A9-BA60-3452B6B52CE0.log

[RhetTbull]

osxphotos has been tested with slo-mo videos so I'm not sure why yours are not working. But now that I know that's what those are I'll look into the code. It appears that any photo that's attempted to download with AppleScript is failing which could be a permission problem. Take a look in Settings | Security & Privacy | Privacy | Automation Is Photos listed and checked under your Terminal? In the example below, I use iTerm as my terminal so it's listed there.

[narensankar0529]

Slow-mo exports if I don’t have the flag to disable originals if edited turned on.

If that flag is turned on then slow mo does not export and fails with the error

Very consistent reproducible.

Thanks! —————

Naren Sankar On Nov 7, 2020, 9:35 PM +0800, Rhet Turnbull notifications@github.com, wrote:

osxphotos has been tested with slo-mo videos so I'm not sure why yours are not working. But now that I know that's what those are I'll look into the code. It appears that any photo that's attempted to download with AppleScript is failing which could be a permission problem. Take a look in Settings | Security & Privacy | Privacy | Automation Is Photos listed and checked under your Terminal? In the example below, I use iTerm as my terminal so it's listed there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[RhetTbull]

OK, getting closer. Could you send a log with a debug-dump of one of the slow mo videos:

osxphotos debug-dump --dump photos --uuid 9FD88C63-4719-49EF-B2EA-E9520475B9F3

I'm trying to figure out why osxphotos thinks those are edited and thus looking for an edited image that is missing.

[narensankar0529]

slowmo.log

[RhetTbull]

I think I've found the problem and I've been able to replicate this. It appears that slow-mo videos are always marked as having adjustments (edits) in the database even if they haven't been edited. Instead of an edited video there's a .plist file and my guess is this contains the information for when to change frame rate between normal speed and slow motion. Thus, when using --skip-original-if-edited, osxphotos looks for an edited file but doesn't find one. If you actually edit the file, e.g. trim it, there will be an edited video generated and the export works as expected.

I'll add a check to osxphotos to look for this situation and export the original video if the file hasn't actually been edited.

[narensankar0529]

Great! Awesome.

Thanks! —————

Naren Sankar On Nov 7, 2020, 10:39 PM +0800, Rhet Turnbull notifications@github.com, wrote:

I think I've found the problem and I've been able to replicate this. It appears that slow-mo videos are always marked as having adjustments (edits) in the database even if they haven't been edited. Instead of an edited video there's a .plist file and my guess is this contains the information for when to change frame rate between normal speed and slow motion. Thus, when using --skip-original-if-edited, osxphotos looks for an edited file but doesn't find one. If you actually edit the file, e.g. trim it, there will be an edited video generated and the export works as expected. I'll add a check to osxphotos to look for this situation and export the original video if the file hasn't actually been edited. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[RhetTbull]

Release v0.36.9 should fix the problem with the slow-mo videos. I think the problem with --download-missing is likely due to permission and system integrity protection (SIP). If you didn't grant permission for AppleScript to access Photos when first run, it'll always fail. If you find that's not the problem, I'll keep investigating.

[narensankar0529]

osxphotos has been tested with slo-mo videos so I'm not sure why yours are not working. But now that I know that's what those are I'll look into the code. It appears that any photo that's attempted to download with AppleScript is failing which could be a permission problem. Take a look in Settings | Security & Privacy | Privacy | Automation Is Photos listed and checked under your Terminal? In the example below, I use iTerm as my terminal so it's listed there.

Automation has the correct permissions, but still download-missing is failing on every picture. Anything else I can capture for you in the logs to help debug? Also while this is running Photos keeps crashing as well. Though I think you have another bug report open on that issue.

[RhetTbull]

Here are a couple of ideas:

1. Uncheck then recheck the "Photos" item under Terminal in Privacy | Automation
2. Reboot the Mac -- I've noticed that sometimes once Photos' AppleScript engine gets hung it can take a reboot to fix things. Photos' AppleScript interface is very flaky but unfortunately it's the only way that osxphotos can currently export photos that aren't downloaded from iCloud. (@neilpa has prototyped a native app that might enable osxphotos to download photos without going through AppleScript)

If neither of those solves the problem let me know and I'll see about instrumenting some additional debug data.

[RhetTbull]

@narensankar0529 partial fix in v0.36.17. Give that a try and let me know if you still have problems. I suspect you will still have some errors but will see fewer errors. I've noticed that since upgrading to Catalina 10.15.7 the Photos AppleScript interface seems much more flaky and prone to crashing. Unfortunately the same is true on latest Big Sur 10.16 / 11.0 release candidate. Seems Apple no longer cares about the AppleScript interface.

[RhetTbull]

@narensankar0529 I've made additional fixes to the AppleScript code in v0.36.18. Let me know if you still have issues with --download-missing.

[narensankar0529]

It works much better and seems to not crash as often. Also when it crashes Photos seems to keep running, and osxphotos also keeps running. Only see a popup with the crash log. Attaching one such log to see if this can help. failed_log.txt

[RhetTbull]

I believe the crashes you're seeing are due to Photos' very buggy AppleScript interface. From your crash log it looks like the script engine is crashing. I've seen similar crashes ever since upgrading to Catalina. osxphotos now tries to work around those and will restart Photos as you've noticed. I am working on an implementation that doesn't require AppleScript that hopefully works better. It uses the native PhotoKit API for requesting photos.

[RhetTbull]

@narensankar0529 In release v0.36.20 I've added a new way to export missing photos. Currently, --download-missing interfaces with Photos via AppleScript to download from iCloud any missing photos. I added a new (currently hidden) option that uses Apple's native PhotoKit framework for requesting the photos. This should be more robust and faster. To try it, use osxphotos export with both the --download-missing --use-photokit options. The first time you run it, you'll be prompted to grant your Terminal access to Photos (and the script may fail) but it should work after you grant access via the dialog box and then re-run the script. This is very much an "alpha" feature now so may be buggy but I welcome any feedback you have!

[narensankar0529]

Seems to work so far. And something interesting is that it seems to now download some pictures it was not downloading before. I suddenly found many new pictures from 2015 that it downloaded. Most of them are selfies/HDR+Live Pics. But a few others as well. Not sure why they were not downloaded before using the AppleScript version.

[RhetTbull]

Thanks for the feedback. There is one known bug (#75) with the AppleScript download code for burst images. Do you know if the images in question were part of a burst set? If not, I'll investigate to see if I can replicate this behavior.

[narensankar0529]

No

They were not burst. It appears some what random. Originally I thought only live selfies were reexported but found some portraits as well

Thanks! —————

Naren Sankar On Nov 24, 2020, 1:23 AM +0800, Rhet Turnbull notifications@github.com, wrote:

Thanks for the feedback. There is one known bug (#75) with the AppleScript download code for burst images. Do you know if the images in question were part of a burst set? If not, I'll investigate to see if I can replicate this behavior. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[RhetTbull]

@narensankar0529 I've done some major refactoring of the export code on osxphotos recently thought should fix this and a number of other issues. Are you still experiencing these issues?

[narensankar0529]

Have not recently seen issues with export. Things seem to work correctly.