When exiftool fails on export | Pic still gets exported with missing data | Subsequent exports will not re-export the pic!

Latest version

Version: 0.55.2

Describe the bug When exiftool fails exporting an edited Pic (rotated + added GPS location in iPhoto/Photos):

File is exported without GPS data (of course it depends on the error from exiftool. Possible other errors to from exiftool
Noticed that touch does not seem to also run on the file!
Although the edited file is still exported
...and subsequent runs consider the file up-to-date, when, in fact it does not have GPS data.

IMPORTANT:

Just want to make you aware of this sequence/logic for this situation in case it may be helpful for other cases. I know I have to fix the pics myself. Will do that with "Script Editor" later on...
The problem happened with a certain Camera (OLYMPUS D800, which I no longer use) and rotated pics in iPhoto.
Somehow the edited pic in Photos (that comes from ~8 years ago through several iPhotos versions) generates "Maker Notes" and "Duplicated Orientation" tag on exiftool
I know I'll have to fix them all.

1st run: exiftool error!

But the file is exported but does not get all the EXIF info. For instance no GPS position data.

2022-12-14 22:24:11.961484 -- Exporting P4021768.JPG (P4021768.JPG) (6/6)
2022-12-14 22:24:11.962024 -- Skipping original version of P4021768.JPG
2022-12-14 22:24:11.963077 -- Exporting edited version of P4021768.JPG (P4021768.JPG)
2022-12-14 22:24:11.967205 -- Writing metadata with exiftool for P4021768_edited.jpeg
2022-12-14 22:24:12.367336 -- Exported P4021768.JPG to /Users/Shared/Pictures/Export/bad_pics_export_osxphotos/local/(...)/P4021768_edited.jpeg
2022-12-14 22:24:12.376409 -- ⚠️  exiftool warning for file /Users/Shared/Pictures/Export/bad_pics_export_osxphotos/local/(...)/P4021768_edited.jpeg: Warning: Duplicate Orientation tag in IFD0 - /var/folders/28/x8vbdy4j4vn680b5pt33hs0w0000gn/T/osxphotos_export_8vg04xwd/CDF617F6-9839-4A65-BE82-B28C45247FC6_1_201_a_exiftool.jpegWarning:  Possibly incorrect maker notes offsets (fix by -18?) - /var/folders/28/x8vbdy4j4vn680b5pt33hs0w0000gn/T/osxphotos_export_8vg04xwd/CDF617F6-9839-4A65-BE82-B28C45247FC6_1_201_a_exiftool.jpeg
2022-12-14 22:24:12.380512 -- Retrying export for photo (CDF617F6-9839-4A65-BE82-B28C45247FC6: P4021768.JPG)
2022-12-14 22:24:12.389445 -- Skipped up to date file /Users/Shared/Pictures/Export/bad_pics_export_osxphotos/local/(...)/P4021768_edited.jpeg

2nd run: consider the file up-to-date! But GPS data is missing.

2022-12-14 22:26:15.367527 -- Exporting P4021768.JPG (P4021768.JPG) (6/6)
2022-12-14 22:26:15.368193 -- Skipping original version of P4021768.JPG
2022-12-14 22:26:15.369319 -- Exporting edited version of P4021768.JPG (P4021768.JPG)
2022-12-14 22:26:15.496074 -- Skipped up to date file /Users/Shared/Pictures/Export/bad_pics_export_osxphotos/local/(...)/P4021768_edited.jpeg

On Photos "Revert to Original"(Cancels rotation but keep GPS data) and Re-edit to rotate: 3rd run

3rd run: perfect run. No exiftool error. GPS data exported. Touched executed.

2022-12-14 22:28:00.247428 -- Skipping original version of P4021768.JPG
2022-12-14 22:28:00.248482 -- Exporting edited version of P4021768.JPG (P4021768.JPG)
2022-12-14 22:28:00.377742 -- Writing metadata with exiftool for P4021768_edited.jpeg
2022-12-14 22:28:00.605418 -- Exported P4021768.JPG to /Users/Shared/Pictures/Export/bad_pics_export_osxphotos/local/(...)/P4021768_edited.jpeg
2022-12-14 22:28:00.613356 -- Exported updated file /Users/Shared/Pictures/Export/bad_pics_export_osxphotos/local/(...)/P4021768_edited.jpeg
2022-12-14 22:28:00.613782 -- Touched date on file /Users/Shared/Pictures/Export/bad_pics_export_osxphotos/local/(...)/P4021768_edited.jpeg

To Reproduce Steps to reproduce the behavior:

Command line on the crontab

osxphotos export "${DSTFOLDER}" --exportdb ${FOLDER}/.osxphotos_export.db --load-config ${WORKFOLDER}/colibri.toml >> ${LOGFILE} 2>&1

TOML file being used

[export]
cleanup = true
db = "/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary"
description_template = "Album:{album,}{newline}Description:{descr,}"
directory = "{album|filter(startswith 1|2)[/,.|:,.]}"
exiftool = true
exiftool_merge_keywords = true
exiftool_merge_persons = true
keyword_template = [ "{keyword}", "{label}", "{searchinfo.activity?activity:{searchinfo.activity},}", "{searchinfo.venue_type?venue_type:{searchinfo.venue_type},}", ]
no_progress = true
not_hidden = true
ramdb = true
retry = 2 
skip_bursts = true
skip_live = true
skip_original_if_edited = true
strip = true
timestamp = true
touch_file = true
update = true

Expected behavior Possibilities thinking out loud:

Either continue to try to re-export the file -- may be dumb as it will continue to give the same error!
One additional message to the user that he must fix it and possibly no re-exports will occur. Partially addressed as I already get the error from exiftool.
Delete the file in case of exif_tool error -- maybe too drastic.
Leave as it is.

Screenshots N/A Desktop (please complete the following information):

OS: Monterey 12.6.32
osxphotos version: 0.55.2

Additional context

DB with referenced pics.
Edited pics done inside iPhoto or Photos

Thanks for the detailed bug report. Will take a look when I get a chance. One guiding philosophy for osxphotos is "preserve as much data as possible" so I don't like the idea of deleting the file if exiftool fails. But the file should be attempted to exported again the next time.

I've been able to replicate this.

The problem is that the file is recorded in the database despite the exiftool error. Any file that gets exported without a file system error gets recorded:

https://github.com/RhetTbull/osxphotos/blob/8b9af7be6758292b03dc291261636f334ff407a4/osxphotos/photoexporter.py#L1180-L1194

One option would be to not record the file in the database at all if there's an exiftool error or warning. However, that means the file would get deleted upon --cleanup and would appear as a name collision upon --update.

Another option would be to consider the file corrupt and not export it all if there's an error but that's undesirable as one of the design philosophies in osxphotos is "preserve as much data as possible" as I use this primarily as an archival tool.

A third option is to somehow annotate in the export database that file was exported but "should be retried next time".

Here's an easy fix that I've tested and works. But I am not sure if this is really the desired behavior in all cases.

If there's an exiftool error or warning (or just an error?) don't write the exifdata to the database thus osxphotos will think that exiftool hasn't been run yet on the exported file and attempt to re-export it.

However, when used with --retry this will guarantee that the photo gets retried multiple times even though it's going to fail again. This is different than a disk or network error where retry is desired but osxphotos doesn't know this.

Also, on subsequent update, these photos will always get re-exported if user doesn't fix the EXIF. If the bad data is in the original image, user cannot fix the EXIF once imported to Photos (not without a bit of hacking the library). So these will always get re-exported which may be confusing to users.

Open to ideas on how to handle this.

Maybe a flag but that won't be obvious to users necessarily.

The current design treats exiftool as a best-effort "post processing" process independent of the actual export. I think this is preferred in most cases with the exception of your case where the edited pic has bad EXIF and can be corrected.

The current design treats exiftool as a best-effort "post processing" process independent of the actual export. I think this is preferred in most cases with the exception of your case where the edited pic has bad EXIF and can be corrected.

I believe the "leave as it is" and document that the user should correct it; would possibly be the way to go.

By document I mean it can be on the "manual" or on the error message of exiftool error.

This to prevent "no action" from the user and loose track of failed files once you re-run the export several times.

After thinking through this more I tend to agree the as-is behavior is probably best. However, here's a possible solution that gives the user control:

Modify the export database to include an error column that would store any errors generated during export including exiftool errrors and warnings. This would be useful in general to be able to examine what errors occurred during export after the fact. This data is stored in the results table but that doesn't happen until after the export is complete. I'd opt for error instead of exiftool_error to account for other errors. Internally, would likely use json to store the errors in the column so I wouldn't have to change the database schema in the future.
Add an option like --update-errors (open to naming ideas!) that would attempt to re-export any file where the last run had an error (exiftool or otherwise) even if that file would not have otherwise been updated. In conjunction with --update this would allow you to re-attempt these files after you fixed the EXIF data.
Add an option to the osxphotos exportdb command to be able to inspect errors, clear the error flag, etc.

Fixed in v0.55.3 which adds --update-errors to export and --last-errors, --errors to osxphotos exportdb for viewing which files produced errors.

I normally run osxphotos export at 3:44 in the morning and it takes ~4 hours to complete up to 7:44. It seems to me:

--last-errors filters the errors for the last date/time run
but some errors (shown with --errors) are tagged with a different date/time (but within the same run), thus not allowing "--last-errors" to tag them correctly.
I believe with the --retry option osxphotos attempts to export the file a few times and gets multiple errors and eventually succeeds, but stages the error with a different date/time.

My interpretation:

I understand last-errors would report last run errors tagged with the same start date, in my case: 2022-12-31T03:45:24.439678

$ osxphotos exportdb --last-errors .osxphotos_export.db
/Volumes/photo-1/Family.Photos/(...)/IMG_20221105_205105.HEIC, 2022-12-31T03:45:24.439678, Warning:  Fixed incorrect list type for XMP-mwg-rs:RegionList - /tmp/osxphotos_export_548nd0vk/IMG_20221105_205105_exiftool.HEIC
/Volumes/photo-1/Family.Photos/(...)/IMG_20220730_162555.HEIC, 2022-12-31T03:45:24.439678, Warning:  Fixed incorrect list type for XMP-mwg-rs:RegionList - /tmp/osxphotos_export_ur7kcj0h/IMG_20220730_162555_exiftool.HEIC
(...)

But checking the --verbose output file and the command --errors I get other errors with different indexed date/time (but within the same run).

$ grep P5290141  runOSxPhotos.cron.err
2022-12-31 07:19:07.541211 -- Exporting P5290141.JPG (P5290141.JPG) (82955/89381)
2022-12-31 07:19:07.769180 -- Writing metadata with exiftool for P5290141.JPG
2022-12-31 07:19:07.973021 -- Exported P5290141.JPG to /Volumes/photo-1/Family.Photos/(...)/P5290141.JPG
2022-12-31 07:19:07.981135 -- ❌️  exiftool error for file /Volumes/photo-1/Family.Photos/(...)/P5290141.JPG: Error:  Bad MakerNotes offset for Olympus_ImageProcessing_0x0801 - /tmp/osxphotos_export_ovuuxubq/P5290141_exiftool.JPG
2022-12-31 07:19:07.982026 -- ❌️  Error exporting photo (1E8448C7-2200-47B2-A869-61285E4C569C: P5290141.JPG) as /Volumes/photo-1/Family.Photos/(...)/P5290141.JPG: Error:  Bad MakerNotes offset for Olympus_ImageProcessing_0x0801 - /tmp/osxphotos_export_ovuuxubq/P5290141_exiftool.JPG
2022-12-31 07:19:07.982560 -- Retrying export for photo (1E8448C7-2200-47B2-A869-61285E4C569C: P5290141.JPG)
2022-12-31 07:19:07.991384 -- Skipped up to date file /Volumes/photo-1/Family.Photos/(...)/P5290141.JPG

$ osxphotos exportdb --errors .osxphotos_export.db 
(...)/IMG_20220806_171248.HEIC, 47E8C0CF-CED4-481C-AED3-149092760399, 2022-12-31 07:22:36.272, {"error": [], "exiftool_error": [], "exiftool_warning": [["/Volumes/photo1/Family.Photos/(...)/IMG_20220806_171248.HEIC", "Warning:  Fixed incorrect list type for XMP-mwg-rs:RegionList - 
/tmp/osxphotos_export_t4hp4d1s/IMG_20220806_171248_exiftool.HEIC"]]}
(...)/IMG_20221224_185404.HEIC, 07D4C738-2B74-4379-8C8D-9B855740E3E7, 2022-12-31 07:22:15.001, {"error": [], "exiftool_error": [], "exiftool_warning": [["/Volumes/photo-1/Family.Photos/(...)/IMG_20221224_185404.HEIC", "Warning:  Fixed incorrect list type for XMP-mwg-rs:RegionList - 
/tmp/osxphotos_export_s8umqo1l/IMG_20221224_185404_exiftool.HEIC"]]}
(...)/IMG_20221127_204729.HEIC, B8FE6BA1-FBE1-492D-8B1F-75752FBA5E3C, 2022-12-31 07:19:54.881, {"error": [], "exiftool_error": [], "exiftool_warning": [["/Volumes/photo-1/Family.Photos/(...)/IMG_20221127_204729.HEIC", "Warning:  Fixed incorrect list type for XMP-mwg-rs:RegionList - 
/tmp/osxphotos_export_gw4lqkex/IMG_20221127_204729_exiftool.HEIC"]]}
(...)/P5290141.JPG, 1E8448C7-2200-47B2-A869-61285E4C569C, 2022-12-31 07:19:07.991, {"error": 
[["/Volumes/photo-1/Family.Photos/(...)/P5290141.JPG", "Error:  Bad MakerNotes offset for Olympus_ImageProcessing_0x0801 - /tmp/osxphotos_export_ovuuxubq/P5290141_exiftool.JPG"]], "exiftool_error": [["/Volumes/photo-1/Family.Photos/(...)/P5290141.JPG", "Error:  Bad MakerNotes offset for Olympus_ImageProcessing_0x0801 - /tmp/osxphotos_export_ovuuxubq/P5290141_exiftool.JPG"]], "exiftool_warning": []}
(...)

I understand last-errors would report last run errors tagged with the same start date,

--last-errors does not look at the date/time, it looks at the most recent export results stored in the database (the source for --report) and extracts any errors found there. --errors searches for any errors stored in the error column of the database. Neither filters on date/time group.

I looked at the code and I can't see any obvious causes nor can I replicate this. I'll take a deeper look when I get a chance but it may be a while as I've got other projects going. I view osxphotos exportd as a sort of a "best effort" project that I don't spend a lot of time on. It's a collection of useful utilities but not intended to be a "primary" feature.

Note: the date/time stamp shown in --errors is the date/time that photo's record was last updated (when --update was run), not the date/time the error occurred. I realize this isn't intuitive but it's a limitation of the export database schema and when I added the errors column I didn't want to make significant changes to the database (which always introduces risk of more severe bugs).

Please try to generate a report for the last run using osxphotos exportdb /path/to/export/ --report debug.csv 0 and see if the missing errors show up in the report.

I looked at the code and I can't see any obvious causes nor can I replicate this. I'll take a deeper look when I get a chance but it may be a while as I've got other projects going. I view osxphotos exportd as a sort of a "best effort" project that I don't spend a lot of time on. It's a collection of useful utilities but not intended to be a "primary" feature.

Sure. No worries at all. I have the output of the run and I get the errors from there. So I'm good!

Coming off a report from #999 one additional report on item 3. reported on this comment (link)

One example. Exporting to the local SSD file with errors. but saving file and then deleting.

$ mkdir XYZ; osxphotos export XYZ  --uuid 18CD34E6-8E32-4873-BBA7-0EE8C9822EC0 --load-config export.toml --exportdb ./.osxphotos_export.db
2023-04-08 16:27:41.133699 -- Loaded options from file export.toml
2023-04-08 16:27:41.134218 -- osxphotos version: 0.59.1
2023-04-08 16:27:41.134485 -- Python version: 3.11.2 (main, Mar 24 2023, 00:28:48) [Clang 14.0.0 (clang-1400.0.29.202)]
2023-04-08 16:27:41.150960 -- Platform: macOS-12.6.4-x86_64-i386-64bit, 12.6.4
2023-04-08 16:27:41.151991 -- Verbose level: True
2023-04-08 16:27:41.152691 -- exiftool path: /usr/local/bin/exiftool
Using last opened Photos library: /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary
2023-04-08 16:27:41.155992 -- Warning: export database '/Users/Shared/Pictures/Logs/X/.osxphotos_export.db' is in a different directory than export destination '/Users/Shared/Pictures/Logs/X/XYZ'
2023-04-08 16:27:41.156976 -- WARNING: found other export database files in this destination directory branch.  This likely means you are attempting to export files into a directory that is either the parent or a child directory of a previous 
export. Proceeding may cause your exported files to be overwritten.
2023-04-08 16:27:41.157601 -- You are exporting to /Users/Shared/Pictures/Logs/X/XYZ, found .osxphotos_export.db files in:
2023-04-08 16:27:41.157929 -- /Users/Shared/Pictures/Logs/.osxphotos_export.db
Do you want to continue? [y/N]: y
2023-04-08 16:27:42.712701 -- Created export database ./.osxphotos_export.db
2023-04-08 16:27:42.714609 -- Processing database /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/photos.db
2023-04-08 16:27:42.716401 -- Processing database /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/Photos.sqlite
2023-04-08 16:27:42.723428 -- Processing database.
2023-04-08 16:27:42.723809 -- Database version: 6000, 7.
2023-04-08 16:27:42.724131 -- Processing persons in photos.
2023-04-08 16:27:42.755999 -- Processing detected faces in photos.
2023-04-08 16:27:42.871317 -- Processing albums.
2023-04-08 16:27:43.445391 -- Processing keywords.
2023-04-08 16:27:43.737461 -- Processing photo details.
2023-04-08 16:27:48.694457 -- Processing import sessions.
2023-04-08 16:27:48.931822 -- Processing additional photo details.
2023-04-08 16:27:50.471268 -- Processing face details.
2023-04-08 16:27:50.864890 -- Processing photo labels.
2023-04-08 16:27:57.290581 -- Processing EXIF details.
2023-04-08 16:27:58.004333 -- Processing computed aesthetic scores.
2023-04-08 16:27:58.730824 -- Processing comments and likes for shared photos.
2023-04-08 16:27:58.735070 -- Processing moments.
2023-04-08 16:27:58.801673 -- Done processing details from Photos library.
2023-04-08 16:27:58.840982 -- Exporting 1 photo to /Users/Shared/Pictures/Logs/X/XYZ...
2023-04-08 16:27:58.879972 -- Exporting IMG_0855.JPG (IMG_0855.JPG) (0/1)
2023-04-08 16:27:58.882187 -- Writing metadata with exiftool for IMG_0855.JPG
2023-04-08 16:27:59.394560 -- Exported IMG_0855.JPG to /Users/Shared/Pictures/Logs/X/XYZ/XXX/IMG_0855.JPG
2023-04-08 16:27:59.472380 -- Error exporting photo (18CD34E6-8E32-4873-BBA7-0EE8C9822EC0: IMG_0855.JPG) as IMG_0855.JPG: '0634722E-DB05-4F18-B6A3-9E612256E1F2'
2023-04-08 16:27:59.472947 -- Retrying export for photo (18CD34E6-8E32-4873-BBA7-0EE8C9822EC0: IMG_0855.JPG)
2023-04-08 16:27:59.475435 -- Writing metadata with exiftool for IMG_0855.JPG
2023-04-08 16:27:59.610230 -- Exported IMG_0855.JPG to /Users/Shared/Pictures/Logs/X/XYZ/XXX/IMG_0855.JPG
2023-04-08 16:27:59.611411 -- Error exporting photo (18CD34E6-8E32-4873-BBA7-0EE8C9822EC0: IMG_0855.JPG) as IMG_0855.JPG: '0634722E-DB05-4F18-B6A3-9E612256E1F2'
2023-04-08 16:27:59.611773 -- Retrying export for photo (18CD34E6-8E32-4873-BBA7-0EE8C9822EC0: IMG_0855.JPG)
2023-04-08 16:27:59.613263 -- Writing metadata with exiftool for IMG_0855.JPG
2023-04-08 16:27:59.750949 -- Exported IMG_0855.JPG to /Users/Shared/Pictures/Logs/X/XYZ/XXX/IMG_0855.JPG
2023-04-08 16:27:59.752195 -- Error exporting photo (18CD34E6-8E32-4873-BBA7-0EE8C9822EC0: IMG_0855.JPG) as IMG_0855.JPG: '0634722E-DB05-4F18-B6A3-9E612256E1F2'
2023-04-08 16:27:59.752651 -- Processed: 1 photo, exported: 0, updated: 0, skipped: 0, updated EXIF data: 0, missing: 0, error: 1, touched date: 0
2023-04-08 16:27:59.753070 -- Elapsed time: 0:00:00
2023-04-08 16:27:59.753397 -- Cleaning up /Users/Shared/Pictures/Logs/X/XYZ
2023-04-08 16:27:59.753926 -- Deleting /Users/Shared/Pictures/Logs/X/XYZ/2016-08-15 Verão Algarve 2016 II/IMG_0855.JPG
2023-04-08 16:27:59.754395 -- Deleting empty directory /Users/Shared/Pictures/Logs/X/XYZ/XXX
2023-04-08 16:27:59.754777 -- Deleting empty directory /Users/Shared/Pictures/Logs/X/XYZ
2023-04-08 16:27:59.755066 -- Deleted: 1 file, 2 directories
2023-04-08 16:27:59.756321 -- Writing export database changes back to ./.osxphotos_export.db
2023-04-08 16:27:59.760528 -- Cleaning up lock files

export.toml used

[export]
cleanup = true
description_template = "Album:{album,}{newline}Description:{descr,}"
directory = "{album|filter(startswith 1|2)[/,.|:,.]}"
exiftool = true
exiftool_merge_keywords = true
exiftool_merge_persons = true
keyword_template = [
    "{keyword}",
    "{label}",
    "{searchinfo.activity?activity:{searchinfo.activity},}",
    "{searchinfo.venue_type?venue_type:{searchinfo.venue_type},}",
]
no_progress = true
not_hidden = true
ramdb = true
retry = 2
skip_bursts = true
skip_live = true
skip_original_if_edited = true
strip = true
timestamp = true
touch_file = true
update = true
verbose = true

Trying to --debug and --breakpoint options but don't seem to be able to. Additionally behaviour seems to be difference. And a crashlog is generated.

$ osxphotos --debug --breakpoint=osxphotos.photoexporter.PhotoExporter.write_exiftool_metadata_to_file export XYZ  --uuid 18CD34E6-8E32-4873-BBA7-0EE8C9822EC0 --load-config export.toml --exportdb ./.osxphotos_export.db
Debugging enabled
2023-04-08 17:59:51.815882 -- Loaded options from file export.toml
2023-04-08 17:59:51.816394 -- osxphotos version: 0.59.1
2023-04-08 17:59:51.816646 -- Python version: 3.11.2 (main, Mar 24 2023, 00:28:48) [Clang 14.0.0 (clang-1400.0.29.202)]
2023-04-08 17:59:51.832841 -- Platform: macOS-12.6.4-x86_64-i386-64bit, 12.6.4
2023-04-08 17:59:51.833983 -- Verbose level: True
2023-04-08 17:59:51.834686 -- exiftool path: /usr/local/bin/exiftool
Using last opened Photos library: /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary
2023-04-08 17:59:51.838331 -- Warning: export database '/Users/Shared/Pictures/Logs/X/.osxphotos_export.db' is in a different directory than export destination '/Users/Shared/Pictures/Logs/X/XYZ'
2023-04-08 17:59:51.839470 -- WARNING: found other export database files in this destination directory branch.  This likely means you are attempting to export files into a directory that is either the parent or a child directory of a previous 
export. Proceeding may cause your exported files to be overwritten.
2023-04-08 17:59:51.839992 -- You are exporting to /Users/Shared/Pictures/Logs/X/XYZ, found .osxphotos_export.db files in:
2023-04-08 17:59:51.840315 -- /Users/Shared/Pictures/Logs/.osxphotos_export.db
Do you want to continue? [y/N]: y
2023-04-08 17:59:53.711372 -- Created export database ./.osxphotos_export.db
2023-04-08 17:59:53,714 - osxphotos - DEBUG - photosdb.py - 290 - dbfile = /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary
2023-04-08 17:59:53,715 - osxphotos - DEBUG - photosdb.py - 306 - dbfile = /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/photos.db
2023-04-08 17:59:53.715154 -- Processing database /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/photos.db
2023-04-08 17:59:53.717373 -- Processing database /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/Photos.sqlite
2023-04-08 17:59:53,725 - osxphotos - DEBUG - photosdb.py - 357 - _dbfile = /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/photos.db, _dbfile_actual = /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/Photos.sqlite
2023-04-08 17:59:53,725 - osxphotos - DEBUG - photosdb.py - 371 - library = /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary, masters = /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/originals
2023-04-08 17:59:53,725 - osxphotos - DEBUG - photosdb.py - 1627 - _process_database5
2023-04-08 17:59:53.725343 -- Processing database.
2023-04-08 17:59:53.725891 -- Database version: 6000, 7.
2023-04-08 17:59:53,726 - osxphotos - DEBUG - photosdb.py - 1650 - Getting information about persons
2023-04-08 17:59:53.726263 -- Processing persons in photos.
2023-04-08 17:59:53.802756 -- Processing detected faces in photos.
2023-04-08 17:59:53.922860 -- Processing albums.
2023-04-08 17:59:54.544071 -- Processing keywords.
2023-04-08 17:59:54.849618 -- Processing photo details.
2023-04-08 18:00:00.343722 -- Processing import sessions.
2023-04-08 18:00:00.628693 -- Processing additional photo details.
2023-04-08 18:00:02.227692 -- Processing face details.
2023-04-08 18:00:02.622020 -- Processing photo labels.
2023-04-08 18:00:09.566328 -- Processing EXIF details.
2023-04-08 18:00:10.348979 -- Processing computed aesthetic scores.
2023-04-08 18:00:11.155977 -- Processing comments and likes for shared photos.
2023-04-08 18:00:11.160614 -- Processing moments.
2023-04-08 18:00:11.232753 -- Done processing details from Photos library.
2023-04-08 18:00:11.283673 -- Exporting 1 photo to /Users/Shared/Pictures/Logs/X/XYZ...
2023-04-08 18:00:11.324557 -- Exporting IMG_0855.JPG (IMG_0855.JPG) (0/1)
2023-04-08 18:00:11.329158 -- Writing metadata with exiftool for IMG_0855.JPG
2023-04-08 18:00:11.969097 -- Exported IMG_0855.JPG to /Users/Shared/Pictures/Logs/X/XYZ/XXX/IMG_0855.JPG
2023-04-08 18:00:12,061 - osxphotos - DEBUG - photosdb.py - 2703 - Caught _dbalbum_folders KeyError for album: 0634722E-DB05-4F18-B6A3-9E612256E1F2
Something went wrong and osxphotos encountered an error:
'0634722E-DB05-4F18-B6A3-9E612256E1F2'
Crash log written to '/Users/Shared/Pictures/Logs/X/osxphotos_crash.log'
Please file a bug report at https://github.com/RhetTbull/osxphotos/issues with the crash log attached.
2023-04-08 18:00:12.072760 -- Cleaning up lock files

the Crash log is:

osxphotos crash log 
Created: 2023-04-08 18:00:12.062135
osxphotos version: 0.59.1
Platform: macOS-12.6.4-x86_64-i386-64bit
Python version: 3.11.2 (main, Mar 24 2023, 00:28:48) [Clang 14.0.0 (clang-1400.0.29.202)]
sys.argv: ['/Users/MSP/Documents/Apps/osxphotos/venv3.11/bin/osxphotos', '--debug', '--breakpoint=osxphotos.photoexporter.PhotoExporter.write_exiftool_metadata_to_file', 'export', 'XYZ', '--uuid', '18CD34E6-8E32-4873-BBA7-0EE8C9822EC0', '--load-config', 'export.toml', '--exportdb', './.osxphotos_export.db']
CRASH_DATA:

(...)

osxphotos version: 0.59.1
Error: '0634722E-DB05-4F18-B6A3-9E612256E1F2'
Traceback (most recent call last):
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/albuminfo.py", line 244, in folder_list
    return self._folders
           ^^^^^^^^^^^^^
AttributeError: 'ProjectInfo' object has no attribute '_folders'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): 
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/crash_reporter.py", line 36, in wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/cli/export.py", line 1445, in export
    export_results = export_photo(**kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/cli/export.py", line 1907, in export_photo
    results += export_photo_to_directory(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/cli/export.py", line 2250, in export_photo_to_directory
    raise e 
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/cli/export.py", line 2224, in export_photo_to_directory
    export_results = exporter.export(
                     ^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/photoexporter.py", line 486, in export
    all_results += self._export_photo(
                   ^^^^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/photoexporter.py", line 1215, in _export_photo 
    rec.photoinfo = self.photo.json()
                    ^^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/photoinfo.py", line 1898, in json
    dict_data = self.asdict()
                ^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/photoinfo.py", line 1795, in asdict
    project_info = [p.asdict() for p in self.project_info]
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/photoinfo.py", line 1795, in <listcomp>
    project_info = [p.asdict() for p in self.project_info]
                    ^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/albuminfo.py", line 306, in asdict
    dict_data["folder_list"] = [f.uuid for f in self.folder_list]
                                                ^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/albuminfo.py", line 246, in folder_list
    self._folders = self._db._album_folder_hierarchy_folderinfo(self._uuid)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/photosdb/photosdb.py", line 2736, in _album_folder_hierarchy_folderinfo
    return self._album_folder_hierarchy_folderinfo_5(album_uuid)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/MSP/Documents/Apps/osxphotos/venv3.11/lib/python3.11/site-packages/osxphotos/photosdb/photosdb.py", line 2777, in _album_folder_hierarchy_folderinfo_5
    folders = self._dbalbum_folders[album_uuid]
              ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: '0634722E-DB05-4F18-B6A3-9E612256E1F2'

I think I've fixed this in the branch I'm currently working on for #999. Will try to get a new release out this weekend.

I'm wondering if this came in @RhetTbull ? I appear to be having a similar issue, but not entirely sure its the same.

Each time I export with --exiftool, I receive the following error for the same files:

exiftool warning for file /Volumes/photos/IMG_6249.HEIC: Warning:  Fixed incorrect list type for XMP-mwg-rs:RegionList - /tmp/osxphotos_export_d0dbdvnn/FB65DC12-F941-4D83-A3CC-BE3C4542067E_exiftool.heic

Second run:

Exporting IMG_6249.HEIC (FB65DC12-F941-4D83-A3CC-BE3C4542067E.heic) (79/579)
Writing metadata with exiftool for IMG_6249.HEIC
Exported IMG_6249.HEIC to /Volumes/photos/IMG_6249.HEIC
Skipped up to date exiftool sidecar /Volumes/photos/IMG_6249.HEIC.json
⚠️  exiftool warning for file /Volumes/photos/IMG_6249.HEIC: Warning:  Fixed incorrect list type for XMP-mwg-rs:RegionList - /tmp/osxphotos_export_xxly0k03/FB65DC12-F941-4D83-A3CC-BE3C4542067E_exiftool.heic
Exported updated file /Volumes/photos/IMG_6249.HEIC

This impacts the stats at the end:

Processed: 579 photos, exported: 4, updated: 284, skipped: 731, updated EXIF data: 288, missing: 1, error: 9, touched date: 290

If I export again, it'll be the same message for the same file.

@teh-hippo can you send me one of the files in question? Ideally I'd like the original from Photos and the version processed by exiftool.

osxphotos.py@gmail.com

@teh-hippo thanks for the test data. I'm not sure why this is happening now but I am able to replicate it. Seems only to be an issue with HEIC. This did not always behave this way so not sure what changed (may be related to specific version of exiftool). I can work around this be changing how the face regions are written from the current format to a a struct:

Current format (same as produced by exiftool...)

[{
    "RegionAppliedToDimensionsW": 4032,
    "RegionAppliedToDimensionsH": 3024,
    "RegionAppliedToDimensionsUnit": "pixel",
    "RegionName": ["Face2", "Face3"],
    "RegionType": ["Face", "Face"],
    "RegionAreaX": [0.28545492666739014, 0.280715821281312],
    "RegionAreaY": [0.4058795908771494, 0.07760194916363539],
    "RegionAreaW": [0.2098364531993866, 0.08417508751153946],
    "RegionAreaH": [0.27978193759918213, 0.11223345001538594],
    "RegionAreaUnit": ["normalized", "normalized"],
    "RegionPersonDisplayName": ["Face2", "Face3"],
}]

New format:

    "RegionInfo": {
        "AppliedToDimensions": {
          "W": 4032,
          "H": 3024,
          "Unit": "pixel"
        },
        "RegionList": [
          {
            "Name": "Face2",
            "Type": "Face",
            "Area": {
              "X": 0.28545492666739014,
              "Y": 0.4058795908771494,
              "W": 0.2098364531993866,
              "H": 0.27978193759918213,
              "Unit": "normalized"
            },
            "PersonDisplayName": "Face2"
          },
          {
            "Name": "Face3",
            "Type": "Face",
            "Area": {
              "X": 0.280715821281312,
              "Y": 0.07760194916363539,
              "W": 0.08417508751153946,
              "H": 0.11223345001538594,
              "Unit": "normalized"
            },
            "PersonDisplayName": "Face3"
          }
        ]
      }

I will try to get to this as soon as able. In the meantime, it does appear the data is actually being written correctly but the warning is annoying.

RhetTbull / osxphotos