RhetTbull / osxphotos

Python app to work with pictures and associated metadata from Apple Photos on macOS. Also includes a package to provide programmatic access to the Photos library, pictures, and metadata.
MIT License
1.84k stars 94 forks source link

Edited photo path missing on Monterey #871

Open RhetTbull opened 1 year ago

RhetTbull commented 1 year ago
  Sorry for the delay in providing answers to your questions. Here they go...

Could you try this:

Find photo P2071699 in your library. Verify that Photos shows the file as edited and try to export it using Photos (do not use "export unmodified original", use "export 1 photo"). What I'm trying to figure out is if Photos itself knows where the edited image is or if its just showing you the preview image.

  • The "export 1 photo" worked correctly.

Another test would be to try to edit the edited photo in Photos. Does Photo show the edit screen or do you get an error about missing photo?

  • Hitting "Edit" it brings up the Photos Editing screen.

If Photos can find it and osxphotos can't, I'll need some additional data to figure out the problem.

Originally posted by @oPromessa in https://github.com/RhetTbull/osxphotos/discussions/856#discussioncomment-4395084

RhetTbull commented 1 year ago

@oPromessa when you have time, please upgrade to v0.55.3 and run the following script (copy the python code and save it to edited_paths.py)

osxphotos run edited_paths.py > debug.txt 2>debug_err.txt

then send me the debug files at rturnbull+git@gmail.com or post them here. Will contain full path to your library and filename of photos but no other sensitive info.

edited_paths.py:

""" Test edited paths for issue #871

    Upgrade at least osxphotos version 0.55.3 (https://github.com/RhetTbull/osxphotos/releases/tag/v0.55.3)

    Save as "edited_paths.py" then run with command:

    osxphotos run edited_paths.py > debug.txt 2>debug_err.txt 

    Then post debug.txt and debug_err.txt here or send to me directly at rturnbull+git@gmail.com

    If you need to specify the path to the library pass it as the first argument to the script:

    osxphotos run edited_paths.py /path/to/library > debug.txt 2>debug_err.txt
"""

import csv
import os
import sys

import osxphotos

def verbose(msg):
    print(msg, file=sys.stderr)
    sys.stderr.flush()

def get_renders(photo):
    """Return list of render resources for given photo"""
    library = photo._db._library_path
    directory = photo._uuid[0]  # first char of uuid
    resource_dir = os.path.join(library, "resources", "renders", directory)
    resources = []
    if os.path.exists(resource_dir):
        resources.extend(
            filename for filename in os.listdir(resource_dir) if photo._uuid in filename
        )
    return resources

def main():
    verbose(f"osxphotos version: {osxphotos._version.__version__}")
    dbfile = sys.argv[1] if len(sys.argv) > 1 else None
    photosdb = osxphotos.PhotosDB(dbfile=dbfile, verbose=verbose)
    path_data = []
    for photo in photosdb.photos():
        if not photo.hasadjustments:
            continue

        candidate_path = photo._path_5()
        candidate_path_edited = photo._path_edited_5()
        renders = get_renders(photo)
        path_data.append(
            {
                "uuid": photo.uuid,
                "filename": photo.original_filename,
                "has_adjustments": photo.hasadjustments,
                "got_path": photo.path is not None,
                "path": photo.path,
                "path_candidate": candidate_path,
                "got_path_edited": photo.path_edited is not None,
                "path_edited": photo.path_edited,
                "path_edited_candidate": candidate_path_edited,
                "renders": ",".join(renders) if renders else "None",
            }
        )

    # print results as CSV
    fieldnames = path_data[0].keys()
    writer = csv.DictWriter(sys.stdout, fieldnames=fieldnames)
    writer.writeheader()
    for row in path_data:
        writer.writerow(row)

if __name__ == "__main__":
    osxphotos.debug.set_debug(True)
    main()
oPromessa commented 1 year ago

Find attached pics. Masked data with (xxx) and (yyy)

For one example I looked at... he was looking for file:

/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/resources/renders/A/ABDA9CB7-99C0-44FC-9097-3B048CAF88D0_1_201_a.jpeg

But the file does not exist on such folder. What does exist is a plist file with a similar file name ABDA9CB7-99C0-44FC-9097-3B048CAF88D0 without _1_201_a...

/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/resources/renders/A/ABDA9CB7-99C0-44FC-9097-3B048CAF88D0.plist 

and the contents of this file seems to indicate there is an adjustment with some data. Wild guess here... does Photos only marks the file has been rotated Anti-clockwise and avoids creating the edited jpg file to save space?

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>adjustmentBaseVersion</key>
        <integer>0</integer>
        <key>adjustmentData</key>
        <data>
        PU7LCoJAFP2Xsx5kfCAyP1BtCgpqES2uOuaEozJzdSP+ez6i3eG8J1jNVBIT1ARLnrU7
        avOuGSqO0kT8uIcpuYaKkiQT6JzRLRObroXKZoGqc5b4rp3fqFBg3PGprbq1OB9MU54H
        m2sHhSg8xJGEAPX9P4QklYEMwk3wRa0tXfVodlEK9A3xurM4LRWXG5ZdKj+DZ7uc8VDP
        1/wF
        </data>
        <key>adjustmentEditorBundleID</key>
        <string>com.apple.Photos</string>
        <key>adjustmentFormatIdentifier</key>
        <string>com.apple.photo</string>
        <key>adjustmentFormatVersion</key>
        <string>1.4</string>
        <key>adjustmentTimestamp</key>
        <date>2022-12-18T22:42:20Z</date>
</dict>
</plist>
RhetTbull commented 1 year ago

Wild guess here... does Photos only marks the file has been rotated Anti-clockwise and avoids creating the edited jpg file to save space?

For RAW files, I have seen this happen (specifically on Big Sur). See #225. However, on Ventura, edits to RAW appear to produce the edited render. But I've not done extensive testing.

It's possible there are certain edits for which Photos does not produce a corresponding edited image and simply renders at run time from the plist. I've not seen this in my testing though with exception of aforementioned RAW images.

RhetTbull commented 1 year ago

Scanning the debug data, it appears that a lot (maybe all) of the missing edited images does have the adjustments plist file. Take a look at a couple of these and see if you can determine a pattern in either type of image or type of edit that was done on the image.

oPromessa commented 1 year ago

Going thru it. Do you now what's the encoding of the "data" file on .plist file. It seems they are a few sets of cases that repeat across the files.

RhetTbull commented 1 year ago

Some of them can be decode, available as PhotoInfo.adjustments. (Thanks to @neilpa who did some great reverse engineering on this)

Try this:

osxphotos repl

That opens an interactive run/eval/print/loop (REPL) environment with your Photos data loaded by osxphotos. Then select an image in Photos and type:

photo = get_selected()[0]; print(photo.adjustments.adjustments if photo.adjustments else None)

If the photo has adjustments, it may print out something like this (this example for a RAW photo converted to Black & White):

[
    {
        'formatVersion': 1,
        'enabled': True,
        'settings': {'offsetTone': 0, 'offsetGrain': 0, 'offsetStrength': 0, 'offsetNeutralGamma': 0, 'auto': False, 'inputBlackAndWhite': -0.32291666666666663},
        'identifier': 'SmartBlackAndWhite',
        'formatIdentifier': 'com.apple.photo'
    },
    {
        'formatVersion': 1,
        'enabled': True,
        'settings': {'boostVersion': 'FB3', 'boostParams': [0.42899999022483826, 0.8050000071525574, 0.9573039412498474], 'inputMethodVersion': '8'},
        'identifier': 'RKRawDecodeOperation',
        'formatIdentifier': 'com.apple.photo'
    }
]

You can then hit up-arrow then return to execute the same command after selecting another photo. Sometimes the adjustment data cannot be decoded.

oPromessa commented 1 year ago

For one pic in particular, yields no adjustments!!!

>>> photo = get_selected()[0]; print(photo.adjustments.adjustments if photo.adjustments else None)
[]

The photo.adjustments field points out to the .plist file.

>>> print(photo.adjustments)
AdjustmentsInfo(plist_file='/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/resources/renders/A/ABDA9CB7-99C0-44FC-9097-3B048CAF88D0.plist')

But it does reference an orientation change... orientation=8 while original_orientation=1

>>> print(photo)
uuid: ABDA9CB7-99C0-44FC-9097-3B048CAF88D0
filename: PC310133-a-2.JPG
original_filename: PC310133-a-2.JPG
(...)
ismissing: false
hasadjustments: true
external_edit: false
favorite: false
hidden: false
latitude: null
longitude: null
path_edited: null
shared: false
isphoto: true
ismovie: false
uti: public.jpeg
burst: false
live_photo: false
path_live_photo: null
iscloudasset: false
incloud: null
date_modified: '2022-12-18T22:42:20.473628+00:00'
portrait: false
screenshot: false
slow_mo: false
time_lapse: false
hdr: false
selfie: false
panorama: false
has_raw: false
uti_raw: null
path_raw: null
place: null
exif: ExifInfo(flash_fired=False, iso=80, metering_mode=5, sample_rate=None, track_format=None,
  white_balance=0, aperture=3.0, bit_rate=None, duration=None, exposure_bias=0.0,
  focal_length=10.309999999999999, fps=None, latitude=None, longitude=None, shutter_speed=0.025,
  camera_make='OLYMPUS IMAGING CORP.  ', camera_model='uD800,S800      ', codec=None,
  lens_model=None)
(...)
intrash: false
height: 3264
width: 2448
orientation: 8
original_height: 2448
original_width: 3264
original_orientation: 1
original_filesize: 1770975
RhetTbull commented 1 year ago

Here's an updated script to collect data on the adjustments. Please run this the same way and send the data. (This one won't output any paths).

osxphotos run edited_paths2.py > debug2.txt 2>debug2_err.txt

""" Test edited paths for issue #871

    Upgrade at least osxphotos version 0.55.3 (https://github.com/RhetTbull/osxphotos/releases/tag/v0.55.3)

    Save as "edited_paths2.py" then run with command:

    osxphotos run edited_paths2.py > debug2.txt 2>debug2_err.txt 

    Then post debug.txt and debug_err.txt here or send to me directly at rturnbull+git@gmail.com

    If you need to specify the path to the library pass it as the first argument to the script:

    osxphotos run edited_paths2.py /path/to/library > debug2.txt 2>debug2_err.txt
"""

import csv
import os
import sys

import osxphotos

def verbose(msg):
    print(msg, file=sys.stderr)
    sys.stderr.flush()

def get_renders(photo):
    """Return list of render resources for given photo"""
    library = photo._db._library_path
    directory = photo._uuid[0]  # first char of uuid
    resource_dir = os.path.join(library, "resources", "renders", directory)
    resources = []
    if os.path.exists(resource_dir):
        resources.extend(
            filename for filename in os.listdir(resource_dir) if photo._uuid in filename
        )
    return resources

def main():
    verbose(f"osxphotos version: {osxphotos._version.__version__}")
    dbfile = sys.argv[1] if len(sys.argv) > 1 else None
    photosdb = osxphotos.PhotosDB(dbfile=dbfile, verbose=verbose)
    path_data = []
    for photo in photosdb.photos():
        if not photo.hasadjustments:
            continue

        candidate_path = photo._path_5()
        candidate_path_edited = photo._path_edited_5()
        renders = get_renders(photo)
        path_data.append(
            {
                "uuid": photo.uuid,
                "filename": photo.original_filename,
                "has_adjustments": photo.hasadjustments,
                "got_path": photo.path is not None,
                "got_path_edited": photo.path_edited is not None,
                "orientation": photo.orientation,
                "original_orientation": photo.original_orientation,
                "orientation_change": photo.orientation != photo.original_orientation,
                "adjustments": bool(photo.adjustments),
                "adjustment_data": bool(photo.adjustments.adjustments)
                if photo.adjustments
                else False,
            }
        )

    # print results as CSV
    fieldnames = path_data[0].keys()
    writer = csv.DictWriter(sys.stdout, fieldnames=fieldnames)
    writer.writeheader()
    for row in path_data:
        writer.writerow(row)

if __name__ == "__main__":
    osxphotos.debug.set_debug(True)
    main()
oPromessa commented 1 year ago

Thanks for the support!

Please find attached

RhetTbull commented 1 year ago

Thanks -- will take a look. Quick examination shows 884 images missing an edited path. Of these, 862 appear to be only an orientation change (photo was rotated) as the photo shows as edited but there's no adjustment data in the plist file. Interestingly there appear to be 2143 images where there is an edited image path that also appear to be only an orientation change.

For the images where the edited path is missing and the only edit was an orientation change, osxphotos could conceivably do the rotation upon export (either through Quartz or through exiftool). For the others, I'm not sure what can be done if the files are truly not in the library. One idea might be to use PhotoKit to request the adjusted image in which case Photos would perform the adjustments in the plist and return the edited image. I've had reports of PhotoKit causing issues on Monterey (#625) so can't be 100% sure this will work until we test it.

Can you try the following and let me know if it works?

osxphotos export /path/to/export --skip-original-if-edited --download-missing --use-photokit --uuid 1DA18E86-596B-4610-BE90-AAE89FAE4522 --verbose

If that works, then we may have a workable solution.

oPromessa commented 1 year ago

Bummer !

$ osxphotos export ./export --skip-original-if-edited --download-missing --use-photokit --uuid 1DA18E86-596B-4610-BE90-AAE89FAE4522 --verbose
osxphotos version 0.55.3
Using last opened Photos library: /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary
Created export database /Users/Shared/Pictures/Export/edited_paths/export/.osxphotos_export.db
Processing database /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/photos.db
Processing database /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/Photos.sqlite
Processing database.
Database version: 6000, 7.
Processing persons in photos.
Processing detected faces in photos.
Processing albums.
Processing keywords.
Processing photo details.
Processing import sessions.
Processing additional photo details.
Processing face details.
Processing photo labels.
Processing EXIF details.
Processing computed aesthetic scores.
Processing comments and likes for shared photos.
Processing moments.
Done processing details from Photos library.
Exporting 1 photo to /Users/Shared/Pictures/Export/edited_paths/export...
Exporting P5130204.JPG (P5130204.JPG) (1/1)
Skipping original version of P5130204.JPG
Exporting edited version of P5130204.JPG (P5130204.JPG)
Skipping missing edited photo P5130204.JPG (1DA18E86-596B-4610-BE90-AAE89FAE4522)
Exporting 1 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 1 photo, exported: 0, missing: 1, error: 0
Elapsed time: 0:00:00
RhetTbull commented 1 year ago

Back to the drawing board. I'll see I'll take a look at which methods to instrument via --watch to get additional debug info.

RhetTbull commented 6 months ago

Is this still an issue? I now have an old machine running Monterey for testing but was not able to replicate this.