Open RhetTbull opened 1 year ago
@oPromessa when you have time, please upgrade to v0.55.3 and run the following script (copy the python code and save it to edited_paths.py
)
osxphotos run edited_paths.py > debug.txt 2>debug_err.txt
then send me the debug files at rturnbull+git@gmail.com or post them here. Will contain full path to your library and filename of photos but no other sensitive info.
edited_paths.py
:
""" Test edited paths for issue #871
Upgrade at least osxphotos version 0.55.3 (https://github.com/RhetTbull/osxphotos/releases/tag/v0.55.3)
Save as "edited_paths.py" then run with command:
osxphotos run edited_paths.py > debug.txt 2>debug_err.txt
Then post debug.txt and debug_err.txt here or send to me directly at rturnbull+git@gmail.com
If you need to specify the path to the library pass it as the first argument to the script:
osxphotos run edited_paths.py /path/to/library > debug.txt 2>debug_err.txt
"""
import csv
import os
import sys
import osxphotos
def verbose(msg):
print(msg, file=sys.stderr)
sys.stderr.flush()
def get_renders(photo):
"""Return list of render resources for given photo"""
library = photo._db._library_path
directory = photo._uuid[0] # first char of uuid
resource_dir = os.path.join(library, "resources", "renders", directory)
resources = []
if os.path.exists(resource_dir):
resources.extend(
filename for filename in os.listdir(resource_dir) if photo._uuid in filename
)
return resources
def main():
verbose(f"osxphotos version: {osxphotos._version.__version__}")
dbfile = sys.argv[1] if len(sys.argv) > 1 else None
photosdb = osxphotos.PhotosDB(dbfile=dbfile, verbose=verbose)
path_data = []
for photo in photosdb.photos():
if not photo.hasadjustments:
continue
candidate_path = photo._path_5()
candidate_path_edited = photo._path_edited_5()
renders = get_renders(photo)
path_data.append(
{
"uuid": photo.uuid,
"filename": photo.original_filename,
"has_adjustments": photo.hasadjustments,
"got_path": photo.path is not None,
"path": photo.path,
"path_candidate": candidate_path,
"got_path_edited": photo.path_edited is not None,
"path_edited": photo.path_edited,
"path_edited_candidate": candidate_path_edited,
"renders": ",".join(renders) if renders else "None",
}
)
# print results as CSV
fieldnames = path_data[0].keys()
writer = csv.DictWriter(sys.stdout, fieldnames=fieldnames)
writer.writeheader()
for row in path_data:
writer.writerow(row)
if __name__ == "__main__":
osxphotos.debug.set_debug(True)
main()
Find attached pics. Masked data with (xxx) and (yyy)
For one example I looked at... he was looking for file:
/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/resources/renders/A/ABDA9CB7-99C0-44FC-9097-3B048CAF88D0_1_201_a.jpeg
But the file does not exist on such folder. What does exist is a plist file with a similar file name ABDA9CB7-99C0-44FC-9097-3B048CAF88D0
without _1_201_a
...
/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/resources/renders/A/ABDA9CB7-99C0-44FC-9097-3B048CAF88D0.plist
and the contents of this file seems to indicate there is an adjustment with some data. Wild guess here... does Photos only marks the file has been rotated Anti-clockwise and avoids creating the edited jpg file to save space?
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>adjustmentBaseVersion</key>
<integer>0</integer>
<key>adjustmentData</key>
<data>
PU7LCoJAFP2Xsx5kfCAyP1BtCgpqES2uOuaEozJzdSP+ez6i3eG8J1jNVBIT1ARLnrU7
avOuGSqO0kT8uIcpuYaKkiQT6JzRLRObroXKZoGqc5b4rp3fqFBg3PGprbq1OB9MU54H
m2sHhSg8xJGEAPX9P4QklYEMwk3wRa0tXfVodlEK9A3xurM4LRWXG5ZdKj+DZ7uc8VDP
1/wF
</data>
<key>adjustmentEditorBundleID</key>
<string>com.apple.Photos</string>
<key>adjustmentFormatIdentifier</key>
<string>com.apple.photo</string>
<key>adjustmentFormatVersion</key>
<string>1.4</string>
<key>adjustmentTimestamp</key>
<date>2022-12-18T22:42:20Z</date>
</dict>
</plist>
Wild guess here... does Photos only marks the file has been rotated Anti-clockwise and avoids creating the edited jpg file to save space?
For RAW files, I have seen this happen (specifically on Big Sur). See #225. However, on Ventura, edits to RAW appear to produce the edited render. But I've not done extensive testing.
It's possible there are certain edits for which Photos does not produce a corresponding edited image and simply renders at run time from the plist. I've not seen this in my testing though with exception of aforementioned RAW images.
Scanning the debug data, it appears that a lot (maybe all) of the missing edited images does have the adjustments plist file. Take a look at a couple of these and see if you can determine a pattern in either type of image or type of edit that was done on the image.
Going thru it. Do you now what's the encoding of the "data" file on .plist file. It seems they are a few sets of cases that repeat across the files.
Some of them can be decode, available as PhotoInfo.adjustments
. (Thanks to @neilpa who did some great reverse engineering on this)
Try this:
osxphotos repl
That opens an interactive run/eval/print/loop (REPL) environment with your Photos data loaded by osxphotos. Then select an image in Photos and type:
photo = get_selected()[0]; print(photo.adjustments.adjustments if photo.adjustments else None)
If the photo has adjustments, it may print out something like this (this example for a RAW photo converted to Black & White):
[
{
'formatVersion': 1,
'enabled': True,
'settings': {'offsetTone': 0, 'offsetGrain': 0, 'offsetStrength': 0, 'offsetNeutralGamma': 0, 'auto': False, 'inputBlackAndWhite': -0.32291666666666663},
'identifier': 'SmartBlackAndWhite',
'formatIdentifier': 'com.apple.photo'
},
{
'formatVersion': 1,
'enabled': True,
'settings': {'boostVersion': 'FB3', 'boostParams': [0.42899999022483826, 0.8050000071525574, 0.9573039412498474], 'inputMethodVersion': '8'},
'identifier': 'RKRawDecodeOperation',
'formatIdentifier': 'com.apple.photo'
}
]
You can then hit up-arrow then return to execute the same command after selecting another photo. Sometimes the adjustment data cannot be decoded.
For one pic in particular, yields no adjustments!!!
>>> photo = get_selected()[0]; print(photo.adjustments.adjustments if photo.adjustments else None)
[]
The photo.adjustments field points out to the .plist file.
>>> print(photo.adjustments)
AdjustmentsInfo(plist_file='/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/resources/renders/A/ABDA9CB7-99C0-44FC-9097-3B048CAF88D0.plist')
But it does reference an orientation change... orientation=8
while original_orientation=1
>>> print(photo)
uuid: ABDA9CB7-99C0-44FC-9097-3B048CAF88D0
filename: PC310133-a-2.JPG
original_filename: PC310133-a-2.JPG
(...)
ismissing: false
hasadjustments: true
external_edit: false
favorite: false
hidden: false
latitude: null
longitude: null
path_edited: null
shared: false
isphoto: true
ismovie: false
uti: public.jpeg
burst: false
live_photo: false
path_live_photo: null
iscloudasset: false
incloud: null
date_modified: '2022-12-18T22:42:20.473628+00:00'
portrait: false
screenshot: false
slow_mo: false
time_lapse: false
hdr: false
selfie: false
panorama: false
has_raw: false
uti_raw: null
path_raw: null
place: null
exif: ExifInfo(flash_fired=False, iso=80, metering_mode=5, sample_rate=None, track_format=None,
white_balance=0, aperture=3.0, bit_rate=None, duration=None, exposure_bias=0.0,
focal_length=10.309999999999999, fps=None, latitude=None, longitude=None, shutter_speed=0.025,
camera_make='OLYMPUS IMAGING CORP. ', camera_model='uD800,S800 ', codec=None,
lens_model=None)
(...)
intrash: false
height: 3264
width: 2448
orientation: 8
original_height: 2448
original_width: 3264
original_orientation: 1
original_filesize: 1770975
Here's an updated script to collect data on the adjustments. Please run this the same way and send the data. (This one won't output any paths).
osxphotos run edited_paths2.py > debug2.txt 2>debug2_err.txt
""" Test edited paths for issue #871
Upgrade at least osxphotos version 0.55.3 (https://github.com/RhetTbull/osxphotos/releases/tag/v0.55.3)
Save as "edited_paths2.py" then run with command:
osxphotos run edited_paths2.py > debug2.txt 2>debug2_err.txt
Then post debug.txt and debug_err.txt here or send to me directly at rturnbull+git@gmail.com
If you need to specify the path to the library pass it as the first argument to the script:
osxphotos run edited_paths2.py /path/to/library > debug2.txt 2>debug2_err.txt
"""
import csv
import os
import sys
import osxphotos
def verbose(msg):
print(msg, file=sys.stderr)
sys.stderr.flush()
def get_renders(photo):
"""Return list of render resources for given photo"""
library = photo._db._library_path
directory = photo._uuid[0] # first char of uuid
resource_dir = os.path.join(library, "resources", "renders", directory)
resources = []
if os.path.exists(resource_dir):
resources.extend(
filename for filename in os.listdir(resource_dir) if photo._uuid in filename
)
return resources
def main():
verbose(f"osxphotos version: {osxphotos._version.__version__}")
dbfile = sys.argv[1] if len(sys.argv) > 1 else None
photosdb = osxphotos.PhotosDB(dbfile=dbfile, verbose=verbose)
path_data = []
for photo in photosdb.photos():
if not photo.hasadjustments:
continue
candidate_path = photo._path_5()
candidate_path_edited = photo._path_edited_5()
renders = get_renders(photo)
path_data.append(
{
"uuid": photo.uuid,
"filename": photo.original_filename,
"has_adjustments": photo.hasadjustments,
"got_path": photo.path is not None,
"got_path_edited": photo.path_edited is not None,
"orientation": photo.orientation,
"original_orientation": photo.original_orientation,
"orientation_change": photo.orientation != photo.original_orientation,
"adjustments": bool(photo.adjustments),
"adjustment_data": bool(photo.adjustments.adjustments)
if photo.adjustments
else False,
}
)
# print results as CSV
fieldnames = path_data[0].keys()
writer = csv.DictWriter(sys.stdout, fieldnames=fieldnames)
writer.writeheader()
for row in path_data:
writer.writerow(row)
if __name__ == "__main__":
osxphotos.debug.set_debug(True)
main()
Thanks -- will take a look. Quick examination shows 884 images missing an edited path. Of these, 862 appear to be only an orientation change (photo was rotated) as the photo shows as edited but there's no adjustment data in the plist file. Interestingly there appear to be 2143 images where there is an edited image path that also appear to be only an orientation change.
For the images where the edited path is missing and the only edit was an orientation change, osxphotos could conceivably do the rotation upon export (either through Quartz or through exiftool). For the others, I'm not sure what can be done if the files are truly not in the library. One idea might be to use PhotoKit to request the adjusted image in which case Photos would perform the adjustments in the plist and return the edited image. I've had reports of PhotoKit causing issues on Monterey (#625) so can't be 100% sure this will work until we test it.
Can you try the following and let me know if it works?
osxphotos export /path/to/export --skip-original-if-edited --download-missing --use-photokit --uuid 1DA18E86-596B-4610-BE90-AAE89FAE4522 --verbose
If that works, then we may have a workable solution.
Bummer !
$ osxphotos export ./export --skip-original-if-edited --download-missing --use-photokit --uuid 1DA18E86-596B-4610-BE90-AAE89FAE4522 --verbose
osxphotos version 0.55.3
Using last opened Photos library: /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary
Created export database /Users/Shared/Pictures/Export/edited_paths/export/.osxphotos_export.db
Processing database /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/photos.db
Processing database /Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/Photos.sqlite
Processing database.
Database version: 6000, 7.
Processing persons in photos.
Processing detected faces in photos.
Processing albums.
Processing keywords.
Processing photo details.
Processing import sessions.
Processing additional photo details.
Processing face details.
Processing photo labels.
Processing EXIF details.
Processing computed aesthetic scores.
Processing comments and likes for shared photos.
Processing moments.
Done processing details from Photos library.
Exporting 1 photo to /Users/Shared/Pictures/Export/edited_paths/export...
Exporting P5130204.JPG (P5130204.JPG) (1/1)
Skipping original version of P5130204.JPG
Exporting edited version of P5130204.JPG (P5130204.JPG)
Skipping missing edited photo P5130204.JPG (1DA18E86-596B-4610-BE90-AAE89FAE4522)
Exporting 1 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 1 photo, exported: 0, missing: 1, error: 0
Elapsed time: 0:00:00
Back to the drawing board. I'll see I'll take a look at which methods to instrument via --watch to get additional debug info.
Is this still an issue? I now have an old machine running Monterey for testing but was not able to replicate this.
osxphotos inspect
of0.55.1
continues not to show the Edite pic Path:Originally posted by @oPromessa in https://github.com/RhetTbull/osxphotos/discussions/856#discussioncomment-4395084