Convert edited RAW files without _edited suffix

[odedia]

When you edit JPEGs, it makes sense to want to preserve the original jpeg and an edited version. However, when dealing with RAW files, usually you just want a RAW+JPEG combo. Currently, all edited JPEGs contain a suffix like _edited to indicate they are edited versions of the original.

It would be great to have a property like --ignore-edited-suffix-for-raw so that if a RAW file was edited in Apple Photos, the result should be just the RAW file and its edited JPEG without suffix, for example:

_DSC8829.NEF
_DSC8829.jpeg
_DSC8829.jpeg.xmp
_DSC8829.NEF.xmp

This should also make it easier for apps like PhotoPrism to stack them correctly.

Thanks

[RhetTbull]

Since --edited-suffix SUFFIX is a template, this should work: --edited-suffix "{photo.has_raw?,_edited}"

But note that often when editing a RAW in Photos, the JPEG is not edited. The edits are stored in a .AAE file in the library as non-destructive edits and the JPEG that OSXPhotos exports will be the original, not the edited version. This will be fixed a future version when I can convert OSXPhotos to using native (but undocumented) API calls instead hacking the Photos library, which it does now. But that day is probably not until next year.

[odedia]

I see, thank you! If plain RAW files (i.e. without a JPEG+RAW combo in Apple Photos) cannot export the edited version at the moment, I think it makes more sense to export them without the corresponding edited JPEG, since most photo management solutions like PhotoPrism and Immich already convert these themselves. Is there a way to tell osxphotos "If it's RAW, just export the RAW file without JPEG conversion, if it's RAW+JPEG, export both the RAW and the JPEG"?

[odedia]

After thinking about it some more, I actually prefer the way OSXPhotos converts the images because it uses the same engine as Apple Photos (I think?) So, Ideally, this would be the naming logic:

1. If there's only a .NEF, create a matching JPEG with the exact same name.
2. If there's a .NEF that was edited in Apple Photos, create a matching JPEG with the exact same name, include an Edited tag in the metadata.
3. If there's a .NEF+JPEG combo, export the .NEF and the .JPEG with the exact same names.
4. If the .JPEG from the .NEF+JPEG combo was edited in Apple Photos, use the edited JPEG in the export, but keep the exact same names (no _edited). Add an Edited metadata.

[RhetTbull]

After thinking about it some more, I actually prefer the way OSXPhotos converts the images because it uses the same engine as Apple Photos (I think?)

For conversion, OSXPhotos uses the native Apple API so it should match what Photos does.

It's important to note OSXPhotos exports only the cached edited version if there is one. This is a smaller file than you'd get with the "Export" command in Photos which renders a JPEG or other format at the time of the export that incorporates edits. That means that currently some edited photos won't have an edited image when exported with OSXPhotos and you'll only get the original. This mostly affects RAW images. I plan to fix this in a future release that will use native code to apply the edits and export the converted image (and maybe let you specify file format). This will be a big change and take some time to implement.

The workflow you outlined should be possible but will take a bit of thought to figure out the best way to do this without over complicating the export options and logic. One thought I've had is a plug-in system that would allow you to install custom "handlers" for certain files. Eg if RAW, hand off to the plug-in to do the export.

For the metadata, you could do something like --keyword-template "{edited_version?Edited,}"

This adds the Edited keyword only if the file being processed is the edited version of the photo.

[odedia]

I've played around with various permutations of the edited images and how they show up as stacks in PhotoPrism. The below stacks all these images together as it should, even though there's a HEIC, a .MOV (live photo) and an .edited in the end of the edited filename:

However, the following was not detected stacked as you'd expect. The right is RAW+JPEG, the left is the .edited.jpeg:

It seems that the original .NEF doesn't have an XMP created for it, so it originally thought the date taken was 2017 instead of the correct-in-apple-photos of 1980. I then ran the export again with both --sidecar and --exiftool and the .NEF now had the corrected date. But that still didn't help stack them together.

I then checked how PhotoPrism uses exif data to stack together and came up with this: https://github.com/photoprism/photoprism/issues/20

We now extract Document and Instance ID via exiftool and use it for grouping related files with different names.

I wonder if osxphotos use the same instance ID for all permutations of the same image?

[RhetTbull]

OSXPhotos does not write an instance ID. If you can tell me exactly what tag this is or send the output of exiftool -G -j I can look at adding this.

[odedia]

It seems like the correct parameter to use is in XMP files and it's called xmpMM:DocumentID. Per https://developer.adobe.com/xmp/docs/XMPNamespaces/xmpMM/, this parameter is The common identifier for all versions and renditions of a resource. It should be based on a UUID; Created once for new resources. Different renditions are expected to have different values for xmpMM:DocumentID.

I guess this should be set to the value of photo.uuid?

Here's a sample XMP that was generated with ChatGPT so take it with a grain of salt :)

<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.6-c140 79.160451, 2017/05/06-01:08:21        ">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about="" xmlns:xmp="http://ns.adobe.com/xap/1.0/" xmlns:pdf="http://ns.adobe.com/pdf/1.3/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/" xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/" xmlns:stEvt="http://ns.adobe.com/xap/1.0/sType/ResourceEvent#" xmlns:stRef="http://ns.adobe.com/xap/1.0/sType/ResourceRef#" xmlns:exif="http://ns.adobe.com/exif/1.0/" xmlns:aux="http://ns.adobe.com/exif/1.0/aux/" xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/">
         <xmp:CreatorTool>Adobe Photoshop CC 2019 (Windows)</xmp:CreatorTool>
         <xmp:MetadataDate>2024-04-11T10:00:00Z</xmp:MetadataDate>
         <xmp:InstanceID>xmp.iid:20edabe6-8e25-4f19-9b30-f8db9d9d8b6a</xmp:InstanceID>
         <pdf:Producer>Adobe PDF Library 15.0</pdf:Producer>
         <pdf:CreationDate>2024-04-11T08:00:00Z</pdf:CreationDate>
         <pdf:ModDate>2024-04-11T08:30:00Z</pdf:ModDate>
         <pdf:Trapped>False</pdf:Trapped>
         <xmpMM:DocumentID>uuid:5d2083e8-5b11-4f42-b7a0-1ee8ef04eaf1</xmpMM:DocumentID>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>

[odedia]

I tried to add the fields manually to two .XMP files (1 below as an example) but it doesn't seem to impact stacking. I'll followup at the photoprism discussion threads.

<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="osxphotos">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about="" 
 xmlns:dc="http://purl.org/dc/elements/1.1/" 
 xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
        <photoshop:SidecarForExtension>jpeg</photoshop:SidecarForExtension>
        <dc:description>
        <rdf:Alt>
         <rdf:li xml:lang='x-default'></rdf:li>
        </rdf:Alt>
        </dc:description>
        <dc:title>
         <rdf:Alt>
          <rdf:li xml:lang='x-default'/>
         </rdf:Alt>
        </dc:title>
        <dc:subject>
         <rdf:Bag>
          <rdf:li>Blue Sky</rdf:li>
          <rdf:li>Outdoor</rdf:li>
          <rdf:li>People</rdf:li>
          <rdf:li>Rocks</rdf:li>
          <rdf:li>Sky</rdf:li>
         </rdf:Bag>
        </dc:subject>
        <photoshop:DateCreated>1980-01-01T22:05:11.900000+02:00</photoshop:DateCreated>
</rdf:Description>
<rdf:Description rdf:about=""  
 xmlns:Iptc4xmpExt='http://iptc.org/std/Iptc4xmpExt/2008-02-29/'>
</rdf:Description>
<rdf:Description rdf:about="" 
 xmlns:digiKam='http://www.digikam.org/ns/1.0/'>
        <digiKam:TagsList>
        <rdf:Seq>
          <rdf:li>Blue Sky</rdf:li>
          <rdf:li>Outdoor</rdf:li>
          <rdf:li>People</rdf:li>
          <rdf:li>Rocks</rdf:li>
          <rdf:li>Sky</rdf:li>
        </rdf:Seq>
        </digiKam:TagsList>
</rdf:Description>
<rdf:Description rdf:about="" 
 xmlns:xmp='http://ns.adobe.com/xap/1.0/'>
        <xmp:CreateDate>1980-01-01T22:05:11</xmp:CreateDate>
        <xmp:ModifyDate>1980-01-01T22:05:11</xmp:ModifyDate>
        <xmp:DocumentID>1234-2211</xmp:DocumentID>
        <xmp:InstanceID>1234-2211</xmp:InstanceID>
</rdf:Description>
<rdf:Description rdf:about=""
 xmlns:exif='http://ns.adobe.com/exif/1.0/'>
</rdf:Description>
<rdf:Description rdf:about=""
 xmlns:mwg-rs="http://www.metadataworkinggroup.com/schemas/regions/"
 xmlns:stArea="http://ns.adobe.com/xmp/sType/Area#"
 xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#">
</rdf:Description>
<rdf:Description rdf:about=""
 xmlns:MP="http://ns.microsoft.com/photo/1.2/"
 xmlns:MPRI="http://ns.microsoft.com/photo/1.2/t/RegionInfo#"
 xmlns:MPReg="http://ns.microsoft.com/photo/1.2/t/Region#">
</rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>

[odedia]

Found it. The field that should be set is ImageUniqueID. Right now PhotoPrism only reads this field from exiftool and not XMP but I guess it can't hurt to set it in both for future proofing. ImageUniqueID must be a UUID for this to work, I believe that is the case with Apple Photos Identifiers, correct? This worked for me:

exiftool -overwrite_original -ImageUniqueID="51b2fbb7-7d5a-4d41-99c8-e96e1a1a0f25" Film0103.edited.jpeg
exiftool -overwrite_original -ImageUniqueID="51b2fbb7-7d5a-4d41-99c8-e96e1a1a0f25" Film0103.jpg
exiftool -overwrite_original -ImageUniqueID="51b2fbb7-7d5a-4d41-99c8-e96e1a1a0f25" Film0103.nef

Edit: It appears that DocumentID field is also supported. I ran the following commands on the 3 (original) output files and it got them stacked in PhotoPrism:

exiftool -overwrite_original -DocumentID="51b2fbb7-7d5a-4d41-99c8-e96e1a1a0f25" Film0103.edited.jpeg
exiftool -overwrite_original -DocumentID="51b2fbb7-7d5a-4d41-99c8-e96e1a1a0f25" Film0103.jpg
exiftool -overwrite_original -DocumentID="51b2fbb7-7d5a-4d41-99c8-e96e1a1a0f25" Film0103.nef

[RhetTbull]

I'll open a new issue to add this. OSXPhotos writes the same data with exiftool and to the XMP for simplicity. One thing to keep in mind is that if assets are exported to multiple folders (for example, same photo in multiple albums) then they'd all have the same document ID.

[odedia]

Thank you! Just a small clarification - when using —convert-to-jpeg, does it do a full render of the raw file or does that also use the cache?

[RhetTbull]

--convert-to-jpeg does a full render with default quality 1.0 (best) but this can be adjusted with --jpeg-quality

[RhetTbull]

@odedia I opened #1510 to track the addition of ImageUniqueID to the exiftool and XMP output. Not sure when I'll get to this. In the mean time, you could do the following to add the ImageUniqueID and DocumentID to the exported images:

osxphotos export /path/to/export --verbose --post-command exported "exiftool -overwrite_original -DocumentID='{photo.uuid}' -ImageUniqueID='{photo.uuid}' {filepath|shell_quote}" --update --ignore-signature

This uses the --post-command option to run exiftool against the exported files and add the ImageUniqueID and DocumentID tags. Note that doing this changes the signature of the file which will now be different than what OSXPhotos has recorded in the export database. This will cause all such modified photos to be re-exported on the next run with --update. Thus, my example above I've added --ignore-signature which instructs OSXPhotos to ignore the signature of the file on disk when deciding whether or not to update the file. The file will still be updated if it changed in Photos or if the metadata changed.

Edit to add more description for the --post-command: --post-command takes two arguments. The first is a keyword telling OSXPhotos which files to apply this command to. In this case we use exported to apply to any file that's been exported during this run by OSXPhotos. The second is the command, which is an OSXPhotos template string. We use {photo.uuid} to insert the UUID. (actually, {uuid} could also have been used as they mean the same thing). We use {filepath|shell_quote} to pass the path of the newly exported file to exiftool. The |shell_quote portion uses the shell_quote filter to ensure that if the filepath includes spaces or special characters, these are properly escaped to pass to exiftool.

If you really wanted to do the same for the XMP file, you could use --sidecar-template option. See osxphotos help export sidecar-template and the example XMP template.

[odedia]

Great idea! Will use that in the meantime, thank you.

[RhetTbull]

And if you wanted to apply this to an existing export, you could use skipped as the first argument to --post-command to run exiftool on all files that were skipped this run (previously exported). That would allow you to "catch up" an existing export.

[RhetTbull]

@all-contributors please add @odedia for ideas, research

[allcontributors[bot]]

@RhetTbull

I've put up a pull request to add @odedia! :tada: