Exif of exported photos are 1 second off from the original and when to use --exiftool or not

[nicad]

~/.local/bin/osxphotos --version
osxphotos, version 0.64.3
Python 3.10.9 (main, Dec 15 2022, 18:18:30) [Clang 14.0.0 (clang-1400.0.29.202)]
macOS 14.1.1, x86_64

Export done with: osxphotos export /my/path --no-progress --export-by-date --exiftool --update --touch-file

(but probably has always been happening)

I compare a random file from ~/Pictures/Photos Library.photoslibrary/originals with the one exported by osxphotos. I use exiftool to dump their metadata. This is the output diff-ing 'exiftool FILENAME':

diff /tmp/a /tmp/b
2,3c2,3
< File Name                       : 96E465E0-5F5F-4069-B9E6-02ED91067220.heic
< Directory                       : ./9
---
> File Name                       : IMG_6860.HEIC
> Directory                       : .
5,7c5,7
< File Modification Date/Time     : 2019:07:10 16:28:20-07:00
< File Access Date/Time           : 2022:08:27 12:01:09-07:00
< File Inode Change Date/Time     : 2021:06:13 13:54:55-07:00
---
> File Modification Date/Time     : 2019:07:10 09:37:15-07:00
> File Access Date/Time           : 2019:07:10 09:37:15-07:00
> File Inode Change Date/Time     : 2022:08:14 17:42:45-07:00
26c26
< Modify Date                     : 2019:07:10 18:37:16
---
> Modify Date                     : 2019:07:10 18:37:15
33,34c33,35
< Date/Time Original              : 2019:07:10 18:37:16
< Create Date                     : 2019:07:10 18:37:16
---
> Date/Time Original              : 2019:07:10 18:37:15
> Create Date                     : 2019:07:10 18:37:15
> Offset Time Original            : +02:00
133c134
< Media Data Size                 : 1642577
---
> Media Data Size                 : 1642595
141,142c142,143
< Create Date                     : 2019:07:10 18:37:16.132
< Date/Time Original              : 2019:07:10 18:37:16.132
---
> Create Date                     : 2019:07:10 18:37:15.132
> Date/Time Original              : 2019:07:10 18:37:15.132+02:00

Note how "Date/Time Original" and "Create Date" are off by 1 second. All the random photos I've been checking seem to have this oddity but I didn't do anything systematic.

I suspect this is happening because I use exiftool which recreates exif data. But then: (1) why is it off by 1s exactly (and at all) ? (2) should I use exiftool at all if the originals already seem to contain the same exif data that --exiftool adds ? (without --exiftool I could then use --export-as-hardlink to use apfs clones and then save disk space + probably much faster export speed) (3) when is --touch-file useful ? in the example above the file's ctime/mtime are correct but maybe it's not always the case ?

This difference in exif makes it hard to reconcile backups taken at different point in time using different methods (I used to have my own rsync script in older versions of iPhoto/Photo app) and I'm trying to consolidate all files from all backups.

[nicad]

I think I can answer (3): file timestamp probably won't match exif when setting up a new macOS machine, it will likely set its own ctime/mtime of when it downloaded the file ? (even though I can't quite explain why a lot of time the exif and file timestamps are the same, my laptop is not always online). So --touch-file is probably best to use to have same file/exif time and because it changes the inode itself apfs clones will still save space but export time will be slower because exif needs to be read.

[RhetTbull]

(1) why is it off by 1s exactly (and at all) ?

Hmm, I've not noticed this before. Will see if I can recreate. If not, will send you some debug commands to run.

should I use exiftool at all if the originals already seem to contain the same exif data that --exiftool adds ?

The primary reason I use --exiftool is to push the keywords and PersonInImage to the file. If you don't care about this and if most of your photos originate from iPhone or camera which embeds EXIF then you don't need --exiftool. If you have adjusted dates/times/locations, etc. because you imported a lot of photos from cameras that don't embed this data, then it might also be helpful. It's really up to you based on what your intended use of the export is. Note that you can always apply the exiftool changes later if you change your mind using the osxphotos exiftool command. If having the keywords and persons embedded in the image is not important to you then I would skip this. As an alternative you can use --sidecar xmp which creates an XMP sidecar with the same metadata that can later be applied with tools like exiftool or Lightroom, Digikam, etc.

I could then use --export-as-hardlink to use apfs clones and then save disk space

On APFS you don't even need to use --export-as-hardlink as osxphotos will automatically use a native "copy-on-write" copy method.

when is --touch-file useful ?

The file modification date in Photos is not reliable. For example, it will reflect the date a photo was downloaded from iCloud not the date the asset was actually modified. --touch-file sets the modification date to match the EXIF data of the image (as reflected in the Photos database, not the actual EXIF data as the user may have changed the date in Photos) and this allows you to correctly sort files by date in Finder.

[nicad]

Thank you for this very useful answer.

[RhetTbull]

@nicad I have not been able to reproduce this. I've tried against multiple files in my library and the time matches the original file when using exiftool. In one photo, a photo without timezone had been imported so Photos adds the timezone (it assumes local computer timezone if not specified in EXIF) and so the exported file shows the timezone applied but the time is identical down to millisecond.

< [Composite]     Date/Time Original              : 2015:02:26 19:25:22.662
---
> [Composite]     Date/Time Original              : 2015:02:26 19:25:22.662-06:00

If you'd like to provide some additional debug data, you can run the following command:

osxphotos debug-dump --dump photos --uuid 96E465E0-5F5F-4069-B9E6-02ED91067220 > debug.log

And send me the debug.log or post here (it will contain location data of the photo in case that is sensitive -- you could strip this before sending as it's just a text file). This debug file will allow me to see what's in the Photos database for the time value to see if it's a problem with Photos, osxphotos, or exiftool.

[nicad]

Nothing sensitive, here is the log, I can also give you the photo itself if you want, some random landscape.

debug.log

[RhetTbull]

@nicad thanks for the debug info. If you don't mind sending the original photo to osxphotos.py@gmail.com I'll take a look. Something is weird with this. Not accounting for timezone (GMT+02:00) which keeps things simpler, the timestamp in the database is ZDATECREATED': 584469435.713315 which is seconds since 2001-01-01, which Apple uses a the time epoch. Converting this to a date/time yields:

>>> import datetime
>>> # Time delta: add this to Photos times to get unix time
>>> # Apple Epoch is Jan 1, 2001
>>> TIME_DELTA = (datetime.datetime(2001, 1, 1, 0, 0) - datetime.datetime(1970, 1, 1, 0, 0)).total_seconds()
>>> timestamp = 584469435.713315
>>> date = datetime.datetime.fromtimestamp(timestamp + TIME_DELTA)
>>> date
datetime.datetime(2019, 7, 10, 9, 37, 15, 713315)
>>> date.isoformat()
'2019-07-10T09:37:15.713315'
>>>

(This is in my local timezone, not GMT+02:00 -- the offset is 'ZTIMEZONEOFFSET': 7200 or 7200 seconds, 2 hours from GMT)

I am very confident this is correct as I've done extensive testing on this.

But notice the microseconds are off. Your data shows:

< Create Date                     : 2019:07:10 18:37:16.132
< Date/Time Original              : 2019:07:10 18:37:16.132
---
> Create Date                     : 2019:07:10 18:37:15.132
> Date/Time Original              : 2019:07:10 18:37:15.132+02:00

So the value written by exiftool matches what's in the Photos database to the second (2019:07:10 18:37:15.132+02:00) but the microseconds don't match which is weird. The data encoded in the EXIF data (as read by Photos at time of import and recorded in the database) is:

        'DateTimeOriginal': '2019:07:10 18:37:16',
        'DateTimeDigitized': '2019:07:10 18:37:16',
        'SubsecTimeDigitized': '132',

And this matches what you're seeing from exiftool but not what is recorded in the database.

It would be helpful to monitor the database as Photos imports this on my end and also play with exiftool to see what's going on.

[oPromessa]

    'imageDate_timestamp': 584469435.713315,
    'imageDate': datetime.datetime(2019, 7, 10, 18, 37, 15, 713315,

Would it be a rounding topic?

cloud_metadata': {
        '{TIFF}': {
            'ResolutionUnit': 2,
            'Software': '12.3.1',
            'TileLength': 512,
            'DateTime': '2019:07:10 18:37:16',

(...)

            'Model': 'iPhone X',
            'Make': 'Apple'
        },
        '{Exif}': {
            'DateTimeOriginal': '2019:07:10 18:37:16',

open rambling

I've seen:

• different times (and one second apart in some cases for some iPhone models and in others hours when they're taken in another time zone)
• in the several date fields (original, created, modified, keys:date) -- and don't get me started on videos!!
• as well as sometimes original date time is not defined
• then Photos picks and chooses the one it thinks adequate ...so I try to equalize all dates (according to some logic of mine!) and add all missing date fields via exiftool before loading into Photos (for pics and videos and status away of PNG and converting older video formats which do no support all the date fields and EXIF data) ... and on export use exiftool and touch extensively!

[RhetTbull]

@nicad thanks for sending the photo. Unfortunately I've not been able to recreate this nor come up with any clues as to why you see this behavior. When I import the photo the value stored in Photos and thus the value exported by osxphotos is correct and does not match the "1 second off" value in your database (data below).

I'm not sure why this happened but you can use the osxphotos timewarp tool to see how many photos this affects and correct them if you wish.

Select some photos in Photos then run osxphotos timewarp --compare-exif. This will show any differences between the time in Photos and that in the EXIF data.

Then to correct them, osxphotos timewarp --pull-exif --verbose

Original (exiftool)
[EXIF]          Date/Time Original              : 2019:07:10 18:37:16
[EXIF]          Create Date                     : 2019:07:10 18:37:16
[Composite]     Create Date                     : 2019:07:10 18:37:16.132
[Composite]     Date/Time Original              : 2019:07:10 18:37:16.132

After import:
The value in ZDATECREATED is 584469436.132 which is Wednesday, July 10, 2019 at 4:37:16.132 PM (in my local time)
osxphotos inspect: 2019-07-10T18:37:16.132000+02:00

In your debug log, the ZDATECREATED value is: 'ZDATECREATED': 584469435.713315 and this is indeed 1 second off.

When exported (exiftool):
[EXIF]          Date/Time Original              : 2019:07:10 18:37:16
[EXIF]          Create Date                     : 2019:07:10 18:37:16
[Composite]     Create Date                     : 2019:07:10 18:37:16.132
[Composite]     Date/Time Original              : 2019:07:10 18:37:16.132+02:00

Sidecar:
XMP: <photoshop:DateCreated>2019-07-10T18:37:16.132000+02:00</photoshop:DateCreated>
JSON: "DateTimeOriginal": "2019:07:10 18:37:16", "CreateDate": "2019:07:10 18:37:16", "OffsetTimeOriginal": "+02:00", "DateCreated": "2019:07:10", "TimeCreated": "18:37:16+02:00",

[nicad]

Ok, thank you for looking at it so thoroughly. I've been sidetracked but I'll check when I get back at organizing my photo library.