Closed oPromessa closed 4 months ago
Great debug info -- thanks! My first guess would've been photoinfo was changing (labels are sometimes re-computed by Photos) but your step 6 above shows that's not the case.
Does the file 1969-11-09-1.jpg
always get re-exported? If so, that'll make tracking this down easier.
How familiar are you with the python debugger? I've built debugger access into the current github version (not yet released) so if you're comfortable trying a bleeding edge feature, you can use your clone of osxphotos, do a git pull
, then try the following:
OSXPHOTOS_IS_TESTING=1 python3 -m osxphotos export /Volumes/photo-1/(...) --exportdb ./.osxphotos_export.db --load-config ./colibri.toml --name "1969-11-09-1" --breakpoint osxphotos.photoexporter.PhotoExporter._should_update_photo
That will drop you into the debugger when osxphotos is about to call the _should_update_photo method to determine if the photo should be updated. Hit s
to step into the the _should_update_photo
then n
to step through each of the tests to see which test is returning True
. Here's an example where I added a keyword forcing the exiftool check to return True:
⇣41% [I] ➜ OSXPHOTOS_IS_TESTING=1 python3 -m osxphotos export ~/Desktop/export --exiftool --touch-file --update --name 3092 --breakpoint osxphotos.photoexporter.PhotoExporter._should_update_photo
Breakpoint added for osxphotos.photoexporter.PhotoExporter._should_update_photo
Using last opened Photos library: /Users/rhet/Pictures/Test-10.15.7.photoslibrary
Exporting 1 photo to /Users/rhet/Desktop/export...
[16] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/debug.py(48)debug_breakpoint()
-> return wrapped(*args, **kwargs)
(Pdb++) s
--Call--
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(681)_should_update_photo()
-> def _should_update_photo(
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(685)_should_update_photo()
-> export_db = options.export_db
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(686)_should_update_photo()
-> fileutil = options.fileutil
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(688)_should_update_photo()
-> file_record = export_db.get_file_record(dest)
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(690)_should_update_photo()
-> if not file_record:
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(694)_should_update_photo()
-> if options.export_as_hardlink and not dest.samefile(src):
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(698)_should_update_photo()
-> if not options.export_as_hardlink and dest.samefile(src):
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(702)_should_update_photo()
-> if not options.ignore_signature and not fileutil.cmp_file_sig(
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(703)_should_update_photo()
-> dest, file_record.dest_sig
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(702)_should_update_photo()
-> if not options.ignore_signature and not fileutil.cmp_file_sig(
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(708)_should_update_photo()
-> if file_record.export_options != options.bit_flags:
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(715)_should_update_photo()
-> if options.exiftool:
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(716)_should_update_photo()
-> current_exifdata = self._exiftool_json_sidecar(options=options)
(Pdb++) n
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(717)_should_update_photo()
-> return current_exifdata != file_record.exifdata
(Pdb++) n
--Return--
[17] > /Users/rhet/Dropbox/Code/osxphotos/osxphotos/photoexporter.py(717)_should_update_photo()->True
Notice that in step [17]
, _should_update_photo
returned True and the test immediately preceding was return current_exifdata != file_record.exifdata
showing the new exifdata differed from the old exifdata.
Update: You can then hit c
to continue the program.
1969-11-09-1.jpg
via it's uuid
._should_update_photo
returns FALSE smb
the function returns TRUE on this line:
https://github.com/RhetTbull/osxphotos/blob/adb90a3364eabb0c4c96ca1f7c0743360a926050/osxphotos/photoexporter.py#L702
[17] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/photoexporter.py(698)_should_update_photo()
-> if not options.export_as_hardlink and dest.samefile(src):
(Pdb++) n
[17] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/photoexporter.py(702)_should_update_photo()
-> if not options.ignore_signature and not fileutil.cmp_file_sig(
(Pdb++) n
[17] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/photoexporter.py(703)_should_update_photo()
-> dest, file_record.dest_sig
(Pdb++) n
[17] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/photoexporter.py(702)_should_update_photo()
-> if not options.ignore_signature and not fileutil.cmp_file_sig(
(Pdb++) n
[17] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/photoexporter.py(706)_should_update_photo()
-> return True
(Pdb++) n
--Return--
[17] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/photoexporter.py(706)_should_update_photo()->True
-> return True
(Pdb++) n
--Return--
[16] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/debug.py(48)debug_breakpoint()->True
-> return wrapped(*args, **kwargs)
(Pdb++) c
Processed: 1 photo, exported: 0, updated: 1, skipped: 0, updated EXIF data: 1, missing: 0, error: 0, touched date: 1
Elapsed time: 0:01:19
[17] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/photoexporter.py(702)_should_update_photo()
-> if not options.ignore_signature and not fileutil.cmp_file_sig(
(Pdb++) dest
PosixPath('/Volumes/photo-1/export/1969-11-09-1.jpg')
(Pdb++) file_record.dest_sig
(32768, 73826, -4535100)
(Pdb++) file_record
<osxphotos.export_db.ExportRecord object at 0x162f4c9c0>
(Pdb++) type(file_record)
<class 'osxphotos.export_db.ExportRecord'>
(Pdb++) s
[17] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/photoexporter.py(703)_should_update_photo()
-> dest, file_record.dest_sig
(Pdb++) s
--Call--
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/export_db.py(844)dest_sig()
-> @property
(Pdb++) n
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/export_db.py(847)dest_sig()
-> conn = self._conn
(Pdb++) n
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/export_db.py(848)dest_sig()
-> c = conn.cursor()
(Pdb++) n
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/export_db.py(849)dest_sig()
-> row = c.execute(
(Pdb++) n
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/export_db.py(850)dest_sig()
-> "SELECT dest_mode, dest_size, dest_mtime FROM export_data WHERE filepath_normalized = ?;",
(Pdb++) n
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/export_db.py(851)dest_sig()
-> (self._filepath_normalized,),
(Pdb++) n
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/export_db.py(849)dest_sig()
-> row = c.execute(
(Pdb++) n
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/export_db.py(853)dest_sig()
-> if row:
(Pdb++) row
(32768, 73826, -4535100.0)
I ran os.stat on the two 1969-11-09-1.jpg
files local and remote
I guess the negative value on st_mtime may cause some disruption...
But on the other hand it will also be saved on the database and the compare would still match.
$ python
Python 3.9.10 (main, Jan 15 2022, 11:48:04)
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import stat
>>> import os
>>> f1 = "/Volumes/photo-1/export/1969-11-09-1.jpg"
>>> f2 = "/Users/MSP/Documents/GitHub/osxphotos/export/1969-11-09-1.jpg"
>>> os.stat(f1)
os.stat_result(st_mode=33216, st_ino=2793585, st_dev=922750023, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646773217, st_mtime=1646773217, st_ctime=1646773217)
>>> os.stat(f2)
os.stat_result(st_mode=33188, st_ino=19254388, st_dev=16777223, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646777019, st_mtime=-4535100, st_ctime=1646773275)
diff indicates the file are the same:
$ diff -s /Users/MSP/Documents/GitHub/osxphotos/export/1969-11-09-1.jpg /Volumes/photo-1/export//1969-11-09-1.jpg
Files /Users/MSP/Documents/GitHub/osxphotos/export/1969-11-09-1.jpg and /Volumes/photo-1/export//1969-11-09-1.jpg are identical
``
A few thoughts/questions:
osxphotos exportdb ./.osxphotos_export.db --sql "select src_mtime, dest_mtime from export_data where filepath_normalized like '%1969-11-09-1%'"
A few thoughts/questions:
- What's the date on the photo in Photos? Is it possible there's an invalid date that is causing an overflow error on the NAS when touching the file? Does the date of the exported file on the NAS match the date in Photos?
1969-11-09 13:15:00 (format YYYY-MM-DD HH:MM:SS)
- What's the result of running os.stat repeatedly on the file on the NAS? (e.g. does it always give the same result?)
- Here I thought the negative in
st_mtime
was coming from the NAS (f2) but it's coming from the MAC disk (f1)!>> import os >> f1 = "/Volumes/photo-1/export/1969-11-09-1.jpg"
os.stat(f1) os.stat_result(st_mode=33216, st_ino=2793585, st_dev=922750909, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646773217, st_mtime=1646773217, st_ctime=1646773217) os.stat(f1) os.stat_result(st_mode=33216, st_ino=2793585, st_dev=922750909, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646773217, st_mtime=1646773217, st_ctime=1646773217) (...) os.stat(f1) os.stat_result(st_mode=33216, st_ino=2793585, st_dev=922750909, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646773217, st_mtime=1646773217, st_ctime=1646773217)
>>> f2 = "/Users/MSP/Documents/GitHub/osxphotos/export/1969-11-09-1.jpg"
>>> os.stat(f2)
os.stat_result(st_mode=33188, st_ino=19254388, st_dev=16777223, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646817645, st_mtime=-4535100, st_ctime=1646773275)
>>> os.stat(f2)
os.stat_result(st_mode=33188, st_ino=19254388, st_dev=16777223, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646817645, st_mtime=-4535100, st_ctime=1646773275)
>>> os.stat(f2)
os.stat_result(st_mode=33188, st_ino=19254388, st_dev=16777223, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646817645, st_mtime=-4535100, st_ctime=1646773275)
>>> os.stat(f2)
os.stat_result(st_mode=33188, st_ino=19254388, st_dev=16777223, st_nlink=1, st_uid=501, st_gid=20, st_size=73826, st_atime=1646817645, st_mtime=-4535100, st_ctime=1646773275)
(...)
>>>
- What's the output of the following command?
osxphotos exportdb ./.osxphotos_export.db --sql "select src_mtime, dest_mtime from export_data where filepath_normalized like '%1969-11-09-1%'"
- On the local MAC disk
$ osxphotos exportdb ./.osxphotos_export.db --sql "select src_mtime, dest_mtime from export_data where filepath_normalized like '%1969-11-09-1%'" (1472486297.0, -4535100.0)
- On the NAS Disk:
$ osxphotos exportdb ./.osxphotos_export.db --sql "select src_mtime, dest_mtime from export_data where filepath_normalized like '%1969-11-09-1%'" (1472486297.0, 1646773217.0)
- Additionally see the output of exit tool for dates on the three fies: 5.
- The original one: /Users/MSP/Pictures
- The exported to the NAS: /Volumes/photo-1/
- The exported locally: /Users/MSP/Documents/GitHub/osxphotos/export
$ alias exifviewdates='exiftool -a -G1 -s -api QuickTimeUTC=1 -time:all -api RequestAll=2'
$ exifviewdates "/Users/MSP/Pictures/(...)/1969-11-09-1.jpg"
[ExifTool] Now : 2022:03:09 09:16:34+00:00
[System] FileModifyDate : 2016:08:29 16:58:17+01:00
[System] FileAccessDate : 2021:12:24 19:00:57+00:00
[System] FileInodeChangeDate : 2021:12:24 19:00:57+00:00
[MacOS] FileCreateDate : 2016:08:29 16:58:17+01:00
[MacOS] MDItemContentCreationDate : 2016:08:29 16:58:17+01:00
[MacOS] MDItemContentCreationDate_Ranking: 2016:08:29 01:00:00+01:00
[MacOS] MDItemContentModificationDate : 2016:08:29 16:58:17+01:00
[MacOS] MDItemContentModificationDate_Ranking: 2016:08:29 01:00:00+01:00
[MacOS] MDItemDateAdded : 2021:12:24 19:00:57+00:00
[MacOS] MDItemDateAdded_Ranking : 2021:12:24 00:00:00+00:00
[MacOS] MDItemFSContentChangeDate : 2016:08:29 16:58:17+01:00
[MacOS] MDItemFSCreationDate : 2016:08:29 16:58:17+01:00
[MacOS] MDItemInterestingDate_Ranking : 2016:08:29 01:00:00+01:00
[IFD0] ModifyDate : 1969:11:09 13:15:00
[ExifIFD] DateTimeOriginal : 1969:11:09 13:15:00
[ExifIFD] CreateDate : 1969:11:09 13:15:00
$ exifviewdates /Volumes/photo-1/export/1969-11-09-1.jpg
[ExifTool] Now : 2022:03:09 09:17:50+00:00
[System] FileModifyDate : 2022:03:08 21:00:17+00:00
[System] FileAccessDate : 2022:03:08 21:00:17+00:00
[System] FileInodeChangeDate : 2022:03:08 21:00:17+00:00
[MacOS] FileCreateDate : 2016:08:29 16:58:17+01:00
[MacOS] MDItemFSContentChangeDate : 2022:03:08 21:00:17+00:00
[MacOS] MDItemFSCreationDate : 1970:01:01 00:00:00+00:00
[IFD0] ModifyDate : 1969:11:09 13:15:00
[ExifIFD] DateTimeOriginal : 1969:11:09 13:15:00
[ExifIFD] CreateDate : 1969:11:09 13:15:00
[ExifIFD] OffsetTimeOriginal : +01:00
[IPTC] DateCreated : 1969:11:09
[IPTC] TimeCreated : 13:15:00+01:00
[Composite] SubSecDateTimeOriginal : 1969:11:09 13:15:00+01:00
[Composite] DateTimeCreated : 1969:11:09 13:15:00+01:00
$ exifviewdates /Users/MSP/Documents/GitHub/osxphotos/export/1969-11-09-1.jpg
[ExifTool] Now : 2022:03:09 09:18:33+00:00
[System] FileModifyDate : 1969:11:09 13:15:00+01:00
[System] FileAccessDate : 2022:03:09 09:18:32+00:00
[System] FileInodeChangeDate : 2022:03:08 21:01:15+00:00
[MacOS] FileCreateDate : 2016:08:29 16:58:17+01:00
[MacOS] MDItemContentCreationDate : 1969:11:09 13:15:00+01:00
[MacOS] MDItemContentCreationDate_Ranking:
[MacOS] MDItemContentModificationDate : 1969:11:09 13:15:00+01:00
[MacOS] MDItemDateAdded : 2022:03:08 21:01:15+00:00
[MacOS] MDItemDateAdded_Ranking : 2022:03:08 00:00:00+00:00
[MacOS] MDItemFSContentChangeDate : 1969:11:09 13:15:00+01:00
[MacOS] MDItemFSCreationDate : 2016:08:29 16:58:17+01:00
[MacOS] MDItemInterestingDate_Ranking :
[IFD0] ModifyDate : 1969:11:09 13:15:00
[ExifIFD] DateTimeOriginal : 1969:11:09 13:15:00
[ExifIFD] CreateDate : 1969:11:09 13:15:00
[ExifIFD] OffsetTimeOriginal : +01:00
[IPTC] DateCreated : 1969:11:09
[IPTC] TimeCreated : 13:15:00+01:00
[Composite] SubSecDateTimeOriginal : 1969:11:09 13:15:00+01:00
[Composite] DateTimeCreated : 1969:11:09 13:15:00+01:00
ls -la
for all 3 files also
638354 144 -rw-r--r--@ 1 MSP wheel 69K 29 Aug 2016 /Users/MSP/Pictures/(...)/1969-11-09-1.jpg
2793585 146 -rwx------@ 1 MSP staff 72K 8 Mar 21:00 /Volumes/photo-1/export/1969-11-09-1.jpg
19254388 152 -rw-r--r--@ 1 MSP staff 72K 9 Nov 1969 /Users/MSP/Documents/GitHub/osxphotos/export/1969-11-09-1.jpg
Here I thought the negative in st_mtime was coming from the NAS (f2) but it's coming from the MAC disk (f1)!
unix dates start on 1-Jan-1970 and since this photo is taken from 1969, that's expected.
This is interesting:
On the NAS, the dates aren't getting set correctly for this pre-1970 date:
[System] FileModifyDate : 2022:03:08 21:00:17+00:00 <---- should be 1969
[MacOS] MDItemFSCreationDate : 1970:01:01 00:00:00+00:00 <----- should be 1969
On the Mac:
[System] FileModifyDate : 1969:11:09 13:15:00+01:00
[MacOS] MDItemContentCreationDate : 1969:11:09 13:15:00+01:00
Do all the files that keep getting re-exported have a date from before 1970? It looks like the NAS is not correctly setting the date for this file. osxphotos is using os.utime
which is the system method that touch
uses.
What happens if you try to use touch
on the file? That is: touch /Volumes/photo-1/export/1969-11-09-1.jpg
Does it successfully update the file date/time?
Do all the files that keep getting re-exported have a date from before 1970? It looks like the NAS is not correctly setting the date for this file. osxphotos is using
os.utime
which is the system method thattouch
uses.
- Yes. All pre-1970 files get re-exported. A few after 1970 (not always the same... so I'll look into them later).
What happens if you try to use
touch
on the file? That is:touch /Volumes/photo-1/export/1969-11-09-1.jpg
Does it successfully update the file date/time?
- Touch seems to work and being able to set date/time prior to 1970.
$ touch -d 1969-11-09T13:15:00 1969-11-09-1.jpg ; $ ls -lisahG -T 1969-11-09-1.jpg 2801415 146 -rwx------ 1 MSP staff 73K 9 Nov 13:15:00 1969 1969-11-09-1.jpg
- os.stat produces the negative number... but it seems it is understood, at least by
datetime.datetime.fromtimestamp()
correctly. Still in the debugger...(Pdb++) f1 PosixPath('/Volumes/photo-1/export/1969-01-05 1969-72 Digitalizadas (Album 2)/1969-11-09-1.jpg') (Pdb++) cls._sig(os.stat(f1)) (32768, 74328, -4535100) (Pdb++) datetime.datetime.fromtimestamp(-4535100) datetime.datetime(1969, 11, 9, 13, 15)
mtime = correct negative time
and recorded on the database (s2)
2 a subsequent run with os.stat gets mtime = exported time
(s1)
3 s2 (database) is different from s1 (current file exported) so it gets updated.
(Pdb++) s2
(32768, 74328, -4535100)
(Pdb++) datetime.datetime.fromtimestamp(s2[2])
datetime.datetime(1969, 11, 9, 13, 15) <---- actual date/time in database.
(Pdb++) s1
(32768, 74328, 1646850845)
(Pdb++) datetime.datetime.fromtimestamp(s1[2])
datetime.datetime(2022, 3, 9, 18, 34, 5) <---- file mtime from exported date/time
os.utime
to double check its behaviour..
What I believe it seems to be happening: a subsequent run with os.stat gets mtime = exported time (s1)
I don't think this can happen in the code. See below -- the files are touched then after that call to _touch_files
, the destination signature is updated in the database with a call to fileutil.file_sig
:
Here's the code for _touch_files
:
And here's file_sig
which is calling os.stat
on the destination file:
https://github.com/RhetTbull/osxphotos/blob/2e501e6a9b517f8c520509053b3ffcd9d57f139f/osxphotos/fileutil.py#L195-L198
Will test out also os.utime to double check its behaviour.
This looks like utime is not correctly setting the modification time on the NAS. If you can confirm that, one possible work around is to have a fallback to make subprocess call directly to touch
. This would be definitely be much slower so would only want to use when needed.
$ ll -T 1969-11-09-1.jpg
2800251 146 -rwx------ 1 MSP staff 73K 9 Mar 17:01:28 2022 1969-11-09-1.jpg
$ touch -d 1969-11-09T13:15:00 1969-11-09-1.jpg
$ ll -T 1969-11-09-1.jpg
2800251 146 -rwx------ 1 MSP staff 73K 9 Nov 13:15:00 1969 1969-11-09-1.jpg
$ sleep 10; ll -T 1969-11-09-1.jpg
2800251 146 -rwx------ 1 MSP staff 73K 9 Nov 13:15:00 1969 1969-11-09-1.jpg
$ sleep 10; ll -T 1969-11-09-1.jpg
2800251 146 -rwx------ 1 MSP staff 73K 9 Mar 17:01:28 2022 1969-11-09-1.jpg <--- date is REVERTED BACK!!!!
$
$ ls -l --full-time 1969-11-09-1.jpg
-rwxrwxrwx 1 ruler users 70682 2016-08-29 16:58:17.000000000 +0100 1969-11-09-1.jpg
$ touch -d 1969-11-09T13:15:00 1969-11-09-1.jpg
$ ls -l --full-time 1969-11-09-1.jpg
-rwxrwxrwx 1 ruler users 70682 1969-11-09 13:15:00.000000000 +0100 1969-11-09-1.jpg
$ sleep 30; ls -l --full-time 1969-11-09-1.jpg
-rwxrwxrwx 1 ruler users 70682 1969-11-09 13:15:00.000000000 +0100 1969-11-09-1.jpg
$ ll -T 1969-11-09-1.jpg
2801415 146 -rwx------ 1 MSP staff 73K 9 Nov 13:15:00 1969 1969-11-09-1.jpg
os.utime
has some issue updating a remote file (via smb protocol) with dates prior to 1970!!!So the os.utime has some issue updating a remote file (via smb protocol) with dates prior to 1970!!!
I think you're right that this appears to be the problem.
Opened up a ticket with Synology. Let's see!
I added a fix for this. It's a bit of a hack but should work: after using utime
to touch the file time, the code will do a stat to verify the utime was successful. If not successful, it'll do a subprocess call to /usr/bin/touch
to set the time. I was concerned the extra stat would slow things down but in my testing when exporting across the network, it added less than a millisecond. Try a git pull
and see if this new version fixes the problem. (I've not yet created a new release)
I'm just concerned the end/result will be the same as the same issue occurs with /usr/bin/touch
It would have to be the touch command executed remotely on the NAS.
The hack I was thinking was to use .osxphotos_db
to find files < 1970 and run touch directly in the NAS to make the change permanent!
I've tested if and it seems to work.
Of course is a mess of a hack. Or maybe I could use the --post-command
option to run a rsh
(remote shell) command.
2022-03-10 15:53:19.592341 -- Exporting 1 photo to /Volumes/photo-1/export...
2022-03-10 15:53:19.612506 -- 2022-03-10 15:53:19.612396 -- Exporting 1969-11-09-1.jpg (1969-11-09-1.jpg) as 1969-11-09-1.jpg (1/1)
[16] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/debug.py(48)debug_breakpoint()
-> return wrapped(*args, **kwargs)
(...)
(Pdb++) n
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/fileutil.py(193)cmp_file_sig()
-> return s1 == s2
(Pdb++) s1
(32768, 74328, -4535100)
(Pdb++) s2
(32768, 74328, -4535100)
(Pdb++) s
--Return--
[18] > /Users/MSP/Documents/GitHub/osxphotos/osxphotos/fileutil.py(193)cmp_file_sig()->True
-> return s1 == s2
(Pdb++) c
2022-03-10 15:53:32.852227 -- 2022-03-10 15:53:32.851954 -- Skipped up to date file /Volumes/photo-1/export/1969-01-05 1969-72 Digitalizadas (Album 2)/1969-11-09-1.jpg
2022-03-10 15:53:32.853696 -- Processed: 1 photo, exported: 0, updated: 0, skipped: 1, updated EXIF data: 0, missing: 0, error: 0, touched date: 0
2022-03-10 15:53:32.854218 -- Elapsed time: 0:00:13
but it then reverts back.
Ahh...I misunderstand your notes above then--I thought touch worked but utime didn't.
I'll take those changes out then as there's no added benefit.
I would use the --post-function
instead of --post-command
because you'll get the PhotoInfo object for the photo and from that you can grab the created time to run the rsh touch
For your post function, there's an example here: https://github.com/RhetTbull/osxphotos/blob/master/examples/post_function.py
You could do something like this in your post function:
def post_function(
photo: PhotoInfo, results: ExportResults, verbose: callable, **kwargs
):
for filename in results.exported:
if photo.date.year < 1970:
touch_date = photo.date.strftime("%Y%m%d%H%M.%S")
# not sure what rsh command will be
subprocess.run(f"rsh... touch {touch_date}")
Any luck with this issue? As I recall, it was an issue with your NAS not osxphotos?
Thanks for the reminder. I've been living with this for some time now and was about to accept it.
But with your reminder I did some digging up...
As we know he have two angles on this:
touch
command from MAC (where osxphotos runs) into a Synology NAS server does not really update the mtime of the file.touch -d
command directly on the file. So, I believe good news...
So I was expecting this to work on other files.
To allow cmp_file_sig
to do the correct comparison and mtime
would be the same for file and record db. But found two issues:
mtime
set incorrectly which will always result in a different file and running (not the full export of the file, luckily) the touch o the file (which actually will fail as it is on a remote folder!)But now I found on some files that the mtime in the exportdb
is incorrect for my test files.It is not taking the actually
sqlite> SELECT dest_mode, dest_size, dest_mtime, filepath_normalized FROM export_data ;
32768|120687|-93437720.0|(...)/1966-12-25-1.jpg
(...)
32768|787087|1658667844.0|(...)/sem título-2.jpg
On the debug I have for "sem título-2.jpg" the difference form the database and the os.stat:
739
740 -> if not options.ignore_signature and not fileutil.cmp_file_sig(
741 dest, file_record.dest_sig
742 ):
743 # destination file doesn't match what was last exported
744 return ShouldUpdate.DEST_SIG_DIFFERENT
745
(Pdb) dest
PosixPath('/Volumes/photo-1/Y/(...)/Sem título-2.jpg')
(Pdb) file_record.dest_sig
(32768, 787087, 1658667844)
(Pdb) os.stat(dest).st_mtime
-11955225.0
So I did a stat
and an exiftool
on the original files and some of the date entries are empty or void. that may scare things up a bit.
$ exif 1966-12-25-1.jpg Sem\ título-2.jpg 2>/dev/null
Directory|FileName|Make%MajorBrand|DateTimeOriginal|FileModifyDate|CreateDate|Keywords|Subject|ImageOrientation|Caption-Abstract|ModifyDate|Description|QuickTimeCreateDate|QuickTimeModifyDate|KeysCreationDate|RegionType|RegionName|
.|1966-12-25-1.jpg|-%-|1967:01:15 14:04:40|2015:09:01 12:17:03|1967:01:15 14:04:40|-|-|-|-|1967:01:15 14:04:40|-|-|-|-|-|-|
.|Sem título-2.jpg|-%-|1969:08:15 16:06:15|0000:00:00 00:00:00|1969:08:15 16:06:15|(...)|(...)|-|-|1969:08:15 16:06:15|-|-|-|-|-|-|
$ stat 1966-12-25-1.jpg Sem\ título-2.jpg
16777223 638194 -rw-r--r-- 1 MSP wheel 0 120681 "Jul 24 13:44:58 2022" "Sep 1 12:17:03 2015" "Jul 24 13:44:58 2022" "Sep 1 12:17:03 2015" 4096 240 0 (...)/1966-12-25-1.jpg
16777223 603359 -rw-r--r-- 1 MSP wheel 0 786126 "Dec 24 17:45:28 2021" "Jan 1 01:00:00 1970" "Dec 24 17:45:28 2021" "Jan 1 01:00:00 1970" 4096 1536 0 (...) Sem título-2.jpg
I am going to expand my test cases and check the full flow:
Sorry to bother you with is... I'm using this thread to collect my thoughts!Possibly shoudn't! Will be doing more investigation on my end prior to my next post!
It's odd you're getting a different result with os.stat and the database. The NAS must be doing something to change the time (or not propagate the changes made by osxphotos)The exportdb record is set with the following code:
The file_sig
method referenced in this code in turn uses os.stat
to get the signature but converts mtime to int because the Mac doesn't always preserve the fractional portion of mtime when copying. However, the internal compare function (cmp_file_sig
) also does this conversion to that shouldn't be the issue
I wrap the os.stat
and other os specific functions in the FileUtil
class because this allows osxphotos to easily do dry-run mode using dependency injection with no other changes to the code (inject a no-op version of the FileUtil
class when running --dry-run
).
Just an update. It seems the latest versions of Synology NAS (I'm currently at 7.1.1-42962 Update 5) and connected via SMB (version 3) now correctly recognizes setting the files' date/time prior to 1970-01-01.
So osxphotos stopped touching such files.
Will keep monitoring to see if we can close this one!
Again, GREEAAATTT JOB!!!!
Describe the bug
--update
,--cleanup
,--exiftool
,--touch-file
options I've noticed that a series of files are always re-exported.osxphotos
to check.db
andcsv
report file to look for changes on a pic (see analysis below).To Reproduce Steps to reproduce the behavior:
colibri.toml
Expected behavior
Analysis
1969-11-09-1
on thecsv
files from the "last" and the "before last" run produce the same output: exported, updated, exif_updateddb
I get the same output for bnoth runs.db
I also get the exact same output for both runs. last and before last.Desktop (please complete the following information):
12.2.1
osxphotos --version
):0.47.4
Additional context Add any other context about the problem here.