Backup files timestamp should retain image creation date

[Solliko]

Thanks for creating this! Worked like a charm on my 200GB 18,000 photo album.

All image files in my backup were given today's timestamp. We need an options to have the backup files created with the original creation date and modified date (from EXIF if the data is unavailable from the smugmug API). This is will help sort, organize and search the backup files.

Also, would be lovely if files in a Gallery are downloaded in the Gallery's sort order so they are created and appear on backup folders in the same order they appear on sumgmug galleries.

Thank!

[tommyblue]

Hi @Solliko, I see the point of the creation/update date, I'll take a look at possible solutions as, afaik, a single solution for all operating systems doesn't exist.

As per the sort order what do you mean? Files are created with their file name, and the sort order of a folder depends on your filesystem or any other app/viewer you use for browsing the files. Could you exand your proposal, please?

Thanks!

[Solliko]

Hi @tommyblue, thanks for considering this. I'm running on macOS. I know that in Swift there are options to specify creation/modified dates at file creation time- not sure about go.

As for the sort order- in smugmug galleries one can specify a Manual sort and arrange photos such that they tell a story. It would be great if photos are downloaded and created on disk in the gallery sort order so when viewing them in Finder they appear in the same order as they do on smugmug.

Thanks!

[tommyblue]

@Solliko v1.2.0 introduces 2 new configurations (see the readme) to override the files last access timestamp, based on the data present in SmugMug

[Solliko]

@tommyblue thanks for adding the metadata times! I added both new options to the config file and ran it against a recent 200 GB backup. Timestamps on existing files were updated, but the time is incorrect - I think it's not adjusted for UTC. Also, the Creation time and Modified time were set to the same value on existing files.

For newly downloaded images (shot today) the Creation time is still the time of the backup download. The Modified time is off by about 10 hours. I am at UTC-2. Attached is a screenshot of a photo taken today, uploaded to Smugmug and downloaded with the backup tool. On the left you can see the Smugmug time stamps (which are correct), on the right the Finder info of the downloaded backup file which is incorrect (Modified time is actually in the future).

[tommyblue]

@Solliko thanks for the feedback. I'll then make further testing. Files timestamps are like in 4 places in the APIs and they're all different 🤦 I'll try to understand which are the correct ones (that correspond to date taken and date modified). One of the problems I see is that not all pictures have enough info and in that case smugmug provides more fields with timestamps (at this point, I don't know the origin...). I need a solution that works for all possible scenarios. As per the date in the future, that's very odd. Need to investigate that too

[Solliko]

@tommyblue Smugmug uses EXIF data when it can. The photo I posted is editing in Lightroom, which keeps the original creation date (aka digitized date) and updates the Modified date whenever edits are made. Most editing software (and cameras) work the same. I think it's just a matter for figuring which date in the API corresponds to Creation and Modified date, as shown in Smugmug photo info screenshot.

As for the future date, I think it's just an error in the UTC offset calculation (aka GMT offset). Once that's fixed it should be fine.

[tommyblue]

I'm investigating the problem. I found a photo named 2019-10-12 06.58.53.jpg (interesting because the name includes the timestamp), but the DateTimeOriginal fields from the AlbumImage API is 2019-10-12T13:58:53+00:00 (7 hrs off). That value is wrong also considering an error in the timezone (I am UTC+2 so also adding/removing 2 hours the time is wrong). The correct time is reported in the metadata api call, that requires an additional API call for each picture. I sent an email to SmugMug support to ask the reason of this time difference. we're chatting, the problem seems related to timezones.

In the meanwhile I measured the impact of the different approaches on my backup:

• without use_metadata_times option: 8m50s
• using DateTimeOriginal (and metadata when DateTimeOriginal is empty): 15m41s
• performing the metadata call for every object: 2h16m

So calling metadata api for each object has a huge impact and should be done only once with force_metadata_times=true to fix all timestamps but then the option should be disabled (I'm adding the instructions in the readme). If Smugmug fixes the issue, then the current behaviour (using DateTimeOriginal) I think is the best option

[Solliko]

@tommyblue I can confirm your observation with DateTimeOriginal. I ran some API calls against my album and the DateTimeOriginal is showing 7 hours ahead of the actual capture time in the image EXIF. The photos were taken at GMT+3. So there appears to be a problem with the smugmug API.

[tommyblue]

Good, I'll post here as soon as I get some feedback from the SmugMug support

[tommyblue]

I've got an answer from SmugMug:

Our uploader process currently isn't time zone aware and takes the DateTimeOriginal field without time zone information. Internally we treat all times without timezone information (which we currently don't support on upload processing) as Pacific Time for sorting purposes and this is surfaced in the Image endpoint while the ImageMetadata endpoint does return the raw values from the exif/iptc metadata with the time as it was included by the camera in the metadata.

While I hope that in the future we might be able to expand on the handling of date/time in photos, right now the best option to get the local time would be to use the ImageMetadata endpoint

[github-actions[bot]]

Stale issue message

[Solliko]

Hello Tommaso,

I would like to reopen this issue with a suggestion.

Having the wrong creation date on my backups is still driving me nuts. I know the problem is on the Smugmug end and we have dealt with this a number of times. Since most of my images are taken at timezone UTC-3 my backup files dates are off by 10 hours.

Perhaps we can add a new Configuration identifier, store.metadata_times_offset to add or subtract hours from the creation date. In my case it would be:

store.metadata_times_offset = -10

For reference, this is my current config.toml file:

[store] destination = "/Volumes/Photos Backup/Album" file_names = "{{.FileName}}" use_metadata_times = true force_metadata_times = true concurrent_downloads = 10 concurrent_albums = 10

Thanks!