RAW (NEF) Files from Raw + Jpeg shooting stored at a different time/date than the Jpeg

smorar commented 4 months ago

The bug

Raw files seem to be stored in the timeline at the time + timezone modifier, whereas the Jpeg file seems to being stored in the timeline at the image creation time. This tends to separate the photos, and makes stacking harder.

The exif data from my example case appear to be the same for creation date, time zone and other critical date/time fields.

exifdata_DSC8135.NEF.txt exifdata_DSC8135.JPG.txt 337872643-bbb5dd03-46d0-4472-8944-be8a9e83b68b 337872636-fe5fce32-dee4-4d24-88be-06e8890e52a8

The OS that Immich Server is running on

Ubuntu 24.04

Version of Immich Server

v1.105.1

Version of Immich Mobile App

v1.105

Platform with the issue

[X] Server
[X] Web
[ ] Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /home/smorar/Pictures:/usr/src/app/picture_library:ro
      - /home/smorar/Videos:/usr/src/app/video_library:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
      #extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
      #  file: hwaccel.transcoding.yml
      #  service: nvenc # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /home/smorar/Pictures:/usr/src/app/picture_library:ro
      - /home/smorar/Videos:/usr/src/app/video_library:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
        file: hwaccel.ml.yml
        service: cuda # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
      - /home/smorar/Pictures:/usr/src/app/picture_library:ro
      - /home/smorar/Videos:/usr/src/app/video_library:ro
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/home/smorar/Immich_Upload
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=<snipped>

#LOG_LEVEL=debug

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Upload a folder containing Raw+Jpeg files
2. View files in timeline, notice that the Jpeg and Raw files are present in separate batches, instead of being interleaved.
3. Check the creation time allocated to each file in Immich

Relevant log output

No response

Additional information

No response

AFumi39 commented 1 month ago

I've the same problem: DateTaken present in metadata is ignored and immich use only modified date

jrasm91 commented 1 month ago

Can you reupload the output from exiftool using this command instead?

exiftool /path/to/file -struct -json

AFumi39 commented 1 month ago

Can you reupload the output from exiftool using this command instead?
exiftool /path/to/file -struct -json

This is my output (I had to set ModifiedDate equal to DateTimeOriginal to "fix" the wrong date problem). For command exiftool -time:all -G1 -a -s "path_to_file"

[System]        FileModifyDate                  : 2019:10:26 12:11:44+02:00
[System]        FileAccessDate                  : 2024:09:04 10:34:03+02:00
[System]        FileCreateDate                  : 2024:09:04 10:33:29+02:00
[IFD0]          ModifyDate                      : 2019:10:26 11:11:45
[XMP-xmp]       CreateDate                      : 2019:10:26 11:11:45.21
[ExifIFD]       DateTimeOriginal                : 2019:10:26 11:11:45
[ExifIFD]       CreateDate                      : 2019:10:26 11:11:45
[ExifIFD]       OffsetTime                      : +02:00
[ExifIFD]       OffsetTimeOriginal              : +02:00
[ExifIFD]       OffsetTimeDigitized             : +02:00
[Nikon]         TimeZone                        : +01:00
[Nikon]         DaylightSavings                 : Yes
[Nikon]         DateDisplayFormat               : Y/M/D
[Nikon]         PowerUpTime                     : 2019:10:25 22:30:25
[ExifIFD]       SubSecTime                      : 21
[ExifIFD]       SubSecTimeOriginal              : 21
[ExifIFD]       SubSecTimeDigitized             : 21
[IFD0]          DateTimeOriginal                : 2019:10:26 11:11:45
[Composite]     SubSecCreateDate                : 2019:10:26 11:11:45.21+02:00
[Composite]     SubSecDateTimeOriginal          : 2019:10:26 11:11:45.21+02:00
[Composite]     SubSecModifyDate                : 2019:10:26 11:11:45.21+02:00

For command exiftool "path_to_file" -struct -json

[{
  "SourceFile": "C:/temp/test/raw/DSC_7958.NEF",
  "ExifToolVersion": 12.67,
  "FileName": "DSC_7958.NEF",
  "Directory": "C:/temp/test/raw",
  "FileSize": "25 MB",
  "FileModifyDate": "2019:10:26 12:11:44+02:00",
  "FileAccessDate": "2024:09:04 10:34:45+02:00",
  "FileCreateDate": "2024:09:04 10:33:29+02:00",
  "FilePermissions": "-rw-rw-rw-",
  "FileType": "NEF",
  "FileTypeExtension": "nef",
  "MIMEType": "image/x-nikon-nef",
  "ExifByteOrder": "Little-endian (Intel, II)",
  "Make": "NIKON CORPORATION",
  "Model": "NIKON D7500",
  "Orientation": "Horizontal (normal)",
  "Software": "Ver.1.00",
  "ModifyDate": "2019:10:26 11:11:45",
  "Artist": "",
  "JpgFromRawStart": 1221632,
  "JpgFromRawLength": 2813439,
  "YCbCrPositioning": "Co-sited",
  "ImageWidth": 5600,
  "ImageHeight": 3728,
  "BitsPerSample": 14,
  "Compression": "Nikon NEF Compressed",
  "PhotometricInterpretation": "Color Filter Array",
  "StripOffsets": 4035072,
  "SamplesPerPixel": 1,
  "RowsPerStrip": 3728,
  "StripByteCounts": 21227517,
  "XResolution": 300,
  "YResolution": 300,
  "PlanarConfiguration": "Chunky",
  "ResolutionUnit": "inches",
  "CFARepeatPatternDim": "2 2",
  "CFAPattern2": "0 1 1 2",
  "SubfileType": "Reduced-resolution image",
  "OtherImageStart": 277504,
  "OtherImageLength": 943757,
  "ReferenceBlackWhite": "0 255 0 255 0 255",
  "CreatorTool": "NIKON D7500 Ver.1.00    ",
  "Rating": 0,
  "Copyright": "",
  "ExposureTime": "1/500",
  "FNumber": 11.0,
  "ExposureProgram": "Not Defined",
  "ISO": 800,
  "SensitivityType": "Recommended Exposure Index",
  "RecommendedExposureIndex": 800,
  "CreateDate": "2019:10:26 11:11:45",
  "OffsetTime": "+02:00",
  "OffsetTimeOriginal": "+02:00",
  "OffsetTimeDigitized": "+02:00",
  "ExposureCompensation": 0,
  "MaxApertureValue": 5.3,
  "MeteringMode": "Multi-segment",
  "LightSource": "Unknown",
  "Flash": "Auto, Did not fire",
  "FocalLength": "80.0 mm",
  "MakerNoteVersion": 2.11,
  "Quality": "RAW",
  "WhiteBalance": "Auto1",
  "FocusMode": "AF-A",
  "FlashSetting": "Normal",
  "FlashType": "",
  "WB_RBLevels": "2.2265625 1.583007813 1 1",
  "ProgramShift": 0,
  "ExposureDifference": 0,
  "PreviewImageStart": 54452,
  "PreviewImageLength": 164480,
  "FlashExposureComp": 0,
  "ExternalFlashExposureComp": 0,
  "FlashExposureBracketValue": 0.0,
  "ExposureBracketValue": 0,
  "CropHiSpeed": "DX Uncropped (5600x3728 cropped to 5600x3728 at pixel 0,0)",
  "ExposureTuning": 0,
  "SerialNumber": 6010653,
  "ColorSpace": "sRGB",
  "VRInfoVersion": "0100",
  "VibrationReduction": "On",
  "VRMode": "Normal",
  "ActiveD-Lighting": "Auto",
  "PictureControlVersion": "0200",
  "PictureControlName": "Auto",
  "PictureControlBase": "Auto",
  "PictureControlAdjust": "Default Settings",
  "PictureControlQuickAdjust": "Normal",
  "Clarity": "n/a",
  "Brightness": "n/a",
  "Hue": "n/a",
  "FilterEffect": "n/a",
  "ToningEffect": "n/a",
  "ToningSaturation": "n/a",
  "TimeZone": "+01:00",
  "DaylightSavings": "Yes",
  "DateDisplayFormat": "Y/M/D",
  "ISOExpansion": "Off",
  "ISO2": 800,
  "ISOExpansion2": "Off",
  "VignetteControl": "Normal",
  "AutoDistortionControl": "Off",
  "BlackLevel": "400 400 400 400",
  "ImageSizeRAW": "Large",
  "WhiteBalanceFineTune": "0 0",
  "CropArea": "16 8 5568 3712",
  "ISOAutoShutterTime": "Auto",
  "FlickerReductionShooting": "Disable",
  "FlickerReductionIndicator": "Enable",
  "MovieISOAutoControlManualMode": "Off",
  "MovieWhiteBalanceSameAsPhoto": "No",
  "AF-SPrioritySel": "Focus",
  "AFActivation": "Shutter/AF-On",
  "FocusPointWrap": "No Wrap",
  "ManualFocusPointIllumination": "On",
  "AF-AssistIlluminator": "On",
  "ManualFocusRingInAFMode": "On",
  "ISOStepSize": "1/3 EV",
  "ExposureControlStepSize": "1/3 EV",
  "EasyExposureCompensation": "Off",
  "MatrixMetering": "Face Detection On",
  "FineTuneOptMatrixMetering": 0,
  "FineTuneOptCenterWeighted": 0,
  "FineTuneOptSpotMetering": 0,
  "FineTuneOptHighlightWeighted": 0,
  "ShutterReleaseButtonAE-L": "Off",
  "SelfTimerTime": "10 s",
  "SelfTimerShotCount": 9,
  "SelfTimerShotInterval": "0.5 s",
  "PlaybackMonitorOffTime": "10 s",
  "MenuMonitorOffTime": "1 min",
  "ShootingInfoMonitorOffTime": "4 s",
  "ImageReviewMonitorOffTime": "4 s",
  "LiveViewMonitorOffTime": "10 min",
  "CLModeShootingSpeed": "1 fps",
  "MaxContinuousRelease": 100,
  "ExposureDelayMode": "0.5 s",
  "ElectronicFront-CurtainShutter": "Off",
  "FileNumberSequence": "On",
  "FramingGridDisplay": "Off",
  "LCDIllumination": "Off",
  "OpticalVR": "On",
  "FlashShutterSpeed": "1/60 s",
  "FlashExposureCompArea": "Entire Frame",
  "AutoFlashISOSensitivity": "Subject and Background",
  "AssignBktButton": "Auto Bracketing",
  "MultiSelectorLiveView": "Select Center Focus Point",
  "CmdDialsChangeMainSub": "Autofocus Off, Exposure Off",
  "CmdDialsMenuAndPlayback": "Off",
  "SubDialFrameAdvance": "10 Frames",
  "ReleaseButtonToUseDial": "No",
  "ReverseIndicators": "- 0 +",
  "MovieShutterButton": "Take Photo",
  "Language": "English",
  "FlickAdvanceDirection": "Left to Right",
  "HDMIOutputResolution": "Auto",
  "HDMIOutputRange": "Auto",
  "CmdDialsReverseRotation": "Exposure Compensation",
  "LensType": "G VR",
  "Lens": "18-140mm f/3.5-5.6",
  "FlashMode": "Did Not Fire",
  "ShootingMode": "Single-Frame",
  "ContrastCurve": "(Binary data 578 bytes, use -b option to extract)",
  "ShotInfoVersion": "0242",
  "FirmwareVersion": "1.00h",
  "RollAngle": 1.1,
  "PitchAngle": -2.3,
  "YawAngle": -70.5,
  "NEFCompression": "Lossy (type 2)",
  "NoiseReduction": "Off",
  "NEFLinearizationTable": "(Binary data 624 bytes, use -b option to extract)",
  "ColorBalanceVersion": "0226",
  "LensDataVersion": "0204",
  "ExitPupilPosition": "107.8 mm",
  "AFAperture": 5.3,
  "FocusPosition": "0x06",
  "FocusDistance": "14.96 m",
  "LensIDNumber": 160,
  "LensFStops": 5.33,
  "MinFocalLength": "18.3 mm",
  "MaxFocalLength": "142.5 mm",
  "MaxApertureAtMinFocal": 3.6,
  "MaxApertureAtMaxFocal": 5.7,
  "MCUVersion": 187,
  "EffectiveMaxAperture": 5.3,
  "RawImageCenter": "2800 1864",
  "RetouchHistory": "None",
  "ShutterCount": 8457,
  "FlashInfoVersion": "0108",
  "FlashSource": "None",
  "ExternalFlashFirmware": "n/a",
  "ExternalFlashZoomOverride": "No",
  "ExternalFlashStatus": "Flash Not Attached",
  "ExternalFlashReadyState": "n/a",
  "FlashCompensation": 0,
  "FlashGNDistance": 0,
  "FlashGroupAControlMode": "Off",
  "FlashGroupBControlMode": "Off",
  "FlashGroupCControlMode": "Off",
  "FlashGroupACompensation": 0,
  "FlashGroupBCompensation": 0,
  "FlashGroupCCompensation": 0,
  "VariProgram": "Auto",
  "MultiExposureVersion": "0101",
  "MultiExposureMode": "Off",
  "MultiExposureShots": 0,
  "MultiExposureAutoGain": "Off",
  "HighISONoiseReduction": "Normal",
  "PowerUpTime": "2019:10:25 22:30:25",
  "AFInfo2Version": "0101",
  "ContrastDetectAF": "Off",
  "AFAreaMode": "Auto-area",
  "PhaseDetectAF": "On (51-point)",
  "PrimaryAFPoint": "C7",
  "AFPointsUsed": "B6,B7,C6,C7,D3,D4,D5,D6,D7,D8,D9,D10,E1,E2,E3,E4,E5,E6,E7,E8,E9",
  "FileInfoVersion": "0100",
  "MemoryCardNumber": 0,
  "DirectoryNumber": 103,
  "FileNumber": 7958,
  "AFFineTune": "Off",
  "AFFineTuneIndex": "n/a",
  "AFFineTuneAdj": 0,
  "AFFineTuneAdjTele": 0,
  "RetouchInfoVersion": "0200",
  "RetouchNEFProcessing": "Off",
  "SilentPhotography": "Off",
  "UserComment": "",
  "SubSecTime": 21,
  "SubSecTimeOriginal": 21,
  "SubSecTimeDigitized": 21,
  "SensingMethod": "One-chip color area",
  "FileSource": "Digital Camera",
  "SceneType": "Directly photographed",
  "CustomRendered": "Normal",
  "ExposureMode": "Auto",
  "FocalLengthIn35mmFormat": "120 mm",
  "SceneCaptureType": "Standard",
  "GainControl": "Low gain up",
  "Contrast": "Normal",
  "Saturation": "Normal",
  "Sharpness": "Normal",
  "SubjectDistanceRange": "Unknown",
  "GPSVersionID": "2.3.0.0",
  "DateTimeOriginal": "2019:10:26 11:11:45",
  "TIFF-EPStandardID": "1 0 0 0",
  "Aperture": 11.0,
  "BlueBalance": 1.583008,
  "CFAPattern": "[Red,Green][Green,Blue]",
  "ImageSize": "5600x3728",
  "JpgFromRaw": "(Binary data 2813439 bytes, use -b option to extract)",
  "Megapixels": 20.9,
  "OtherImage": "(Binary data 943757 bytes, use -b option to extract)",
  "PreviewImage": "(Binary data 164480 bytes, use -b option to extract)",
  "RedBalance": 2.226563,
  "ScaleFactor35efl": 1.5,
  "ShutterSpeed": "1/500",
  "SubSecCreateDate": "2019:10:26 11:11:45.21+02:00",
  "SubSecDateTimeOriginal": "2019:10:26 11:11:45.21+02:00",
  "SubSecModifyDate": "2019:10:26 11:11:45.21+02:00",
  "ThumbnailTIFF": "(Binary data 57816 bytes, use -b option to extract)",
  "AutoFocus": "On",
  "LensID": "AF-S DX Nikkor 18-140mm f/3.5-5.6G ED VR",
  "LensSpec": "18-140mm f/3.5-5.6 G VR",
  "CircleOfConfusion": "0.020 mm",
  "DOF": "20.79 m (9.89 - 30.68 m)",
  "FOV": "17.0 deg (4.46 m)",
  "FocalLength35efl": "80.0 mm (35 mm equivalent: 120.0 mm)",
  "HyperfocalDistance": "29.05 m",
  "LightValue": 12.9
}]

smorar commented 1 month ago

Can you reupload the output from exiftool using this command instead?
exiftool /path/to/file -struct -json

Here is my re-uploaded output as requested in json format. exifdata_DSC8135.NEF.txt exifdata_DSC8135.JPG.txt

jrasm91 commented 1 month ago

@smorar - I am not able to reproduce the issue. Can you upload a pair of files for me to test with as a zip?

smorar commented 1 month ago

@jrasm91 - The files are too big. Any preference for an alternative upload location?

smorar commented 1 month ago

@jrasm91 - Can you download the file from here: https://smorar.kicks-ass.net/wp-content/uploads/JPEG_NEF_example.zip

jrasm91 commented 1 month ago

I was able to access them, yes. They both show up as the correct date for me.

"dateTimeOriginal": "2021-12-12T01:49:20.000Z",
"timeZone": "UTC+10",

Can you multi-select both in the main timeline and run "Refresh metadata" and verify that they are still broken? I cannot reproduce it, so it is likely that the issue has since been fixed and that running metadata extraction again for these files would fix them.

jrasm91 commented 1 month ago

I'm going to close this as resolved, since I cannot reproduce it. If it is still an issue, or you have a different file you'd like me to look at, just upload it again and let me know.

immich-app / immich