Altered or incomplete metadata when exporting to JXL

RafaelLinux commented 2 months ago

Describe the bug

Lately I have decided to export to JXL instead of TIFF and I have noticed that the exported metadata (from RAW files) is altered or simply ignored when exporting to JXL. However, in TIFF format they are respected. This is very strange to me, because JXL is the format that supports the largest amount of metadata and the most flexible even in the way of storing them inside the JXL container (for example, in JSON format).

Steps to reproduce

Select a RAW image
Export it to TIFF, WEBP, JXL

Expected behavior

Respect metadata in all exported images

Logfile | Screenshot | Screencast

Commit

No response

Where did you obtain darktable from?

distro packaging

darktable version

4.8.1

What OS are you using?

Linux

What is the version of your OS?

openSUSE Tumbleweed

Describe your system?

Operating System: openSUSE Tumbleweed 20240902 KDE Plasma Version: 6.1.4 KDE Frameworks Version: 6.5.0 Qt Version: 6.7.2 Kernel Version: 6.10.7-1-default (64-bit) Graphics Platform: Wayland Processors: 24 × AMD Ryzen 9 3900X 12-Core Processor Memory: 31.3 GiB of RAM Graphics Processor: AMD Radeon RX 7600

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

AMD Ryzen 9 3900X, AMD official driver

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

Is not related to XMP file, cause same issue in ANY unprocessed RAW file

kmilos commented 2 months ago

There are known limitations of the JPEG XL format (i.e. no write support by the external Exiv2 library at this moment), as already described in the release notes. You can currently export either all (unmodified/unfiltered) metadata or none, there is no fine-grained control like there is for e.g. TIFF.

P.S. Instead of digikam, it's more robust to check the metadata output using e.g. exiftool -a -u -s -g1 and exiftool -v3. (And/or provide a link to raw/tif/jxl samples for eventual reproduction.)

RafaelLinux commented 2 months ago

I understand. I assume then that the metadata will be incorrect until the external library works properly in this respect. If so, can the thread be left open as ‘pending external resolution’ or similar? I say this because more people will come up with the same problem and if they see this open thread, they will know why it is.

Regarding the extraction of the information, I find it much more comfortable to show it with Digikam than from the command line, which would force me to access the subfolder (up to 7 levels) where these images were. Besides, on the screenshot it is easier to show the differences in the image line by line, that's the reason to do it this way ;)

kmilos commented 2 months ago

There is one strange thing in your report that needs following up: some data is partially there. To eliminate the possibility that digiKam is the source of the error, another method is needed to double check. So, you either need to provide that command line output via exiftool and exiv2, or some samples.

RafaelLinux commented 2 months ago

Yes, it's very strange. This is the output of the commands you have kindly asked me to execute:

>exiftool -a -u -s -g1 _1030523.jxl
---- ExifTool ----
ExifToolVersion                 : 12.96
---- System ----
FileName                        : _1030523.jxl
Directory                       : .
FileSize                        : 91 MB
FileModifyDate                  : 2024:04:17 07:58:57+02:00
FileAccessDate                  : 2024:09:05 17:21:45+02:00
FileInodeChangeDate             : 2024:04:17 07:58:58+02:00
FilePermissions                 : -rw-r--r--
---- File ----
FileType                        : JXL
FileTypeExtension               : jxl
MIMEType                        : image/jxl
ImageWidth                      : 5200
ImageHeight                     : 3904
---- Jpeg2000 ----
MajorBrand                      : JPEG XL Image (.JXL)
MinorVersion                    : 0.0.0
CompatibleBrands                : jxl
---- Composite ----
ImageSize                       : 5200x3904
Megapixels                      : 20.3

and the second one

> exiftool -v3 _1030523.jxl
  ExifToolVersion = 12.96
  FileName = _1030523.jxl
  Directory = .
  FileSize = 90800118
  FileModifyDate = 1713333537
  FileAccessDate = 1725549705
  FileInodeChangeDate = 1713333538
  FilePermissions = 33188
  FileType = JXL
  FileTypeExtension = JXL
  MIMEType = image/jxl
  + [JP2 directory]
  | FileType (SubDirectory) -->
  | - Tag 'ftyp' (12 bytes):
  |     0014: 6a 78 6c 20 00 00 00 00 6a 78 6c 20             [jxl ....jxl ]
  | + [BinaryData directory, 12 bytes]
  | | MajorBrand = jxl 
  | | - Tag 0x0000 (4 bytes, undef[4]):
  | |     0014: 6a 78 6c 20                                     [jxl ]
  | | MinorVersion = 
  | | - Tag 0x0001 (4 bytes, undef[4]):
  | |     0018: 00 00 00 00                                     [....]
  | | CompatibleBrands = jxl 
  | | - Tag 0x0002 (4 bytes, undef[4]):
  | |     001c: 6a 78 6c 20                                     [jxl ]
  | Jpeg2000_jxll = .
  | - Tag 'jxll' (1 bytes):
  |     0028: 0a                                              [.]
  | JXLCodestream = ...y...G......rb....zO.....;..-H4...M..%<.f..GK..g}M..u...DFc#2....[snip]
  | - Tag 'jxlc' (90800069 bytes):
  |     0031: ff 0a fa 79 e8 89 e2 47 f0 03 08 cf 13 03 72 62 [...y...G......rb]
  |     0041: 1e 80 d0 1c 7a 4f 12 0a 8e f1 f9 3b 8f 13 2d 48 [....zO.....;..-H]
  |     0051: 34 bc a6 ee 4d a2 a9 25 3c a1 66 97 06 47 4b e2 [4...M..%<.f..GK.]
  |     0061: d8 67 7d 4d a1 c1 75 97 d0 08 44 46 63 23 32 84 [.g}M..u...DFc#2.]
  |     0071: e7 b2 f9 47 74 6c 00 c8 1c 36 e7 f4 03 dc e2 67 [...Gtl...6.....g]
  |     [snip 90799989 bytes]
  | ImageWidth = 5200
  | ImageHeight = 3904

And, if it helps, the RW2 output of first command:

> exiftool -a -u -s -g1 _1030523.RW2
---- ExifTool ----
ExifToolVersion                 : 12.96
---- System ----
FileName                        : _1030523.RW2
Directory                       : .
FileSize                        : 24 MB
FileModifyDate                  : 2018:07:30 05:16:14+02:00
FileAccessDate                  : 2024:09:05 17:19:16+02:00
FileInodeChangeDate             : 2023:09:14 16:28:13+02:00
FilePermissions                 : -rw-r--r--
---- File ----
FileType                        : RW2
FileTypeExtension               : rw2
MIMEType                        : image/x-panasonic-rw2
ExifByteOrder                   : Little-endian (Intel, II)
ExifByteOrder                   : Little-endian (Intel, II)
ImageWidth                      : 1920
ImageHeight                     : 1440
EncodingProcess                 : Baseline DCT, Huffman coding
BitsPerSample                   : 8
ColorComponents                 : 3
YCbCrSubSampling                : YCbCr4:2:2 (2 1)
---- IFD0 ----
PanasonicRawVersion             : 0360
SensorWidth                     : 5264
SensorHeight                    : 3904
SensorTopBorder                 : 8
SensorLeftBorder                : 8
SensorBottomBorder              : 3896
SensorRightBorder               : 5192
SamplesPerPixel                 : 1
CFAPattern                      : [Green,Blue][Red,Green]
BitsPerSample                   : 12
Compression                     : Panasonic RAW 1
PanasonicRaw_0x000d             : 1
LinearityLimitRed               : 4095
LinearityLimitGreen             : 4095
LinearityLimitBlue              : 4095
ISO                             : 3200
HighISOMultiplierRed            : 1
HighISOMultiplierGreen          : 1
HighISOMultiplierBlue           : 1
NoiseReductionParams            : 5 100 4 4 4 200 8 8 8 400 16 16 16 800 32 32 32 1600 64 64 64
BlackLevelRed                   : 131
BlackLevelGreen                 : 129
BlackLevelBlue                  : 132
WBRedLevel                      : 577
WBGreenLevel                    : 256
WBBlueLevel                     : 445
PanasonicRaw_0x0029             : 
PanasonicRaw_0x002a             : 
PanasonicRaw_0x002b             : 
PanasonicRaw_0x002c             : [...]
RawFormat                       : 4
Make                            : Panasonic
Model                           : DMC-GX8
Orientation                     : Horizontal (normal)
XResolution                     : 180
YResolution                     : 180
ResolutionUnit                  : inches
Software                        : Ver.2.2
ModifyDate                      : 2018:07:30 05:16:14
WhitePoint                      : 0.313 0.329
PrimaryChromaticities           : 0.64 0.33 0.21 0.71 0.15 0.06
YCbCrCoefficients               : 0.299 0.587 0.114
YCbCrPositioning                : Co-sited
JpgFromRaw                      : (Binary data 353792 bytes, use -b option to extract)
CropTop                         : 8
CropLeft                        : 8
CropBottom                      : 3896
CropRight                       : 5192
Make                            : Panasonic
Model                           : DMC-GX8
StripOffsets                    : 4294967295
Orientation                     : Horizontal (normal)
RowsPerStrip                    : 3904
StripByteCounts                 : 0
RawDataOffset                   : 358400
PanasonicRaw_0x011a             : 2
PanasonicRaw_0x011b             : �����.v�.�n�./�.s��..��n.�.�.D.�b�.���9���9��.�[...]
Gamma                           : 2.7109375
PanasonicRaw_0x011d             : [...]
PanasonicRaw_0x011e             : .,0hr��
PanasonicRaw_0x011f             : ....>:400004>............<0[...]
---- PanasonicRaw ----
NumWBEntries                    : 7
WBType1                         : Fine Weather
WB_RGBLevels1                   : 587 256 481
WBType2                         : Cloudy
WB_RGBLevels2                   : 622 256 444
WBType3                         : Shade
WB_RGBLevels3                   : 664 256 417
WBType4                         : Tungsten (Incandescent)
WB_RGBLevels4                   : 427 256 705
WBType5                         : Flash
WB_RGBLevels5                   : 593 256 459
WBType6                         : D55
WB_RGBLevels6                   : 557 256 483
WBType7                         : ISO Studio Tungsten
WB_RGBLevels7                   : 427 256 705
DistortionParam02               : 0
DistortionParam04               : 0
DistortionScale                 : 1
DistortionCorrection            : Off
DistortionParam08               : 0
DistortionParam09               : 0
DistortionParam11               : 0
DistortionN                     : 3240
---- ExifIFD ----
ExposureTime                    : 1/320
FNumber                         : 4.0
ExposureProgram                 : Program AE
ISO                             : 3200
SensitivityType                 : Standard Output Sensitivity
ExifVersion                     : 0230
DateTimeOriginal                : 2018:07:30 05:16:14
CreateDate                      : 2018:07:30 05:16:14
ComponentsConfiguration         : Y, Cb, Cr, -
CompressedBitsPerPixel          : 2
ExposureCompensation            : 0
MaxApertureValue                : 2.8
MeteringMode                    : Spot
LightSource                     : Unknown
Flash                           : Off, Did not fire
FocalLength                     : 33.0 mm
SubSecTime                      : 097
SubSecTimeOriginal              : 097
SubSecTimeDigitized             : 097
FlashpixVersion                 : 0100
ColorSpace                      : Uncalibrated
ExifImageWidth                  : 1920
ExifImageHeight                 : 1440
SensingMethod                   : One-chip color area
FileSource                      : Digital Camera
SceneType                       : Directly photographed
CustomRendered                  : Normal
ExposureMode                    : Auto
WhiteBalance                    : Auto
DigitalZoomRatio                : 0
FocalLengthIn35mmFormat         : 66 mm
SceneCaptureType                : Standard
GainControl                     : High gain up
Contrast                        : Normal
Saturation                      : Normal
Sharpness                       : Normal
Gamma                           : 2.2
ExposureTime                    : 1/320
FNumber                         : 4.0
ExposureProgram                 : Program AE
ExifVersion                     : 0230
DateTimeOriginal                : 2018:07:30 05:16:14
CreateDate                      : 2018:07:30 05:16:14
ExposureCompensation            : 0
MaxApertureValue                : 2.8
MeteringMode                    : Spot
Flash                           : Off, Did not fire
FocalLength                     : 33.0 mm
SubSecTimeOriginal              : 097
SubSecTimeDigitized             : 097
FileSource                      : Digital Camera
---- Panasonic ----
ImageQuality                    : RAW
FirmwareVersion                 : 0.2.2.0
WhiteBalance                    : Auto
FocusMode                       : Auto
AFAreaMode                      : Tracking
ImageStabilization              : Dual IS
MacroMode                       : Off
ShootingMode                    : Program
Audio                           : No
DataDump                        : (Binary data 24584 bytes, use -b option to extract)
Panasonic_0x0022                : 0
FlashBias                       : 0
InternalSerialNumber            : (XGR) 2016:11:07 no. 0083
PanasonicExifVersion            : 0411
VideoFrameRate                  : n/a
ColorEffect                     : Off
TimeSincePowerOn                : 00:00:52.50
BurstMode                       : Off
SequenceNumber                  : 0
ContrastMode                    : High
NoiseReduction                  : Standard
SelfTimer                       : Off
Panasonic_0x002f                : 1
Rotation                        : Horizontal (normal)
AFAssistLamp                    : Enabled but Not Used
ColorMode                       : Normal
BabyAge                         : (not set)
OpticalZoomMode                 : Standard
ConversionLens                  : Off
TravelDay                       : n/a
Panasonic_0x0037                : 257
BatteryLevel                    : Full
Contrast                        : Normal
WorldTimeLocation               : Home
TextStamp                       : Off
ProgramISO                      : 3200
AdvancedSceneType               : 1
TextStamp                       : Off
FacesDetected                   : 0
Saturation                      : Normal
Sharpness                       : Normal
JPEGQuality                     : n/a (RAW only)
ColorTempKelvin                 : 6300
BracketSettings                 : No Bracket
WBShiftAB                       : 0
WBShiftGM                       : 0
FlashCurtain                    : n/a
LongExposureNoiseReduction      : On
PanasonicImageWidth             : 5184
PanasonicImageHeight            : 3888
AFPointPosition                 : 0.5 0.5
NumFacePositions                : 0
Panasonic_0x004f                : 0
LensType                        : LUMIX G VARIO 12-35/F2.8
LensSerialNumber                : 16111200561
AccessoryType                   : NO-ACCESSORY
AccessorySerialNumber           : 0000000
Panasonic_0x0055                : 1
Panasonic_0x0058                : ..
Panasonic_0x005e                : 
Panasonic_0x005f                : 
LensFirmwareVersion             : 0.1.3.0
FacesRecognized                 : 0
RecognizedFaceFlags             : 0 0 0 0
Title                           : 
BabyName                        : 
Location                        : 
Panasonic_0x0068                : 0
Country                         : 
Panasonic_0x006a                : 0
State                           : 
Panasonic_0x006c                : 0
City                            : 
Panasonic_0x006e                : 0
Landmark                        : 
IntelligentResolution           : Off
Panasonic_0x0071                : [...]
Panasonic_0x0072                : 0
Panasonic_0x0073                : 0
Panasonic_0x0074                : 0
Panasonic_0x0075                : 0
MergedImages                    : 0
BurstSpeed                      : 0
Panasonic_0x0078                : 0
IntelligentD-Range              : Off
Panasonic_0x007a                : 0
ClearRetouch                    : Off
Panasonic_0x007f                : 0
City2                           : 
Panasonic_0x0081                : [...]
Panasonic_0x0082                : [...]
Panasonic_0x0083                : [...]
Panasonic_0x0084                : [...]
Panasonic_0x0085                : [...]
Panasonic_0x0087                : 65535
PhotoStyle                      : Vivid
ShadingCompensation             : On
WBShiftIntelligentAuto          : 0
AccelerometerZ                  : 257
AccelerometerX                  : -14
AccelerometerY                  : 1
CameraOrientation               : Normal
RollAngle                       : -0.7
PitchAngle                      : -0.1
WBShiftCreativeControl          : 0
SweepPanoramaDirection          : Off
SweepPanoramaFieldOfView        : 0
TimerRecording                  : Off
Panasonic_0x0097                : 0
Panasonic_0x0099                : 0
Panasonic_0x009a                : 1
Panasonic_0x009b                : 1
InternalNDFilter                : 0
HDR                             : Off
ShutterType                     : Mechanical
Panasonic_0x00a0                : �.....�.....................
FilterEffect                    : Off
Panasonic_0x00a2                : 0 0
ClearRetouchValue               : undef
Panasonic_0x00a6                : 0
OutputLUT                       : (Binary data 864 bytes, use -b option to extract)
Panasonic_0x00a8                : 0
Panasonic_0x00a9                : 0
TouchAE                         : Off
MonochromeFilterEffect          : Off
HighlightShadow                 : 0 0
TimeStamp                       : 2018:07:30 04:16:14
Panasonic_0x00b0                : 50000
Panasonic_0x00b2                : 0
VideoBurstResolution            : Off or 4K
MultiExposure                   : Off
Panasonic_0x00b5                : 0
Panasonic_0x00b6                : 
Panasonic_0x00b7                : 0
Panasonic_0x00b8                : 
RedEyeRemoval                   : Off
Panasonic_0x00ba                : 0
VideoBurstMode                  : Off
DiffractionCorrection           : Auto
TimeLapseShotNumber             : 0
MakerNoteVersion                : 0151
SceneMode                       : Off
HighlightWarning                : No
DarkFocusEnvironment            : No
WBRedLevel                      : 2310
WBGreenLevel                    : 1024
WBBlueLevel                     : 1780
Panasonic_0x8007                : 1
TextStamp                       : Off
TextStamp                       : Off
BabyAge                         : (not set)
---- InteropIFD ----
InteropIndex                    : R03 - DCF option file (Adobe RGB)
InteropVersion                  : 0100
---- PrintIM ----
PrintIMVersion                  : 0250
PrintIM_0x0001                  : 0x00160016
PrintIM_0x0002                  : 0x00000000
PrintIM_0x0003                  : 0x00000064
PrintIM_0x0007                  : 0x00000000
PrintIM_0x0008                  : 0x00000000
PrintIM_0x0009                  : 0x00000000
PrintIM_0x000a                  : 0x00000000
PrintIM_0x000b                  : 0x000000ac
PrintIM_0x000c                  : 0x00000000
PrintIM_0x000d                  : 0x00000000
PrintIM_0x000e                  : 0x000000c4
PrintIM_0x0100                  : 0x00000005
PrintIM_0x0101                  : 0x00000001
PrintIM_0x0110                  : 0x00000080
---- IFD1 ----
Compression                     : JPEG (old-style)
Orientation                     : Horizontal (normal)
XResolution                     : 180
YResolution                     : 180
ResolutionUnit                  : inches
ThumbnailOffset                 : 35840
ThumbnailLength                 : 2971
YCbCrPositioning                : Co-sited
ThumbnailImage                  : (Binary data 2971 bytes, use -b option to extract)
---- CameraIFD ----
PanasonicRaw_CameraIFD_0x1000   : 0
FocusStepNear                   : 30
FocusStepCount                  : 8022
FlashFired                      : No
PanasonicRaw_CameraIFD_0x1104   : 0
LensAttached                    : Yes
LensTypeMake                    : 2
LensTypeModel                   : 15 10
FocalLengthIn35mmFormat         : 66 mm
PanasonicRaw_CameraIFD_0x1300   : 2253
ApertureValue                   : 3.9
ShutterSpeedValue               : 1/337
SensitivityValue                : 5
PanasonicRaw_CameraIFD_0x1304   : 0
PanasonicRaw_CameraIFD_0x1306   : 1894
PanasonicRaw_CameraIFD_0x1400   : 0
PanasonicRaw_CameraIFD_0x1401   : 0
PanasonicRaw_CameraIFD_0x1402   : 0
PanasonicRaw_CameraIFD_0x140b   : 518
PanasonicRaw_CameraIFD_0x140c   : 1
PanasonicRaw_CameraIFD_0x140d   : 2
PanasonicRaw_CameraIFD_0x140e   : 0
PanasonicRaw_CameraIFD_0x140f   : 0
PanasonicRaw_CameraIFD_0x1410   : 303
PanasonicRaw_CameraIFD_0x1411   : 11 1 0 9 11 1 0 9
FacesDetected                   : No
PanasonicRaw_CameraIFD_0x1413   : 1
PanasonicRaw_CameraIFD_0x1500   : 1
PanasonicRaw_CameraIFD_0x150d   : 256
PanasonicRaw_CameraIFD_0x1512   : 6 6 5 5
PanasonicRaw_CameraIFD_0x1513   : 201 168 -42 -342
PanasonicRaw_CameraIFD_0x1514   : 23 337 718 1181
PanasonicRaw_CameraIFD_0x1515   : 2 2
PanasonicRaw_CameraIFD_0x1516   : 30 30
PanasonicRaw_CameraIFD_0x2000   : 4
PanasonicRaw_CameraIFD_0x2009   : 529 519 518 527 529 519 518 527
PanasonicRaw_CameraIFD_0x3000   : 1710
PanasonicRaw_CameraIFD_0x3001   : 2802
PanasonicRaw_CameraIFD_0x3002   : 2228
PanasonicRaw_CameraIFD_0x3003   : 1937
PanasonicRaw_CameraIFD_0x3004   : 1711
PanasonicRaw_CameraIFD_0x3005   : 2823
PanasonicRaw_CameraIFD_0x3006   : 2228
PanasonicRaw_CameraIFD_0x3007   : 1935
PanasonicRaw_CameraIFD_0x3100   : 1711
PanasonicRaw_CameraIFD_0x3101   : 2823
PanasonicRaw_CameraIFD_0x3102   : 2228
PanasonicRaw_CameraIFD_0x3103   : 1935
PanasonicRaw_CameraIFD_0x3104   : 1686
PanasonicRaw_CameraIFD_0x3105   : 2807
PanasonicRaw_CameraIFD_0x3106   : 2202
PanasonicRaw_CameraIFD_0x3107   : 1949
PanasonicRaw_CameraIFD_0x3108   : 1711
PanasonicRaw_CameraIFD_0x3109   : 2841
PanasonicRaw_CameraIFD_0x310a   : 2238
PanasonicRaw_CameraIFD_0x310b   : 1950
WB_CFA0_LevelDaylight           : 2228
WB_CFA1_LevelDaylight           : 1024
WB_CFA2_LevelDaylight           : 1024
WB_CFA3_LevelDaylight           : 1935
PanasonicRaw_CameraIFD_0x3204   : 1024
PanasonicRaw_CameraIFD_0x3205   : 1024
PanasonicRaw_CameraIFD_0x3206   : 1024
PanasonicRaw_CameraIFD_0x3207   : 1024
PanasonicRaw_CameraIFD_0x3208   : 0
PanasonicRaw_CameraIFD_0x3209   : 512
PanasonicRaw_CameraIFD_0x320a   : 16383
PanasonicRaw_CameraIFD_0x320b   : 0
PanasonicRaw_CameraIFD_0x320c   : 0
PanasonicRaw_CameraIFD_0x320d   : 0
WhiteBalanceSet                 : Auto
PanasonicRaw_CameraIFD_0x3400   : 2176
PanasonicRaw_CameraIFD_0x3401   : 2182
PanasonicRaw_CameraIFD_0x3410   : 0
PanasonicRaw_CameraIFD_0x3411   : 0
WB_RedLevelAuto                 : 2310
WB_BlueLevelAuto                : 1780
PanasonicRaw_CameraIFD_0x3500   : 256
Orientation                     : Horizontal (normal)
PanasonicRaw_CameraIFD_0x3502   : 0
PanasonicRaw_CameraIFD_0x3503   : 0
WhiteBalanceDetected            : Auto
PanasonicRaw_CameraIFD_0x3601   : 0
PanasonicRaw_CameraIFD_0x3700   : 0
PanasonicRaw_CameraIFD_0x3701   : 0
PanasonicRaw_CameraIFD_0x3702   : 0
PanasonicRaw_CameraIFD_0x3703   : 0
PanasonicRaw_CameraIFD_0x3704   : 128
PanasonicRaw_CameraIFD_0x3705   : 0
PanasonicRaw_CameraIFD_0x3706   : 0
PanasonicRaw_CameraIFD_0x3707   : 0
PanasonicRaw_CameraIFD_0x3708   : 0
PanasonicRaw_CameraIFD_0x3709   : 0
PanasonicRaw_CameraIFD_0x4000   : 0
PanasonicRaw_CameraIFD_0x4001   : 3
PanasonicRaw_CameraIFD_0x4002   : 2
PanasonicRaw_CameraIFD_0x4003   : 3
---- Composite ----
Aperture                        : 4.0
BlueBalance                     : 1.738281
RedBalance                      : 2.253906
ShutterSpeed                    : 1/320
SubSecCreateDate                : 2018:07:30 05:16:14.097
SubSecDateTimeOriginal          : 2018:07:30 05:16:14.097
SubSecModifyDate                : 2018:07:30 05:16:14.097
LensType                        : Lumix G X Vario 12-35mm F2.8 Asph. Power OIS
AdvancedSceneMode               : Off
ImageHeight                     : 3888
ImageWidth                      : 5184
ImageSize                       : 5184x3888
LensID                          : Lumix G X Vario 12-35mm F2.8 Asph. Power OIS
LightValue                      : 7.3
Megapixels                      : 20.2
ScaleFactor35efl                : 2.0
CircleOfConfusion               : 0.015 mm
FOV                             : 30.5 deg
FocalLength35efl                : 33.0 mm (35 mm equivalent: 66.0 mm)
HyperfocalDistance              : 18.12 m

kmilos commented 2 months ago

Yes, it's very strange.

Thanks for those, getting somewhere finally. Exiftool clearly shows there so no metadata whatsoever in the JXL file, because its export has not been enabled as per the release notes. This is expected and not strange. Please enable it and test again.

I trust exiftool. Why digiKam shows/guesses some partial metadata and where it comes from, I have no idea, and makes me trust digiKam even less.

P.S. Please don't paste large output directly, attachments are preferred.

RafaelLinux commented 2 months ago

I don't understand what must I enable ("Please enable it and test again."). When exporting from DT, I only choose "Export" and maybe change some other parameters not related to metadata.

I reported it to DK developers and the fact is that if I export from DK, after enabling "Delegate in ExifTool engine all write operations" in configuration, the exported images from DK shows ALL metadata in JXL and TIFF.

This is one of the smallest raw files I have, I share it in case you want to use it for testing.

RafaelLinux commented 2 months ago

I have done more tests. I exported the same RAW from DK and from DT to JXL. Then I looked at the EXIF metadata with DK and with XNView. Both show metadata in the JXL exported from DK but no metadata in the JXL file exported from DT. Is it possible that there is some option I don't know about in DT that needs to be enabled?

kmilos commented 2 months ago

Is it possible that there is some option I don't know about in DT that needs to be enabled?

Yes. It's mentioned in the release notes and the manual. Please search the issues as well as this has been addressed several times already.

Example: https://github.com/darktable-org/darktable/issues/12448#issue-1367897005

All metadata boxes on the left need to be checked for JPEG XL.

RafaelLinux commented 2 months ago

Before your answer, I looked at the manual, and this is the configuration I had (the default). I tried adding some fields one by one, to check if this was the problem you were referring to. However, I found it too tedious and that's when I looked at the email again and saw your message.

I tried to activate the options you mentioned and indeed, now the metadata is exported correctly in JXL.

However, I am still very surprised that it is necessary to go through this process to be able to have the metadata in JXL as in any other format. It is not, in my opinion, something ‘obvious’ and neither is it documented that the user needs to do what you are now telling me.

Wouldn't it be more appropriate and simpler for the user if these options were activated (for all formats or when activating the JXL format)? For me, Darktable is a clearly professional and above all intuitive program, but this configuration is not intuitive.

kmilos commented 2 months ago

neither is it documented

It is explicitly called out in the release notes, as I mentioned several times.

It had to be done that way because of exiv2 limitations. "Opt in" rather than "opt out" is safer from user privacy and personal security standpoints.

ptilopteri commented 2 months ago

Rafael Linux User @.***> [09-07-24 16:54]:

Before your answer, I looked at the manual, and this is the configuration I had (the default). I tried adding some fields one by one, to check if this was the problem you were referring to. However, I found it too tedious and that's when I looked at the email again and saw your message.

I tried to activate the options you mentioned and indeed, now the metadata is exported correctly in JXL.

However, I am still very surprised that it is necessary to go through this process to be able to have the metadata in JXL as in any other format. It is not, in my opinion, something ‘obvious’ and neither is it documented that the user needs to do what you are now telling me.

Wouldn't it be more appropriate and simpler for the user if these options were activated (for all formats or when activating the JXL format)? For me, Darktable is a clearly professional and above all intuitive program, but this configuration is not intuitive.

but your idea of "default" is not necessarily everyone else's. I get the metadata that I want exported and did not have any problem setting the configuration.

there were very few "docs" when I started with dt.

-- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc

RafaelLinux commented 2 months ago

but your idea of "default" is not necessarily everyone else's. I get the metadata that I want exported and did not have any problem setting the configuration. there were very few "docs" when I started with dt. … -- (paka)Patrick Shanahan Plainfield,

I think the word ‘default’ is not the most appropriate, because it can indeed be interpreted. What is not interpretable is that when exporting to another format, you want to keep the resolution, the bit depth and yes, the metadata as well. From there, the modifications are options for the user. This is the usual way programmes work. I am not inventing anything new. Even when transcoding video, the metadata is maintained. If the user doesn't want it or wants to change it, it's a user choice, like changing the resolution.

kmilos commented 2 months ago

I have already explained that because of exiv2 limitations in supporting certain formats, there can't be talk of any "defaults". It's either all metadata or none.

Exporting all metadata without user's knowledge or consent could actually put their lives in danger in some extreme situations.

That is why it had to be made as explicitly opt-in.

Furthermore, this is clearly noted as a known issue/limitation in the release notes.

Please be mindful that is an issue tracker, and not a discussion forum. You have https://discuss.pixls.us/c/software/darktable/19 for that.

RafaelLinux commented 2 months ago

Yes, I am well aware of the dangers of metadata. It is up to an informed user to remove metadata before publishing. But it is at that moment, when it is necessary among other things to convert to web-compatible formats such as JPG, PNG or WEBP, that the user can decide to remove the metadata. And in the case of this ‘discussion’ we were talking about formats basically oriented to image storage, as you know, TIFF and JXL, so any photographer wants to keep the metadata in his stored images. I think the confusion comes from being clear about what each image storage format was created for.

I hope I have explained myself better, but as you say, this is not the place to talk about it, but I don't think it deserves a discussion thread because I am talking about facts, not user preferences.

victoryforce commented 1 month ago

And in the case of this ‘discussion’ we were talking about formats basically oriented to image storage, as you know, TIFF and JXL

Official description of the format says, “JPEG XL is designed to meet the needs of image delivery on the web and professional photography”. As you can see, there is no orientation to image storage here. This is just your perception of the JXL format (perhaps influenced by the fact that Google has long resisted supporting this format), but it is wrong.

The new iPhones will even shoot in this format and it is fully supported in the Apple ecosystem (Safari, operating systems). So it is clear that shots in this format will not be restricted to personal storage and a secure metadata approach should apply.

1div0 commented 1 month ago

Let's wait and C/C++ when there will be plenty of images in JPEG XL originating from iPhone 16.

4ND BTW FSCK 6006L3

kmilos commented 1 month ago

I am talking about facts

I think your time could be better spent contributing ISO BMFF write support to Exiv2.

RafaelLinux commented 1 month ago

Official description of the format says, “JPEG XL is designed to meet the needs of image delivery on the web and professional photography”. As you can see, there is no orientation to image storage here. This is just your perception of the JXL format (perhaps influenced by the fact that Google has long resisted supporting this format), but it is wrong.

The new iPhones will even shoot in this format and it is fully supported in the Apple ecosystem (Safari, operating systems). So it is clear that shots in this format will not be restricted to personal storage and a secure metadata approach should apply.

I think it's perfect that you put the original sentence in full. Professional photography is based on being able to retain most of the information from the original image, and this information is obviously not only pixels, but also metadata. JXL has full native support for EXIF, IPTC and XMP metadata (TIFF only natively supports EXIF and IPTC), plus JXL allows more to be added on a discretionary basis, and can store this metadata internally as JSON. This complete information, as we have already discussed, is unnecessary for the Web.

And why did I say that JXL is basically a storage format? Because as of today, only Safari - and now I understand why thanks to your comment - can decode JXL. Have you seen any page that uses JXL images? I doubt it, because it will be incompatible with the browsers of 78% to 80% of users (fortunately, there is a happier and freer life beyond the ‘Apple ecosystem’).

If you take the trouble, as I have done since the beginning of the year, to follow the progress of JXL implementation progress of JXL implementation in web browsers based on Blink or Gecko, you will find out that there are no intentions to implement JXL in those web engines. Why? Well, I'll summarize it for you so you don't have to waste time: Google has been developing JPEG-LI (I'll leave it to you to find out more about it, but I can tell you in advance that it is based on JXL) and therefore I predict that JXL will not be a format for the web, although it was one of its pillars.

Therefore, your final sentence is totally wrong. We'll see in a year or two.

RafaelLinux commented 1 month ago

I am talking about facts

I think your time could be better spent contributing ISO BMFF write support to Exiv2.

Of course, I always try to help, to the best of my knowledge, but I don't understand what I could contribute to the thread you link to. I can test (on my RAW files) and little else, unless you suggest something else.

parafin commented 1 month ago

Professional photography is based on being able to retain most of the information from the original image, and this information is obviously not only pixels, but also metadata.

Professional photographers surely read the manual for their tools (be it camera or software) and double-check, that metadata they care about is retained. Otherwise what actually does “professional” mean?

On the other hand other users, who don’t care about metadata enough and so don’t touch the defaults, shouldn’t be exposed to private information leakage (like geolocation). This is independent from chosen format, because splitting them to “storage” and “share” ones is total nonsense and at the very least very obscure for the users - why would behaviour differ just because you’ve changed one unrelated technical setting? Of course currently there is an issue with “all or nothing” metadata for some formats, so behaviour does differ, but it’s a technical limitation, and the decision was made that it’s better to export less than requested than more, again for privacy reasons. If you don’t care about privacy, or deal with it differently, doesn’t mean that it’s not a valid argument.

RafaelLinux commented 1 month ago

Professional photographers surely read the manual for their tools (be it camera or software) and double-check, that metadata they care about is retained. Otherwise what actually does “professional” mean?

On the other hand other users, who don’t care about metadata enough and so don’t touch the defaults, shouldn’t be exposed to private information leakage (like geolocation). This is independent from chosen format, because splitting them to “storage” and “share” ones is total nonsense and at the very least very obscure for the users - why would behaviour differ just because you’ve changed one unrelated technical setting? Of course currently there is an issue with “all or nothing” metadata for some formats, so behaviour does differ, but it’s a technical limitation, and the decision was made that it’s better to export less than requested than more, again for privacy reasons. If you don’t care about privacy, or deal with it differently, doesn’t mean that it’s not a valid argument.

Darktable (at least for me) is a professional tool at the level of LightRoom or “Capture ONE”. I'll tell you to first take the trouble to install those applications and export to WebP and JPG. You will see that both the mentioned applications keep BY DEFAULT the geolocation among their metadata, unless expressly the user restricts at that time or in the previous configuration the metadata to be exported.

Cause intelligent and well-informed users are able to decide what information they want to include or not in their export formats, they don't need an application to limit by default what they can export arguing privacy reasons. Limiting something, by default, is also somewhat inconsistent being free software.

Following absurd arguments, when exporting photos with faces of people who may be minors, Darktable should by default block the export or blur their faces, don't you think?

parafin commented 1 month ago

Intelligent and well-informed users are able to control the preferences, that's the point. Just because defaults work like you describe in other software doesn't mean it's a good idea. Difference between faces and metadata, is that faces are in the picture, so it's impossible to miss the fact that they are there, while amount of included metadata for many users is a surprise. So your absurd argument doesn't hold.

RafaelLinux commented 1 month ago

Intelligent and well-informed users are able to control the preferences, that's the point.

Exactly, although it was not your intention, you are proving me right. Let the user choose, you should not limit the user by your own fears or limitations.

Just because defaults work like you describe in other software doesn't mean it's a good idea.

Again, what under your criteria should not be so, happens to be accepted and validated by the vast majority of users of the mentioned applications, which multiply by several times the number of Darktable users. But all of us who clearly understand that limitations should not be imposed are wrong, of course. Do you mean to tell me that Darktable users are less savvy than those of paid applications and that's why you have to limit their options by default?

The difference between faces and metadata, is that the faces are in the image, so it is impossible to overlook the fact that they are there,

I was taking for granted that you know that face recognition algorithms exist, but I see that you don't. And not all users know that they cannot publish faces of minors under certain legislations, so implementing that functionality to protect something as important as the rights of third parties (minors in this case) would be as appropriate as limiting the metadata you export.

while the amount of metadata included for many users is a surprise. So your absurd argument doesn't hold water.

If the metadata contained in an image is a surprise to a user, then that user is uninformed and doesn't even know what metadata is, so I doubt very much that such a user would use a RAW editing tool. I have seen a person drive nails with the wooden handle of a hammer, instead of the metal head of the hammer. can it be done? Yessssss... Is that the right way to use it? ... I'm starting to doubt it after reading your theories. I'm starting to think that “it's not a good idea” and maybe the hammer head should be made out of wood as well, so that the user of the hammer doesn't get hurt.

parafin commented 1 month ago

You keep saying, that user options are limited by darktable's defaults, but in the same sentence you say, that users should be let to choose. They can choose by modifying the preferences, and therefore default values don't limit them.

You basically intentionally ignore any arguments against your position, therefore continuing this discussion is pointless.

RafaelLinux commented 1 month ago

You keep saying, that user options are limited by darktable's defaults, but in the same sentence you say, that users should be let to choose. They can choose by modifying the preferences, and therefore default values don't limit them.

Exporting to another format, as everybody knows, is to transform the information (metadata included) from one format to another. Assuming that the image format to which I convert supports the same characteristics of the original image format both at the image level (color depth, color profiles, etc.) and at the metadata field level, the resulting export file should retain the same information as the original image. If the user wants to alter or truncate part of the metadata, it is up to him to decide, just as he does if he wants to alter the image size. It is my data and, from that point, I decide what I want to keep and what not. Something so simple and that, you don't want to understand. My 16 year old son understands it and I don't know how to explain it to you anymore, sorry.

You basically intentionally ignore any arguments against your position, therefore continuing this discussion is pointless.

Perhaps you should understand the arguments instead of just reading them to prepare a sentence to try to refute them with.

wpferguson commented 1 month ago

IIRC the reason we don't export metadata by default was the enactment of GDPR.

RafaelLinux commented 1 month ago

IIRC the reason we don't export metadata by default was the enactment of GDPR.

I live in the European Union and I know that law, so it gives me the impression that you may have misunderstood it, which usually happens because everything is subject to interpretation if you do not take into account the details.

In the case we are talking about, the storage of our own photos on our hard drive (not in the cloud) the GDPR does not affect at all because it is considered personal and domestic use, which is exempt from GDPR obligations under Article 2(2)(c), i.e. photographs taken and used solely for personal and domestic purposes are exempt from GDPR obligations.

It is in the case of intending to publish these photos on the Internet or transfer them to third parties, when it would be necessary to take into account in which cases the GDPR must be applied, but not only attending to the metadata as it is being mentioned repeatedly, but also attending to the CONTENT of the photographs, since the GDPR will apply if the photograph shows:

Clearly a person: For example, if a face is easily recognizable.
Details that make it possible to identify a person: For example, a distinctive tattoo, a unique item of clothing or a very specific location that is associated with a specific person or shows a street address or location.
To a group of people and can identify someone in particular: For example if a person is easily distinguishable in a group.

The user is the one who must be aware that by publishing photos with this metadata on the Internet, he/she MAY be disclosing relevant information.

In summary, the GDPR does not set limits on the storage and digital transformation of images and their metadata in the personal or professional sphere, it does set limits on their use (publication), especially when it comes to images where people appear.

Darktable is, as its name derived from analog processing indicates, a “preprocessing” tool. For post-processing, there are plenty of batch image management tools, which are the ones that are really designed to perform bulk resizing, color profiling, metadata anonymization and exporting to standard web image formats. And such tools exist precisely because they are the right ones to make those changes that are necessary before publishing to the Internet.

As an interesting information, I develop corporate websites, where users autonomously publish both images in standard formats for the web (JPG, WebP and PNG) and documents (PDF, ODT, ODS). All these files are uploaded with their metadata (yes, PDFs also have metadata) but since I am aware of the RGPD, I automatically apply specific anonymization tools on the server on the metadata of each new file that a user uploads to the web. And it is not something special that I do, it is done by many other websites that are aware of the law.

darktable-org / darktable