Closed douzzer closed 6 months ago
seems to be dependent on camera or lens:
Canon 7D:
exiv2 -pt --grep lens/i /c/Users/.../20130625-IMG_5459.CR2
Exif.CanonCs.LensType Short 1 Canon EF 24-105mm f/4L IS USM
Exif.CanonCs.Lens Short 3 24.0 - 105.0 mm
Exif.Canon.LensModel Ascii 74 EF24-105mm f/4L IS USM
Exif.CanonLe.LensSerialNumber SLong 1 1980760064
Exif.Photo.LensSpecification Rational 4 24-105mm
Exif.Photo.LensModel Ascii 23 EF24-105mm f/4L IS USM
Canon M6:
exiv2 -pt --grep lens/i /c/Users/.../20170604-IMG_0790.CR2
Exif.CanonCs.LensType Short 1 Unknown Lens (254)
Exif.CanonCs.Lens Short 3 100.0 mm
Exif.Photo.LensSpecification Rational 4 100mm
Exif.Photo.LensModel Ascii 28 EF100mm f/2.8L Macro IS USM
Exif.Photo.LensSerialNumber Ascii 11 0000008672
Canon 5DmIV:
exiv2 -pt --grep lens/i /c/Users/.../20210227-IMG_0511.CR2
Exif.CanonCs.LensType Short 1 Unknown Lens (747)
Exif.CanonCs.Lens Short 3 100.0 - 400.0 mm
Exif.Canon.LensModel Ascii 74 EF100-400mm f/4.5-5.6L IS II USM
Exif.CanonLe.LensSerialNumber SLong 1 -1879011276
Exif.CanonAfC.USMLensElectronicMF SLong 1 Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible SLong 1 Continue Focus Search
Exif.Photo.LensSpecification Rational 4 100-400mm
Exif.Photo.LensModel Ascii 33 EF100-400mm f/4.5-5.6L IS II USM
Exif.Photo.LensSerialNumber Ascii 11 3490009075
Canon R6 - EF Lens: exiv2 -pt --grep lens/i /c/Users/.../IMG_3123.CR3
Exif.Photo.LensSpecification Rational 4 100mm
Exif.Photo.LensModel Ascii 28 EF100mm f/2.8L Macro IS USM
Exif.Photo.LensSerialNumber Ascii 11 0000008672
Exif.CanonCs.LensType Short 1 Unknown Lens (254)
Exif.CanonCs.Lens Short 3 100.0 mm
Exif.CanonFi.RFLensType SShort 1 n/v
Exif.Canon.LensModel Ascii 138 EF100mm f/2.8L Macro IS USM
Exif.CanonLe.LensSerialNumber SLong 1 -2046820352
Exif.CanonAfC.USMLensElectronicMF SLong 1 Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible SLong 1 Continue Focus Search
Canon R6 - RF Lens:
exiv2 -pt --grep lens/i /c/Users/.../20231008-IMG_0599.CR3
Exif.Photo.LensSpecification Rational 4 24-105mm
Exif.Photo.LensModel Ascii 23 RF24-105mm F4 L IS USM
Exif.Photo.LensSerialNumber Ascii 11 7413007751
Exif.CanonCs.LensType Short 1 Canon RF 24-105mm F4-7.1 IS STM
Exif.CanonCs.Lens Short 3 24.0 - 105.0 mm
Exif.CanonFi.RFLensType SShort 1 Canon RF 24-105mm F4L IS USM
Exif.Canon.LensModel Ascii 138 RF24-105mm F4 L IS USM
Exif.CanonLe.LensSerialNumber SLong 1 1996493684
Exif.CanonAfC.USMLensElectronicMF SLong 1 Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible SLong 1 Continue Focus Search
I've noticed something interesting about the Exif.CanonLe.LensSerialNumber
values shown above. -1879011276 in hex is 90009034
, which is a rearrangement of the first eight characters of the Exif.Photo.LensSerialNumber
value 3490009075
. Similarly -2046820352 in hex is 86000000
for a lens with serial number 0000008672
, and 1996493684 in hex is 77001374
, serial number 7413007751
.
My Canon EOS 100D with 18-55mm kit lens reports:
Exif.CanonLe.LensSerialNumber SLong 1 422576128
Exif.CanonLe.0x0001 SLong 1 129
Exif.Photo.LensSerialNumber Ascii 11 0000301981
422576128 in hex is 19300000
and 129 in hex is 81
. Join those bytes together in the right order and you get the complete serial number 0000301981
.
I spotted this while investigating #2138 - is it possible that Exif.CanonLe.LensSerialNumber
should actually be a 6-byte hex coded value instead of a 4 byte integer?
A brief follow-up:
Yesterday, motivated by CVE-2023-44398, I finally put together a workaround that restored functionality (and then some) for Canon lenses with extenders. To get it to work right, two patches were needed:
src/canonmn_int.cpp
, so that the mapping from lens ID number to model name is unique and well-behaved.My strategy isn't generally applicable because it removes all support for 3rd party EF mount lenses with conflicting ID numbers, but it probably points in the direction of how to resolve the problem.
And it is well to note, the problem seems to be a system-level interaction between exiv2 and lensfun.
Lensfun patch attached for reference. parse_lens_name_extenders.patch.txt
Hello all.... forgive me, I did'nt read to all this thoroughly, as I am not feeling well.
I have similar case (gentoo, exiv2-0.28.1) but with Nikon camera and Tamron lense (70/200 2.8 Di G2) it was recognized well in the past, but not anymore. It get listed as ID=226, even well implemented in ~/.exiv2
BTW gentoo also has a lensfun trouble. Only Version 0.3.4_rc1 is available, which is afaik not the official stable version.
With that 0.3.4_rc1, the lensfun-update-data does not work (bug is opened since long time). Donno how much this can be related
either @douzzer or maybe @kmilos would you mind changing the title, as I have strong indication it is not just canon lenses but also the Tamron 70/200 2.8 G2 connected to my Nikon D750?
Do you need me to provide some prove? If yes, what do you need me to do?
Do you need me to provide some prove?
Yes, please. For example exiv2 ex
extracted data and exiv2 -Pkvt
output for the current 0.28.1 failing case and the previously working older one.
@kmilos per your request :)
the current situation:
$ exiv2 ex 2023-12-29_151301_AG7_9385.NEF
Warning: Exif tag Exif.NikonPreview.JPEGInterchangeFormatLength not encoded
Warning: Exif IFD NikonPreview not encoded
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.NikonSi02xx.0x027a not encoded
Warning: Exif tag Exif.Nikon3.0x00bc not encoded
and here the output of -Pkvt exiv2_current_Pkvt_20240109.txt
and here comes the former scenario (hopefully because I cannot downgrade to exiv2-0.27......
first a screenshot:
$ exiv2 ex 2018-01-08_180136_AG7_1503.NEF
Warning: Exif tag Exif.NikonPreview.JPEGInterchangeFormatLength not encoded
Warning: Exif IFD NikonPreview not encoded
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.NikonSi02xx.0x027a not encoded
Warning: Exif tag Exif.Nikon3.0x00bc not encoded
and here the output of -Pkvt
exiv2_former-scenario_Pkvt_20240109.txt
I used diff and there is no difference :-(
my .exiv2
$ cat .exiv2
# how to locate, where this file should be:
# exiv2 --verbose --version --grep config_path
# first approach should be:
# exiftool -G -n -Lens* PathToFileandFilename
# then use something like:
# [MakerNotes] Lens ID Number : 226
# or if there is no Lens ID Number
# exiv2 -pv --grep lenstype/i PathToFileandFilename
# then use something like:
# 0x0201 OlympusEq LensType Byte 6 2 0 5 16 0 17
[nikon]
#
#226=Tamron/ Tamron SP 70-200mm F/2.8 Di VC USD G2
#226=Tamron SP 70-200mm F/2.8 Di VC USD G2
226=Tamron SP 70-200mm F/2.8 Di VC USD G2
[olympus]
2 0 5 16 0 17=Lumix G 20mm F1.7 II Asph.
#0 0 35 0 1 0=Funny Wee Lens
[canon]
137=Tamron SP 24-70mm F/2.8 Di VC USD
502=TAMRON 17-35mm F/2.8-4 Di LD Aspherical (IF)
Tonight I have to focus on other priorities, so please understand, I will be a bit silent, but that topic really bothers me...
Thanks for the data.
I used diff and there is no difference :-(
As the output between 0.27.x and 0.28.x is the same, and this lens was never in the internal database, can't really say this is the same bug.
Now added in https://github.com/Exiv2/exiv2/pull/2889
@kmilos my data "former scenario" was also created with exiv2-0.28.1, just old files, where I did refresh the metadata via darktable at that time.
I have dependency blockage and hence cannot downgrade exiv2. That blockage is tricky for some reason, but this would digress here too much. Maybe in the next few days that resolves (as kde currently not fully upgraded due to missing other deps)
My major trouble is, that the ~/.exiv2 is not obayed
my data "former scenario" was also created with exiv2-0.28.1, just old files, where I did refresh the metadata via darktable at that time
Well, that's not very useful here.
that topic really bothers me...
My major trouble is, that the ~/.exiv2 is not obayed
Back to square one then. We'll just have to wait until someone w/ "skin in the game" (or otherwise interested) digs into the code and fires up a debugger.
my data "former scenario" was also created with exiv2-0.28.1, just old files, where I did refresh the metadata via darktable at that time
Well, that's not very useful here.
I am really sorry for this. Currently I do not see a way to downgrade exiv2 in gentoo.
I will amend as soon as any possible
@kmilos
I made it to downgrade exiv-0.27.7 (finally it was my boon mistake why I stuck, I am embarrassed :-( )
The output of "exiv2 ex [File]" looks same but actually the files are not same, when I diff them
exiv2-0.28.1
$ exiv2 ex 2023-12-29_151301_AG7_9385.NEF
Warning: Exif tag Exif.NikonPreview.JPEGInterchangeFormatLength not encoded
Warning: Exif IFD NikonPreview not encoded
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.NikonSi02xx.0x027a not encoded
Warning: Exif tag Exif.Nikon3.0x00bc not encoded
exiv-028_2023-12-29_151301_AG7_9385.exv.zip
exiv2-0.28_current_Pkvt_20240109.txt
exiv2-0.27.7
$ exiv2 ex 2023-12-29_151301_AG7_9385.NEF
Warning: Exif tag Exif.NikonPreview.JPEGInterchangeFormatLength not encoded
Warning: Exif IFD NikonPreview not encoded
Warning: Exif tag Exif.Photo.MakerNote not encoded
Warning: Exif tag Exif.NikonSi02xx.0x027a not encoded
Warning: Exif tag Exif.Nikon3.0x00bc not encoded
exiv-027_2023-12-29_151301_AG7_9385.exv.zip
exiv2-0.27_current_Pkvt_20240112.txt
So if I do a diff between the two Pkvt outputs, I get: (left is old 0.27.7 and right is new 0.28.1 with the problem not to obay ~/.exiv2)
$ diff ~/analysis/exiv2-0.27_current_Pkvt_20240112.txt ~/analysis/exiv2-0.28_current_Pkvt_20240109.txt
108c108
< Exif.NikonPc.QuickAdjust 255 n/a
---
> Exif.NikonPc.QuickAdjust 255 n/v
153c153
< Exif.NikonLd3.LensIDNumber 226 Tamron SP 70-200mm F/2.8 Di VC USD G2
---
> Exif.NikonLd3.LensIDNumber 226 226
164c164,189
< Exif.Nikon3.FlashInfo 48 49 48 55 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 48 49 48 55 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
---
> Exif.NikonFl7.Version 48 49 48 55 1.07
> Exif.NikonFl7.FlashSource 0 Keine
> Exif.NikonFl7.ExternalFlashFirmware 0 n/v
> Exif.NikonFl7.ExternalFlashData1 0 External flash off
> Exif.NikonFl7.ExternalFlashData2 0 Aus
> Exif.NikonFl7.FlashCompensationMaster 0 n/v
> Exif.NikonFl7.FlashFocalLength 0 n/v
> Exif.NikonFl7.RepeatingFlashRate 0 n/v
> Exif.NikonFl7.RepeatingFlashCount 0 n/v
> Exif.NikonFl7.FlashGNDistance 0 n/v
> Exif.NikonFl7.FlashColorFilter 0 Keine
> Exif.NikonFl7.FlashGroupAControlData 0 Aus
> Exif.NikonFl7.FlashGroupBCControlData 0 Aus, Aus
> Exif.NikonFl7.FlashCompensationGroupA 0 n/v
> Exif.NikonFl7.FlashCompensationGroupB 0 n/v
> Exif.NikonFl7.FlashCompensationGroupC 0 n/v
> Exif.NikonFl7.ExternalFlashData3 0 Exposure Comp.: Entire frame
> Exif.NikonFl7.CameraFlashOutput 0 0.0 EV
> Exif.NikonFl7.CameraExposureCompensation 0 0.0 EV
> Exif.NikonFl7.CameraFlashCompensation 0 0.0 EV
> Exif.NikonFl7.ExternalFlashData4 0 Illumination Pat.: Standard
> Exif.NikonFl7.FlashZoomHeadPosition 0 n/v
> Exif.NikonFl7.FlashMasterOutput 0 n/v
> Exif.NikonFl7.FlashGroupAOutput 0 n/v
> Exif.NikonFl7.FlashGroupBOutput 0 n/v
> Exif.NikonFl7.FlashGroupCOutput 0 n/v
183c208
< Exif.NikonAf2.ContrastDetectAFInFocus 0 0
---
> Exif.NikonAf2.ContrastDetectAFInFocus 0 Nein
Thanks @AxelG-DE for your effort, that's more useful. 👍
However, I can't reproduce locally (w/ 0.28.x branch), using either of your .exv
files and this exiv2.ini
(I'm on Windows):
[nikon]
226=Foo Bar 3
I get
Exif.NikonLd3.LensIDNumber 226 Foo Bar 3
Just a wild guess: perhaps Gentoo maintainers (and other distros having similar issues) forgot to add and enable the inih dependency needed for parsing these config files since 0.28.0?
Edit: Yep. Didn't forget, but not enabled on Gentoo - @douzzer as well, so please file bug there so it's tracked somewhere. @MStraeten your (Canon only?) case might still be different/unrelated I think.
@kmilos seems you have the right smell
from the ebuild (that controls how a package will be built and installed):
multilib_src_configure() {
local mycmakeargs=(
-DEXIV2_BUILD_SAMPLES=NO
-DEXIV2_ENABLE_BROTLI=OFF
-DEXIV2_ENABLE_NLS=$(usex nls)
-DEXIV2_ENABLE_PNG=$(usex png)
-DEXIV2_ENABLE_CURL=$(usex webready)
-DEXIV2_ENABLE_INIH=OFF # multilib/libdir logic is broken
-DEXIV2_ENABLE_WEBREADY=$(usex webready)
-DEXIV2_ENABLE_XMP=$(usex xmp)
-DEXIV2_ENABLE_BMFF=$(usex bmff)
Yep. As requested, let's please not discuss the Gentoo problem here any further but move it to their tracker (as Team Exiv2 has nothing to do with how distros package), and leave this to analysing the remaining issue @MStraeten is seeing.
@kmilos I have filed an upstream bug https://bugs.gentoo.org/921937
I can confirm that this bug also affect myself. Is there change that you'll fix this?
With the config file problem out of the way (although one place I see inih
missing still is OpenBSD @ajacoutot), it would appear there is still the occasional (and not easy to reproduce) misdetection left...
@hassec @neheb I'm starting to wonder if this is also maybe related to the switch to std::regex
and its different and flaky implementations, as I can't seem to reproduce in my environment (MSYS2 UCRT64 nor CLANG64)?
I get Unknown Lens from darktable on all my Canon lenses (35-80mm, 18-135mm, 70-300mm) but NOT FOR ALL SHOTS Identification seems to succeed for photos taken with focal lengths on the lower side of the lens range and fail for the upper side. All photos are RAW format but I have one JPEG case too Just took 6 shots with the 35-80mm. For shots at focal lengths 35, 43 and 48mm, the lens was correctly identified as Canon EF 35-80mm f/4-5.6 USM by darktable but got Unknown Lens (193) for shots at 59, 65 and 80mm. The 35-80 is a rather old faithful horse (1992!), but the pattern seems to replicate for the other Canon lenses I own. Shots were done on a Canon 60D (firmware 1.1.2) and imported to darktable 4.6.1 on a Mac There can be no question about corrupt exif data since Canon's Digital Photo Professional 4 correctly identified the lenses on all cases as did Mac's Preview(!) I can provide a couple of "film rolls" where this behaviour can be observed
git blame shows https://github.com/Exiv2/exiv2/commit/bdd8a386b5 to be the problem.
@joaobrisson please provide example files, then I'll try to have a look if I can find the time.
@joaobrisson https://github.com/joaobrisson please provide example files.
gladly need mail address to send 19 raw files totalling 447 MB (wetransfer?) it's 3 series of photos taken with 3 different Canon lenses at focal lengths covering each lens focal length range
@joaobrisson would you mind just extracting the exif from your raw files and attaching a zip archive of the *.exv
files instead? Those can then also directly be used to extend our test suite. If ok with you, please also state that you share these files under e.g. CC0 or another permissive license such that we are allowed to include them in the tests.
The steps are explained in the creating a test file wiki.
@joaobrisson https://github.com/joaobrisson would you mind just extracting the exif from your raw files and attaching a zip archive of the *.exv files instead?
here they are together with a .csv describing the files and lens ID status
@joaobrisson you need to attach them to the actual comment on github.com. The email reply doesn't add them.
here they are together with a .csv describing the files and lens ID status
@joaobrisson thanks for sharing those files.
I just tried to reproduce the problem but using exiv2 0.28.2
I see the following:
❯ exiv2 -pa -g LensType IMG_*.exv
IMG_0001.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0002.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0003.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0004.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0005.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0006.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM
IMG_0007.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0008.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0009.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0010.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0011.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0012.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS
IMG_0013.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0014.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0015.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0016.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0017.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0018.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
IMG_0019.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM
What version of exvi2
and what operating system are you on?
Also please see my earlier comment regarding a license:
... If ok with you, please also state that you share these files under e.g. CC0 or another permissive license such that we are allowed to include them in the tests.
files are shared under CC0 and I am willing to share the original CR2 files under the same license for testing
to answer your early question and make things weirder
exiv2 0.28.2 on MacOS
however repeat of your test held
% exiv2 -pa -g LensType IMG_*.exv \IMG_0001.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM IMG_0002.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM IMG_0003.exv Exif.CanonCs.LensType Short 1 Canon EF 35-80mm f/4-5.6 USM IMG_0004.exv Exif.CanonCs.LensType Short 1 Unknown Lens (193) IMG_0005.exv Exif.CanonCs.LensType Short 1 Unknown Lens (193) IMG_0006.exv Exif.CanonCs.LensType Short 1 Unknown Lens (193) IMG_0007.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS IMG_0008.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS IMG_0009.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS IMG_0010.exv Exif.CanonCs.LensType Short 1 Canon EF-S 18-135mm f/3.5-5.6 IS IMG_0011.exv Exif.CanonCs.LensType Short 1 Unknown Lens (51) IMG_0012.exv Exif.CanonCs.LensType Short 1 Unknown Lens (51) IMG_0013.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM IMG_0014.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM IMG_0015.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM IMG_0016.exv Exif.CanonCs.LensType Short 1 Canon EF 70-300mm f/4-5.6 IS USM IMG_0017.exv Exif.CanonCs.LensType Short 1 Unknown Lens (238) IMG_0018.exv Exif.CanonCs.LensType Short 1 Unknown Lens (238) IMG_0019.exv Exif.CanonCs.LensType Short 1 Unknown Lens (238)
search for lens model did not flinch
% exiv2 -pa -g LensModel IMG_*.exv IMG_0001.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0001.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0002.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0002.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0003.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0003.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0004.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0004.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0005.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0005.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0006.exv Exif.Canon.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0006.exv Exif.Photo.LensModel Ascii 70 EF35-80mm f/4-5.6 USM IMG_0007.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0007.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0008.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0008.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0009.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0009.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0010.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0010.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0011.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0011.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0012.exv Exif.Canon.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0012.exv Exif.Photo.LensModel Ascii 70 EF-S18-135mm f/3.5-5.6 IS IMG_0013.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0013.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0014.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0014.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0015.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0015.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0016.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0016.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0017.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0017.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0018.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0018.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0019.exv Exif.Canon.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM IMG_0019.exv Exif.Photo.LensModel Ascii 70 EF70-300mm f/4-5.6 IS USM
On MacOS Preview Inspector Exif tab lenses ids are shown twice, first as "EF...." and then as "Canon EF..."
Last, the Metadata Converter app on MacOS correctly identifies the lenses on all cases
May I hint that this is a problem with the exiv2 implementation for MacOS?
May I hint that this is a problem with the exiv2 implementation for MacOS?
I already hinted above std::regex
implementation is not to be trusted across platforms.
Another idea: regex could also be affected by locale settings?
@joaobrisson My test was also on macOS.
Are you on an ARM based mac (like M1-M3 processor) or x86? How was exiv2 installed, or did you compile it from source? If so, which flags? What is the output of the following commands:
exiv2 --version --verbose
(you can skip the lines starting with xmlns=....
)file $(which exiv2)
otool -L $(which exiv2)
locale
@kmilos without a reproducer or actual evidence, just blaming it on std::regex
isn't very productive. So let's focus on getting a reproducer that one can debug and fix.
x86 64 bit got exiv2 from https://macappstore.org/exiv2/
% exiv2 --version --verbose exiv2 0.28.2 exiv2=0.28.2 platform=apple compiler=Clang bits=64 dll=1 debug=0 cplusplus=201703 version=Apple LLVM 14.0.0 (clang-1400.0.29.202) date=Feb 13 2024 time=18:49:36 processpath=/usr/local/bin localedir=../share/locale package_name=exiv2 curlprotocols=dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp curl=1 executable=/usr/local/Cellar/exiv2/0.28.2/bin/exiv2 library=/usr/local/Cellar/exiv2/0.28.2/lib/libexiv2.0.28.2.dylib library=/usr/local/Cellar/gettext/0.22.5/lib/libintl.8.dylib library=/usr/local/Cellar/brotli/1.1.0/lib/libbrotlicommon.1.1.0.dylib library=/usr/local/Cellar/brotli/1.1.0/lib/libbrotlidec.1.1.0.dylib library=/usr/local/Cellar/inih/58/lib/libinih.0.dylib library=/usr/local/Cellar/inih/58/lib/libINIReader.0.dylib library=/usr/lib/system/libsystem_kernel.dylib library=/usr/lib/system/libsystem_platform.dylib library=/usr/lib/system/libsystem_pthread.dylib have_inttypes=1 have_libintl=1 have_lensdata=1 have_iconv=1 have_memory=1 have_stdbool=1 have_stdint=1 have_stdlib=1 have_strlib=0 have_strerror_r=1 have_strings_h=0 have_mmap=1 have_munmap=1 have_sys_stat=1 have_unistd_h=0 have_sys_mman=1 have_libz=1 have_brotli=1 have_xmptoolkit=1 adobe_xmpsdk=0 have_bool=0 have_strings=1 have_sys_types=1 have_unistd=1 enable_bmff=1 enable_webready=1 enable_nls=1 enable_video=1 use_curl=1 config_path=/Users/brisson/.exiv2 uid=502 euid=502 gid=20
file $(which exiv2) /usr/local/bin/exiv2: Mach-O 64-bit executable x86_64
% otool -L $(which exiv2) /usr/local/bin/exiv2: /usr/local/Cellar/exiv2/0.28.2/lib/libexiv2.28.dylib (compatibility version 28.0.0, current version 0.28.2) /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 13.0.0, current version 13.0.0) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.23.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)
% locale LANG="pt_PT.UTF-8" LC_COLLATE="pt_PT.UTF-8" LC_CTYPE="pt_PT.UTF-8" LC_MESSAGES="pt_PT.UTF-8" LC_MONETARY="pt_PT.UTF-8" LC_NUMERIC="pt_PT.UTF-8" LC_TIME="pt_PT.UTF-8" LC_ALL=
@zisoft is reproducing this on macOS as well
@zisoft also on a x86 mac?
The weird thing for me is, that the command line gives fine results for my Sigma lens but darktable reports Unknown Lens (183)
.
~ exiv2 -pt --grep lens/i img_sigma.CR2
Exif.CanonCs.LensType Short 1 Sigma 105mm f/2.8 EX DG OS HSM Macro
Exif.CanonCs.Lens Short 3 105.0 mm
Exif.Canon.LensModel Ascii 74 105mm
Exif.CanonLiOp.DigitalLensOptimizer SLong 1 Off
Exif.CanonLe.LensSerialNumber Byte 5 0000000000
Exif.CanonAfC.USMLensElectronicMF SLong 1 Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible SLong 1 Continue Focus Search
Exif.Photo.LensSpecification Rational 4 105mm
Exif.Photo.LensModel Ascii 6 105mm
Exif.Photo.LensSerialNumber Ascii 11 0000000000
~ exiv2 -pe --grep lens/i img_sigma.CR2
Exif.CanonCs.LensType Short 1 183
Exif.CanonCs.Lens Short 3 105 105 1
Exif.Canon.LensModel Ascii 74 105mm
Exif.CanonLiOp.DigitalLensOptimizer SLong 1 0
Exif.CanonLe.LensSerialNumber Byte 5 0 0 0 0 0
Exif.CanonAfC.USMLensElectronicMF SLong 1 0
Exif.CanonAfC.LensDriveWhenAFImpossible SLong 1 0
Exif.Photo.LensSpecification Rational 4 105/1 105/1 0/1 0/1
Exif.Photo.LensModel Ascii 6 105mm
Exif.Photo.LensSerialNumber Ascii 11 0000000000
To me, it looks like Exiv2::ExifData &exifData = image->exifData();
is using -pe
internally.
@zisoft also on a x86 mac?
Yes, MacBook Pro 2019 (Intel)
The weird thing for me is, that the command line gives fine results for my Sigma lens but darktable reports
Unknown Lens (183)
.
How about locales, are both terminal and darktable running as de_DE for you?
How about locales, are both terminal and darktable running as de_DE for you?
Good catch!
I have my terminal's locale usually set to en_US.UTF-8
. Setting the locale to de_DE.UTF-8
now gives:
~ exiv2 -pt --grep lens/i img_sigma.CR2
Exif.CanonCs.LensType Short 1 Unknown Lens (183)
Exif.CanonCs.Lens Short 3 105.0 mm
Exif.Canon.LensModel Ascii 74 105mm
Exif.CanonLiOp.DigitalLensOptimizer SLong 1 Aus
Exif.CanonLe.LensSerialNumber Byte 5 0000000000
Exif.CanonAfC.USMLensElectronicMF SLong 1 Enable After AF
Exif.CanonAfC.LensDriveWhenAFImpossible SLong 1 Continue Focus Search
Exif.Photo.LensSpecification Rational 4 105mm
Exif.Photo.LensModel Ascii 6 105mm
Exif.Photo.LensSerialNumber Ascii 11 0000000000
Yeah! It's the locale on macOS! The DE locale yelds the same behaviour as the PT locale for both exiv2 and darktable (Refresh Exif): Unknown Lens! But try switching to the UK and all of a sudden you get it right
So, for the time being, the workaround seems to be to set the locale to the UK or the US when importing to darktable while waiting for the exiv2 team to come up with some fix
It's the locale on macOS!
Sure it is macOS only?
Sure it is macOS only?
Yes, I have tried both DE and PT locales and exiv2 and darktable fail
But when I switched to UK locale, everything worked to perfection
My questions was, is this a general locale issue in exiv2 on each system, or is this macOS specific?
@hassec std:stof
seems to be locale dependent (expecting ,
as decimal separator in both de_DE and pt_PT). So it shouldn't be macOS only. Not std::regex
it would seem after all, but it did come in w/ that commit.
@hassec Maybe swicth to std::from_chars
?
We had a similar issue in darktable recently: Fix locale for aperture filter. There f/5.6
becomes f/5,6
in some locales.
Solution was to switch locale for LC_NUMERIC
to C
but this needs to be done thread-safe:
Describe the bug
Up to and including 0.27.7-release, I'm able to get correct lens detection for this lens setup in Darktable, by putting this in
~/.exiv2
:But with 0.28.0-release, that doesn't work anymore, and the lens is misdetected as Tamron. Nothing else I've tried has worked. In particular, when I tried patching out the conflicting Tamron lenses, Darktable reported "Lens 748 not found" or words to that effect, even though obviously the Canon 748 entry is still there. This is the patch I tried:
The most worrisome syndrome though is that the override entry in
~/.exiv2
is being ignored.To Reproduce
I've posted a CR2 with this lens here
(github has a 25MB attachment limit, and doesn't allow CR2 files besides.)
I'm running Darktable 4.4.2 but it wasn't the variable, and it's detecting the lens correctly when built and linked against exiv2-0.27.7.
To see the syndrome, set up the
~/.exiv2
above, import the picture into DT, and admire the detected lens ID on the left.Note that you'll see the no-lens-info error in the lens correction tool for both the Canon and Tamron versions of ID 748. I run a Lensfun with the correction data for the Canon setup patched in.
Expected behavior
In the left pane with the image attributes, the lens entry should read
(Which is what it says when built+linked with v0.27.7, and it's what I have in my
~/.exiv2
.)Desktop (please complete the following information):
Additional context
I believe this DT issue relates: https://github.com/darktable-org/darktable/issues/14882