Canon EF 28-105 Lens Recognition

Describe the bug

When probing exif information with exiv2 for a specific camera/lens combination (see below), for a relevant fraction of the images the lens gets not identified correctly: Although the ID seems to match, exiv2 is unable to identify the LensType-String.

There is no obvious difference between the files.
It occurs for images which where taken seconds apart
It is reproducible per image, i.e. if the LensType cannot be identified by exiv2 the first time, the behaviour remains the same for consecutive calls to exiv2

This also affects downstreams products (as in this case darktable - the import stores the ID instead of the name, and automatic lens correction fails)

To Reproduce

Run exiv2 as shown below on the two provided, consecutively taken images.

-pv: identical output concerning the LensType: 156
-pa / -pt: For one image, the lens gets identified correctly, for the other just the id is shown

Expected behavior

exiv2 should resolve the lens for both images correctly, giving the desired output ("EF28-105mm f/3.5-4.5 USM").

Workaround

Place the lens identification in exiv2 config file:

[canon]
156=Canon EF 28-105mm f/3.5-4.5 USM

Desktop:

OS and version: Windows 10
Compiler and version (Download-Packages via https://www.exiv2.org/download.html: -- MSVC 2019, exiv2-0.27.5-2019msvc64 -- MinGW, exiv2-0.27.5-MinGW64

Additional context

Camera: Canon EOS 400D Lens: EF28-105mm f/3.5-4.5 USM (Identified with id 156)

Side Notes:

The issue came up when trying to migrate from Lightroom to darktable. Lightroom was able to correctly interpret exif data, so I would assume that the files themselves are corrupted.
The issue has not been seen for other lenses so far (Canon/Tamron with same camera, another Canon lens with another camera)

Console outputs:

// print all
.\exiv2.exe -pa --grep lens/i  .\exif_correct.CR2
Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
Exif.CanonCf.LensAFStopButton                Short       1  2048
Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM

.\exiv2.exe -pa --grep lens/i  .\exif_incorrect.CR2
Exif.CanonCs.LensType                        Short       1  156
Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
Exif.CanonCf.LensAFStopButton                Short       1  2048
Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM

// print plain exif
.\exiv2.exe -pv --grep lens/i  .\exif_correct.CR2
0x0016 CanonCs      LensType                    Short       1  156
0x0017 CanonCs      Lens                        Short       3  105 28 1
0x0009 CanonCf      LensAFStopButton            Short       1  2048
0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM

.\exiv2.exe -pv --grep lens/i  .\exif_incorrect.CR2
0x0016 CanonCs      LensType                    Short       1  156
0x0017 CanonCs      Lens                        Short       3  105 28 1
0x0009 CanonCf      LensAFStopButton            Short       1  2048
0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM

// print interpreted exif
.\exiv2.exe -pt --grep lens/i  .\exif_correct.CR2
Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
Exif.CanonCf.LensAFStopButton                Short       1  2048
Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM

.\exiv2.exe -pt --grep lens/i  .\exif_incorrect.CR2
Exif.CanonCs.LensType                        Short       1  156
Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
Exif.CanonCf.LensAFStopButton                Short       1  2048
Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM

Sample files: CR2.zip

I set up the build environment:

Building the release version reproduces the bug
Building main does not reproduce the bug

Conclusion: Issue seems to have been fixed after the 0.27.5 Release - Bug can be closed

Out of curiosity - if anyone could give me a pointer to the corresponding PR, I would be delighted :)

Thank you for reporting this. And thank you for providing test data.

I'm unable reproduce your report.

Using exiv2 0.25 (built from the current 0.27-maintenance branch)

509 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ exiv2 -vVg exiv2
exiv2 0.27.5
exiv2=0.27.5
package_name=exiv2
executable=/usr/local/bin/exiv2
library=/usr/local/lib/libexiv2.0.27.5.dylib
config_path=/Users/rmills/.exiv2
510 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ cd ~/Downloads/CR2/
511 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pv -g lens/i *
exif_correct.CR2      0x0016 CanonCs      LensType                    Short       1  156
exif_correct.CR2      0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_correct.CR2      0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_correct.CR2      0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    0x0016 CanonCs      LensType                    Short       1  156
exif_incorrect.CR2    0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_incorrect.CR2    0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_incorrect.CR2    0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
512 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pt -g lens/i *
exif_correct.CR2      Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_correct.CR2      Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
exif_correct.CR2      Exif.CanonCf.LensAFStopButton                Short       1  2048
exif_correct.CR2      Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    Exif.CanonCs.LensType                        Short       1  156
exif_incorrect.CR2    Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
exif_incorrect.CR2    Exif.CanonCf.LensAFStopButton                Short       1  2048
exif_incorrect.CR2    Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM
513 rmills@rmillsm1:~/Downloads/CR2 $

Using the 'main' branch:

513 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -vVg exiv2
exiv2 1.0.0.9
exiv2=1.0.0
package_name=exiv2
executable=/usr/local/bin/exiv2
library=/usr/local/lib/libexiv2.1.0.0.9.dylib
config_path=/Users/rmills/.exiv2
514 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pv -g lens/i *
exif_correct.CR2      0x0016 CanonCs      LensType                    Short       1  156
exif_correct.CR2      0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_correct.CR2      0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_correct.CR2      0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    0x0016 CanonCs      LensType                    Short       1  156
exif_incorrect.CR2    0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_incorrect.CR2    0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_incorrect.CR2    0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
515 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pt -g lens/i *
exif_correct.CR2      Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_correct.CR2      Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
exif_correct.CR2      Exif.CanonCf.LensAFStopButton                Short       1  2048
exif_correct.CR2      Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_incorrect.CR2    Exif.CanonCs.Lens                            Short       3  28.0 - 105.0 mm
exif_incorrect.CR2    Exif.CanonCf.LensAFStopButton                Short       1  2048
exif_incorrect.CR2    Exif.Canon.LensModel                         Ascii      64  EF28-105mm f/3.5-4.5 USM
516 rmills@rmillsm1:~/Downloads/CR2 $

Running the same test with the download of Exiv2 v0.27.5 from exiv2.org

510 rmills@rmillsmm-local:~/Downloads $ tar xf exiv2-0.27.5-Darwin.tar
511 rmills@rmillsmm-local:~/Downloads $ cd exiv2-0.27.5-Darwin
512 rmills@rmillsmm-local:~/Downloads/exiv2-0.27.5-Darwin $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -vVg exiv2
exiv2 0.27.5
exiv2=0.27.5
processpath=/Users/rmills/Downloads/exiv2-0.27.5-Darwin/bin
package_name=exiv2
executable=/Users/rmills/Downloads/exiv2-0.27.5-Darwin/bin/exiv2
library=/Users/rmills/Downloads/exiv2-0.27.5-Darwin/lib/libexiv2.0.27.5.dylib
config_path=/Users/rmills/.exiv2
513 rmills@rmillsmm-local:~/Downloads/exiv2-0.27.5-Darwin $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pv -g lens/i ~/Downloads/CR2/*
exif_correct.CR2  0x0016 CanonCs      LensType                    Short       1  156
exif_correct.CR2  0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_correct.CR2  0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_correct.CR2  0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
exif_incorrect.CR2  0x0016 CanonCs      LensType                    Short       1  156
exif_incorrect.CR2  0x0017 CanonCs      Lens                        Short       3  105 28 1
exif_incorrect.CR2  0x0009 CanonCf      LensAFStopButton            Short       1  2048
exif_incorrect.CR2  0x0095 Canon        LensModel                   Ascii      64  EF28-105mm f/3.5-4.5 USM
514 rmills@rmillsmm-local:~/Downloads/exiv2-0.27.5-Darwin $

Please work with me to get "on the same page" so that we agree on the issue being investigated.

I'm going to modify the title of this issue. In general the Lens is not stored in the metadata and Exiv2 has to "guess" the lens by inspecting other metadata. In your case, the lens is stored int the metadata and the tag Exif.Canon.LensModel is a 64 byte ASCII string and correctly identified on all three builds.

It is possible that darktable is using an earlier version of the exiv2 library that does not recognise this lens. The remedy for that is to use the ~/.exiv2 configuration file to identify Lens 156. This is documented in the exiv2 man page exiv2.1.

It's also possible that I'm a little blind this today and failed to understand your report (and my tests). Please forgive and help me to understand.

Hi @clanmills,

thanks for your quick response. I'll try to shed some light on the issue.

In general, the lens can be recognised, i.e. it is part of the database (e.g. the "working image")
However, for some images/files, the resolution/translation with exiv2 -pt breaks in version 0.27.5. The result is, that exiv2 only reports the bare id and not the resolved LensType -- It breaks for the download binaries 0.27.5 (You did not run the -pt option to reproduce the behavior) -- It breaks for a "fresh" build of the 0.27-maintenance branch Actually, you just demonstrated that as well:
```
515 rmills@rmillsm1:~/Downloads/CR2 $ exiv2 -pt -g lens/i *
exif_correct.CR2      Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
...
exif_incorrect.CR2    Exif.CanonCs.LensType                        Short       1  156
...
```

But luckily, this seems to be fixed on the main branch (see my previous comment). I think I should have emphasized that a bit more in the first place as soon as I found out.

So, nothing to do here 👍 (Unless, by chance somebody has an idea what the problem actually was - I'm curious ;))

I'm still having difficult understanding the scope/versions involved in this issue. Have I understood this correctly -pt -g lenstype/i reports tag Exif.CanonCs.LensType as:

file	branch main	branch 0.27-maintance
exif_correct.CR2	`Short 1 Canon EF 28-105mm f/3.5-4.5 USM`	`Short 1 Canon EF 28-105mm f/3.5-4.5 USM`
exif_incorrect.CR2	`Short 1 Canon EF 28-105mm f/3.5-4.5 USM`	`Short 1 156`

921 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pt -g lenstype/i ~/Downloads/CR2/*
exif_correct.CR2  Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_incorrect.CR2  Exif.CanonCs.LensType                        Short       1  156
922 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ 
922 rmills@rmillsm1:~/gnu/github/exiv2/0.27-maintenance/build $ cd ../../main/build
923 rmills@rmillsm1:~/gnu/github/exiv2/main/build $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pt -g lenstype/i ~/Downloads/CR2/*
exif_correct.CR2  Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
exif_incorrect.CR2  Exif.CanonCs.LensType                        Short       1  Canon EF 28-105mm f/3.5-4.5 USM
924 rmills@rmillsm1:~/gnu/github/exiv2/main/build $

I'll have to investigate why -pt is incorrectly translating the file exif_incorrect.CR2 on branch 0.27-maintenance. The behaviour is unlikely to be "sporadic". I expect to discover a significant difference in those files.

Hi @Profoktor,

as @clanmills mentioned, lens recognition is a bit of a guessing game based on the metadata we find in the file.
The good news is that the lens recognition for pictures taken with canon cameras did get quite a bit better on the main branch with #1692.
But these changes didn't and likely won't make it into the 0.27.x release series.

Now that should hopefully answer your question as to what changed :wink:

And as for why you are seeing different results for your different images in this particular case:
it is the different value of CanonCs.MaxAperture for the two images. This value is one of the values that exiv2 uses to guess the correct lens description.
The function that does this guessing work for these images and LensType=156 is: https://github.com/Exiv2/exiv2/blob/1605fe54b9b48192749db04ee696fa19890d507b/src/canonmn_int.cpp#L2862

And lastly since you also mention darktable, I just also want to reiterate, that @clanmills suggestion to use the configuration file is really an easy and straightforward fix for this until a future release of exiv2 can fix this :wink:

Hope the above helps, but feel free to ping me again if you have any other questions!

Ahhh, Merry Christmas @hassec. And Merry Christmas @Profoktor

There's a substantial difference in the code in src/canonmn_int.cpp between 0.27-maintenance and main. The function printCsLensTypeByMetadata() is now being used on 'main' for lens Canon lens "recognition/guess-work".

bdd8a386b src/canonmn_int.cpp (Christoph Hasse    2021-06-08 21:56:04 +0200 2737)

Thank You, Christoph for mentioning MaxAperture. It would have taken a lot more digging for me to discover this. However this is a significant difference between the exif_correct and exif_incorrect image - probably better called exif_different. Not sporadic. Exiv2 is deterministic. There are plenty of other differences:

.../main/build $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pt -g canon/i ~/Downloads/CR2/exif_incorrect.CR2 > /tmp/incorrect.txt
.../main/build $ env DYLD_LIBRARY_PATH=$PWD/lib bin/exiv2 -pt -g canon/i ~/Downloads/CR2/exif_incorrect.CR2 > /tmp/incorrect.txt
.../main/build $ diff /tmp/correct.txt /tmp/incorrect.txt 
21,22c21,22
< Exif.CanonCs.MaxAperture                     Short       1  F4
< Exif.CanonCs.MinAperture                     Short       1  F27
---
> Exif.CanonCs.MaxAperture                     Short       1  F3.5
> Exif.CanonCs.MinAperture                     Short       1  F22
35c35
< Exif.Canon.FocalLength                       Short       4  70.0 mm
---
> Exif.Canon.FocalLength                       Short       4  28.0 mm
38c38
< Exif.CanonSi.MeasuredEV                      Short       1  12.50
---
> Exif.CanonSi.MeasuredEV                      Short       1  12.75
40c40
< Exif.CanonSi.TargetShutterSpeed              Short       1  1/128 s
---
> Exif.CanonSi.TargetShutterSpeed              Short       1  1/161 s
44c44
< Exif.CanonSi.CameraTemperature               Short       1  4 °C
---
> Exif.CanonSi.CameraTemperature               Short       1  6 °C
49c49
< Exif.CanonSi.SubjectDistance                 Short       1  18.58 m
---
> Exif.CanonSi.SubjectDistance                 Short       1  6.82 m
51,52c51,52
< Exif.CanonSi.ShutterSpeedValue               Short       1  1/140 s
< Exif.CanonSi.MeasuredEV2                     Short       1  12.25
---
> Exif.CanonSi.ShutterSpeedValue               Short       1  1/161 s
> Exif.CanonSi.MeasuredEV2                     Short       1  12.62
55c55
< Exif.CanonSi.AutoRotate                      Short       1  0
---
> Exif.CanonSi.AutoRotate                      Short       1  3
82c82
< Exif.CanonFi.FileNumber                      Long        1  100-0105
---
> Exif.CanonFi.FileNumber                      Long        1  100-0106
94c94
< Exif.CanonFi.MacroMagnification              SShort      1  136
---
> Exif.CanonFi.MacroMagnification              SShort      1  149
111c111
< Exif.Canon.MeasuredColor                     Short       5  10 496 1024 1024 552
---
> Exif.Canon.MeasuredColor                     Short       5  10 549 1024 1024 604
113c113
< Exif.Canon.VRDOffset                         Long        1  10210790
---
> Exif.Canon.VRDOffset                         Long        1  9622655
115,116c115,116
< Exif.Canon.ColorData                         Short     796  1 850 1024 ...
< Exif.Canon.CRWParam                          Short     11229  22458 4098 ...
---
> Exif.Canon.ColorData                         Short     796  1 850 1024 1024 ...
> Exif.Canon.CRWParam                          Short     11229  22458 4098 0 0 0 ...

As @hassec has said, it's unlikely that we'll back-port printCsLensTypeByMetadata() to 0.27-maintenance. We released 0.27.5 in October and no case was made to back port this change at that time.

I think your original assertion "Case Closed: fixed on main" is correct.

Hi @clanmills and @hassec,

Merry Christmas to you as well! And a happy new year, while we're at it 😄

@clanmills : yes, you're correct about the term "sporadic" - I just wanted to point out that it affects only a portion of the files. Maybe "scattered" or "isolated" would be more fitting.

@hassec : Thanks for the pointer! Actually, the culprit being the max/min apperture values makes sense, as they change with the focal length and hence might lead to a misinterpretation.

I'll also give the workaround a try (I manipulated the lensfun xml s.th. lens correction is possible, but helping exiv2 to identify the lens appears to be a more robust approach)

@profoktor. I would be helpful if you could ask LensFun/darktable to use Exif.CanonCs.LensModel which is an ascii string added to the metadata by Canon.

I'm not a photographer, so I don't know what LensFun does. However, I know darktable/LensFun (and other applications) want Exiv2 to identify the lens with 100% accuracy. The business of deducing the lens from the metadata is at best a guess. In this case, the data in Exif.CanonCs.LensModel is correct, so why do they use the "guessed" interpreted (-pt) value of Exif.CanonCs.LensType?

Hi @clanmills,

the answer to that is quite simple: Canon only provides this information for their own lenses. As soon as you use a third party lens, these fields are left empty. Only the lens type is provided, which is a short and which exiv2 decodes into something meaningful (I suppose the id is provided by the lens itself, and collides between makes. Adding the fact that there is no indication of the lens maker as well, we end up at the wild guessing we have)

I attached the output for exiv2 -pv for reference: exif_Tamron.txt exif_canon.txt

Ah yes. Thanks for your files.

Applications use the lowest common denominator which is to demand that exiv2 figure out lens recognition for them. I don't know why applications ignore a clear/unambiguous signal such as Exif.CanonCs.LensModel.

I discuss the Lens recognition puzzle in my book: https://clanmills.com/exiv2/book/#4 Exiv2 supports Exif 2.32 which supports lens recognition.

Exiv2 is deterministic. It accurately decodes data in the image. It's unfortunate that the exiv2 option -pt attempts lens recognition. Perhaps 50% of exiv2 issues concern lens recognition although the lens is not stored in the metadata! Even when the lens is stored in the metadata (such as Exif.CanonCs.LensModel), applications ignore the data.

I've enjoyed our discussion about this. I hope you'll agree that we can't change an imperfect world. However we can ask darktable/LensFun to: please respect Exif.CanonCs.LensModel. Right?

Hi @clanmills ,

I never doubted that exiv2 is deterministic. Just bad wording on my side 😉

I don't know why applications ignore a clear/unambiguous signal such as Exif.CanonCs.LensModel. As for the third-party lenses, this field is not filled:
0x0095 Canon        LensModel                   Ascii      64  
Hence you cannot use it to always and ever identify the lens. And if it is set, you would have to correct it so that the entry matches the one in the LensType list:
LensModel EF28-105mm f/3.5-4.5 USM
LensType Canon EF 28-105mm f/3.5-4.5 USM

The field LensType appears to be always populated, at least, ~~and is unambiguous per lens maker~~ (just noticed, it is not... even Canon uses the same id for >10 lenses). For further narrowing it down, the construct using apperture values as indicator for the lens at hand comes into play. Certainly not ideal.

So what would be the alternatives? The exact knowledge of the lens is required for aberration correction and such.

exiv2 just provides plain exif values / the integer value of the lens id

Consequence: All downstream projects need to implement / maintain their own decoding algo + data base. Or they drop the feature as well. Which I would doubt, as it is something the user expects (commercial tools such as Lightroom provide this as well) IMHO, that wouldn't get any better results.

exiv2 maintains the lens decoding

Consequence: Lots and lots of issues for exiv2.. I get it 😄 Unless someone came up (recent rework on main branch?) or comes up with a better idea to identify lenses.

One possibility would be to incorporate the LensModel to detect Canon lenses in a very robust way. But that would probably imply a change of the underlying data structure in struct Exiv2::Internal::TagDetails and Exiv2::Internal::canonCsLensType[480]

So, in my opinion, providing lens recognition in exiv2 is the "least bad" possibility.

Applications unfairly dump lens recognition on Exiv2. If you report an issue with the lens on darktable/digiKam, I believe you will be told "It's a bug in Exiv2". It is not. Exiv2 should never have used -pt to convert lensID (and other undocumented manufacturer makernote magic) into the name of the lens. The bug is to have ever provided that feature. It's responsible for 50% of the activity of Team Exiv2 and consumes resources that would be better used to develop the metadata engine.

Exiv2 supports an "easy-access" API. This supports a "cascade" of keys to be searched. It's documented here: https://github.com/Exiv2/exiv2/wiki/EasyAccess-API It would be possible to provide an "Easy Access" API to return the LensModelName (by cascading through the various places). I would strongly resist this development.

I have given a lot of thought to another way to solve the lens recognition problem. It's discussed in my book: https://clanmills.com/exiv2/book Search for "Future Development Projects".

Exiv2 / exiv2