Closed jcubic closed 3 years ago
darktable official last release is 3.0.2 and many things have been added since 2.6.3, so first thing would be to update your release. 3.0.0 adds some new features about metadata.
And check also what you have on same field before exporting (see image infos). IThat could also be a need to update exiv2 library which is used to detect lens from the lens ID.
Sorry, didn't realized that Fedora package is out of date. Will try to build from source. But last time I was trying to compile darktable on Fedora I was not able to.
This is the output of latest version from git (3.1.0+1407~ga4c447d6c
):
$ exiftool /home/kuba/projects/jcubic/sprzedarz/cyfrowe/darktable_exported/DSC_6614_01.jpg | grep -i lens
Lens Type : E VR
Lens : 35-150mm f/2.8-4
Lens F Stops : 5.00
Iop order list : rawprepare,0,invert,0,temperature,0,highlights,0,cacorrect,0,hotpixels,0,rawdenoise,0,demosaic,0,denoiseprofile,0,bilateral,0,rotatepixels,0,scalepixels,0,lens,0,hazeremoval,0,ashift,0,clipping,0,liquify,0,spots,0,retouch,0,flip,0,exposure,0,mask_manager,0,tonemap,0,toneequal,0,graduatednd,0,profile_gamma,0,equalizer,0,colorin,0,negadoctor,0,nlmeans,0,colorchecker,0,defringe,0,atrous,0,lowpass,0,highpass,0,sharpen,0,lut3d,0,colortransfer,0,colormapping,0,channelmixer,0,basicadj,0,colorbalance,0,rgbcurve,0,rgblevels,0,basecurve,0,filmic,0,filmicrgb,0,colisa,0,tonecurve,0,levels,0,shadhi,0,zonesystem,0,globaltonemap,0,relight,0,bilat,0,colorcorrection,0,colorcontrast,0,velvia,0,vibrance,0,colorzones,0,bloom,0,colorize,0,lowlight,0,monochrome,0,grain,0,soften,0,splittoning,0,vignette,0,colorreconstruct,0,colorout,0,clahe,0,finalscale,0,overexposed,0,rawoverexposed,0,dither,0,borders,0,watermark,0,gamma,0
Lens Spec : 35-150mm f/2.8-4 E VR
Same no lens ID that have details about the lens.
And check also what you have on same field before exporting (see image infos). IThat could also be a need to update exiv2 library which is used to detect lens from the lens ID.
I've showed the output of exiftool the original was NEF file that had that tag exported jpeg don't.
Sorry if this does not go here. If I should create a new issue, please point me to instructions of how I should do it.
Tamron SP 70-200 f2.8 Di VC USD (first version) lens is not recognized with canon cameras, at least not with my canon 80D and under Windows 10 64bits. So automatic lens correction does not work, it is detected just as "255" lens. But the lens is listed in lensfun database.
After some research I have been able to config exiv2 to get it working configuring exiv2.ini under C:\Users\USERNAME\exiv2.ini
This is the entry I created to get it working [canon] 255=Tamron SP 70-200mm f/2.8 Di VC USD A009
I report it in case it can be added to the exiv2 lens database distributed with darktable for other users.
@ariznaf this is completely not relatated, EXIF data and lens information are different thing, this is issue is like compare to missing file extension and you bring issue about not supported lens. Please create another issue.
@ariznaf: it's absolutely not related to this one and nor to darktable. There's no exiv2 distributed with darktable. Exiv2 is a library installed in your OS regardless of darktable (but needed by darktable). So all Exiv2 issues should be on Exiv2 github library. And you will see on the Exiv2 github that you're not the first about Tamron lenses.
And check also what you have on same field before exporting (see image infos). IThat could also be a need to update exiv2 library which is used to detect lens from the lens ID.
And @jcubic: you never answer to my asking (quoted again just above here). I never ask of the Exiftool output but on image infos module on the left of lighttable. Did you have lens id number or the complete normal lens name? A screenshot of this part about the NEF you export would be good.
@ariznaf: it's absolutely not related to this one and nor to darktable. There's no exiv2 distributed with darktable. Exiv2 is a library installed in your OS regardless of darktable (but needed by darktable). So all Exiv2 issues should be on Exiv2 github library. And you will see on the Exiv2 github that you're not the first about Tamron lenses.
Sorry if I misunderstood the title of this topic. I will open a new issue then, just wanted to contribute with the info for this lens. I don't know if exiv2 is distributed or not with darktable, I am just a darktable user (not long ago). But some internal use of exiv2 is doing darktable. I just created the exiv2.ini file in my user config directory and darktable recognized the lens in photos taken with it. I am not user of exiv2. I am on windows, so exiv2 is not installed in my system if darktable does not distribute it.
Sorry if I misunderstood the title of this topic. I will open a new issue then, just wanted to contribute with the info for this lens. I don't know if exiv2 is distributed or not with darktable, I am just a darktable user (not long ago). But some internal use of exiv2 is doing darktable. I just created the exiv2.ini file in my user config directory and darktable recognized the lens in photos taken with it. I am not user of exiv2. I am on windows, so exiv2 is not installed in my system if darktable does not distribute it.
I understand totally that you don't know that, no problem at all but why open a new issue If I inform you that an exiv2 issue and not a darktable one. darktable only shown what exiv2 report.
I have not installed exiv2, just the download from darktable for windows. I don't know about exiv2 nor use it or have installed it in my windows machine. Don't know where to report it to exiv2 users nor have been registered with exiv2 in any site.
I put it here in order to inform that small problem with windows and the solution if other windows user has a similar problem using darktable, until someone in darktable or exiv2 (I don't know) fixes it.
One comment about Original Issue, I have this message when I've run the app from console:
[dev_read_history] the module `lens' requested by image `DSC_5328.NEF' is not installed on this computer!
If this module is required maybe it should be included in cmake. I have no idea how to install it.
and next lot of
[_ioppr_check_rules] found rule flip clipping module clipping (16) is before flip (20) image 9956 (dt_dev_read_history_no_image end)
[_ioppr_check_rules] found rule flip clipping module flip (20) is after clipping (16) image 9956 (dt_dev_read_history_no_image end)
I closed the app after few minutes I think that I've only exported one file first opened galler/roll (that had 3 Nef files).
And check also what you have on same field before exporting (see image infos). IThat could also be a need to update exiv2 library which is used to detect lens from the lens ID.
From what I see there are no lens info at all for my lenses that are not in lensfun. I think it should not remove lens info if it don't recognize the lens. There is lens info on my old photos like nikon 50mm f/1.4.
But it also don't export that Lens ID for lens that IS in left info like my old photos with Nikon 28-300. I think it's a bug in darktable that feild is never in export used and you loose information.
You don't answer to my question for the second time so I will let others help here if they want... I'm quite it's also here an exiv2 issue and not a darktable one but without what I'm asked for, I can't confirm. And I really don't know why you talk about lensfun. Lensfun only apply lens corrections for lenses identified and on lensfun database. So if the lens is not identified by Exiv2 library, nor darktable, nor lensfun will know which lens it is...
@Nilvus I've installed exiv2 from source still no Lens ID on exported jpeg for both known and unknown lens.
How can I check if exiv2 detect the lens? I don't really care if Darktable detect the lens or not all I want is to pass through the tag from input to output. It don't work for any lens it seems.
@Nilvus I've installed exiv2 from source still no Lens ID on exported jpeg for both known and unknown lens.
How can I check if exiv2 detect the lens? I don't really care if Darktable detect the lens or not all I want is to pass through the tag from input to output. It don't work for any lens it seems.
I already precise that... You select your raw file and check what is written on lens in image infos module on left panel in lighttable or darkroom... if you see number id and not the name, that means the problem is your lens is unknown to exiv2, so an exiv2 issue, not a darktable one. If in input, exiv2 don't know lens name, then darktable also. And of course if no name in input, no name possible on output (export here).
I already written that information:
From what I see "there are no lens info at all for my lenses that are not in lensfun". I think it should not remove lens info if it don't recognize the lens. There is lens info on my old photos like nikon 50mm f/1.4.
The Lens ID is in exif data outside of Darktable there is in lighttable on left but not in exported output file.
But if you're asking about exported JPEG files if I open directory of darktable_exported then the lens info is there on JPEG files there is Nikon AF-S Nikkor 28-300 but no Lens ID with same information, I'm not sure from where that information cam from, Maybe some exiv2 ID or serial number, but the output exif don't have "Lens ID" tag, that's the point. Lens ID need to be in output same as on input file.
I think that EXIF should not be changed only software should be updated and maybe some meta data like author if user change it.
I already written that information:
From what I see "there are no lens info at all for my lenses that are not in lensfun". I think it should not remove lens info if it don't recognize the lens. There is lens info on my old photos like nikon 50mm f/1.4.
No it's not the answer. Image infos and so Exif infos (but exiv2) as nothing to do with lensfun and you talk about old photos. Why talking about old photos? Sorry but your infos are really not clear... as you talk on a thing and another...
So please just be simple and start again. On your first post you talk about DSC_6447.jpg file and so DSC_6447.NEF one. So on darktable, select this NEF (just this one, by now we don't care about others) and then take a screenshot of image infos lines on the left module of lighttable in darktable and post it here to see what lens info you have. Nothing else...
Or share the NEF to check if we have same behavior.
Exporting lens model in EXIF data of my jpg is working great on my side but I know that my lens is correctly identified by exiv2 so shown on image infos module...
As you wish if it's helful the screenshot but I've told you that there are no lens information on that panel, if you want to see this here it is:
By old photo I mean photo taken with old lens that show up on left panel:
That old photo have lens that show up in left panel that's why I use this one to show you that no matter if you have lens information or don't the output file don't have "Lens ID" that that is required because you loose lens information that is in orginal Exif data, it may be accessed by exiv2 or whatever darktable is using to get the information but it's on in "Lens ID" that I need to have.
The problem is that on Flickr there are no lens information because there are no "Lens ID" tag in Exif. that is used by other tools not everyone use exiv2 magic to discover what type of lens the image have. Why Darktable remove important information from Exif.
As you wish if it's helful the screenshot but I've told you that there are no lens information on that panel, if you want to see this here it is:
Sorry but as said, when you said that, you talk on same time about old photos and lensfun. So it was hard to know if that sentence was related to which images and the module... lensfun is not the problem here.
That old photo have lens that show up in left panel that's why I use this one to show you that no matter if you have lens information or don't the output file don't have "Lens ID" that that is required because you loose lens information that is in orginal Exif data, it may be accessed by exiv2 or whatever darktable is using to get the information but it's on in "Lens ID" that I need to have.
So now clear, first screenshot show that exiv2 can't identify your lens. So for this one, you have an exiv2 issue. And of course, having an issue could hide another, so if for other photos where lens is there, you should have it of course on jpg.
So next step would be now sharing a NEF (with free rights) with lens shown on module infos in darktable and another without to test. I don't use NEF but I always have Lens ID on my jpg exported from darktable.
Just remember this PR #4357 (but seems to have been merged on 3.0.1) and this related issue #4355. @elstoc and @phweyland: now that this issue is clear, any ideas on this one? On actual master and with my Raw from Panasonic, lens id field is ok on my jpg export. But maybe on NEF?
I can share the NEF file but it's every one of the photos I've used in Darktable, probably your own exported photos uploaded to Flickr website will not have lens data, because what I've tracked there is no "Lens ID" Exif tag in output file, whitch in any case should not be related to exiv2 and details in program. It should just read the property and write that property.
But if that exiv2 library is used to write/read exif data, I have no idea if this is the case, then it's issue with exiv2.
Try unticking 'develop history' in the export options (gear icon next to the export button). If that resolves it, then it's probably related to #4355.
I can share the NEF file but it's every one of the photos I've used in Darktable, probably your own exported photos uploaded to Flickr website will not have lens data, because what I've tracked there is no "Lens ID" Exif tag in output file, whitch in any case should not be related to exiv2 and details in program. It should just read the property and write that property.
As I said, it's the opposite for me: I have this lens ID name on all my jpg exports. So on my side, with my RAW, darktable read the property and write that property. The issue is elsewhere (or has been resolved on master).
But if that exiv2 library is used to write/read exif data, I have no idea if this is the case, then it's issue with exiv2.
Never said that. exiv2 issues are only when they're no LENS ID name on image module infos (why I insist for that). write exif data on export is a darktable thing.
Have made some checks with current master. The field Exif.NikonLd3.LensIDNumber is present in jpg file. The exported jpg has a bit less exif data than the nef file. Two cases, none of which covering your case:
Note. In the case of #4355, no exif data were exported. This is not the case here.
Please share the nef file. That's the best way to understand where is the problem.
@Nilvus is there any old library, that may case this?
If this helps here is my NEF file http://jcubic.pl/DSC_3862.NEF it it works on your system then it's probably something with my setup. Maybe some other package is also old.
I don't remember if I've used Darkatable on other then Fedora system but from what I've had I think that I probably never had "Lens ID" in output files because it's been a while since I had Lens info on Flickr, (that is like 99% caused by missing "Lens ID" to be 100% sure I'll copy that tag and upload a file)
I've tried to write Lens ID tag by hand but it seems it's not possible it's composite tag. See Setting value to LensID tag at exiftool.org/forum
@elstoc unticking 'develop history' don't make a difference.
@jcubic: I've tested your RAW on my actual darktable release (recent master) and I also install darktable 3.0.2. On both I have lens name on image infos in darktable, so in input and on jpg also after export.
3.0.2 export:
master export:
So it seems you have a problem on your side. I don't know why but yes, maybe it's an old library. I have exiv2 0.27, but as you install it from source, you probably have same (at least) version. So something else probably. I don't think it could be related to your darktable personal config but as, sometimes, issues are strange, I suggest you to test with a clean and new darktable (so just move your .config/darktable folder to save it but don't have any darktable folder here). Then launch darktable and just import this NEF to test if export change/work... Maybe a database problem on your side...
If you want to see on your side, jpg from your NEF exported with darktable master:
I've downloaded the jpeg file and it don't have "Lens ID" according to exiftool? There was strage behavior I was able to see lens detail when I opened exported jpeg file in Darktable (my own version of the file and the same the file downloaded from comment above) but the exiftool showed that it don't have Lens ID. Maybe Darktable to some extra stuff and don't show actual lens ID maybe it calculate those tag value but don't save it.
$ exiftool 81495137-641fec80-92ae-11ea-9b71-61ba60bc9f6d.jpg | grep -i lens
Lens Type : G VR
Lens : 28-300mm f/3.5-5.6
Lens F Stops : 5.33
Lens Spec : 28-300mm f/3.5-5.6 G VR
To be sure I've also upload it to flickr and no lens info, probably because of missing lens id.
But in my case in left panel there are no "Lens ID" just "lens". One issue I see on your screen (not sure how to get actual EXIF) but you have "Lens ID Number" and tag should be "Lens ID" maybe it's related.
But gthumb show lens ID so there's another problem. And indeed exiftool doesn't show this line too on my side. So I check with exiftool my recent jpg (and older ones) exported from my Panasonic Raw with darktable and exiftool show Lens ID for my files. So the problem is probably related to some lens. Maybe something similar finally to @ariznaf and #4951. I can't help more here. @phweyland or @TurboGit: if you have any ideas.
Just an idea, gthumb is using exiv2 library maybe LensIDNumber is generated by the library but actual "LensID" tag is missing or maybe some of the child tags (LensID is combo tag) is missing. I can create issue bug/question on exiv2 repo maybe they know or have some clue.
On this is the output of my check:
kuba@jcubic:~/projects/jcubic/tmp$ exiftool -f $(cat tags | sed -e 's/^/-/' | tr '\n' ' ') 81495137-641fec80-92ae-11ea-9b71-61ba60bc9f6d.jpg
Lens ID Number : -
Lens F Stops : 5.33
Min Focal Length : -
Max Focal Length : -
Max Aperture At Min Focal : -
Max Aperture At Max Focal : -
MCU Version : -
Lens Type : G VR
kuba@jcubic:~/projects/jcubic/tmp$ exiftool -f $(cat tags | sed -e 's/^/-/' | tr '\n' ' ') ~/Zdjecia/Toskania/Wrzesień\ 2019/DSC_3862.NEF
Lens ID Number : 165
Lens F Stops : 5.33
Min Focal Length : 28.3 mm
Max Focal Length : 302.0 mm
Max Aperture At Min Focal : 3.6
Max Aperture At Max Focal : 5.7
MCU Version : 167
Lens Type : G VR
tags file is:
LensIDNumber
LensFStops
MinFocalLength
MaxFocalLength
MaxApertureAtMinFocal
MaxApertureAtMaxFocal
MCUVersion
LensType
According to exitool forum "Lens ID" is composite tag that use all those exif tags, and I think that LensIDNumber is fake tag created by exiv2 using lenses it recognize.
If this is not the issue of Darktable but exiv2 please create a ticket there (with a link) you probably explain it better then me.
Just an idea, gthumb is using exiv2 library maybe LensIDNumber is generated by the library but actual "LensID" tag is missing or maybe some of the child tags (LensID is combo tag) is missing. I can create issue bug/question on exiv2 repo maybe they know or have some clue.
I don't check that gthumb is using exiv2, so if it use it, that's not exiv2 related. So don't create issue on exiv2 repo. And as I said, I have lens ID on export for my RAW. So this is related to your RAW and it's not a general problem but a specific one in darktable. So please wait (and avoid posting any other thing to help others check your issue and if that needs a fix on darktable), like what I link on my previous comment.
@jcubic : looking at your screenshot it seems you are building an unsupported branch as you have the 4 new timestamps. This PR is not yet merged into master.
So please, build a version from master or using latest release for your OS. And it would be nice to send screenshot after having selected the English language it would really help us. Thanks.
At this stage I'm confused it started as if the exported JPEP was missing lens info and now it looks like you have issues with lens information altogether, and a missing lens module on your build, that plus the fact your are building using a non validated PR just makes it impossible to understand where the problem could be... There is far more too many moving parameters.
dt doesn't transform these tags. All the work is done by exiv2. Indeed, Exiv2 doesn't report the Lens Id tag, even for your nef file, but the code (165) is available under Lens ID Number:
philippe@philippe-pc:~/Desktop/test$ exiv2 -PEtvlyc DSC_3862.NEF | grep Lens
Lens Type Byte 1 14 D G VR
Lens Rational 4 280/10 3000/10 35/10 56/10 28-300mm F3.5-5.6
Lens FStops Undefined 4 64 1 12 0 5.33333
Lens ID Number Byte 1 165 Nikon AF-S Nikkor 28-300mm f/3.5-5.6G ED VR
Lens F-Stops Byte 1 64 F5.3
philippe@philippe-pc:~/Desktop/test$ exiv2 -PEtvlyc 81.jpg | grep Lens
Lens ID Number Byte 1 165 Nikon AF-S Nikkor 28-300mm f/3.5-5.6G ED VR
Lens F-Stops Byte 1 64 F5.3
Lens Type Byte 1 14 D G VR
Lens Rational 4 280/10 3000/10 35/10 56/10 28-300mm F3.5-5.6
Lens FStops Undefined 4 64 1 12 0 5.33333
The enumeration of available tags for LensID according to exiv2 is:
Maybe something similar finally to @ariznaf and #4951.
Yes, similar, but not the same as here the lens is recognized by dt.
EDIT Third party lenses metadata seem to be challenging ! I just see that the provided NEF file (with Nikon lens) doesn't correspond to the initial post (Tamron lens).
Nevertheless it seems you have the issue with both types of lens.
@phweyland : the lens tags are using Nickon3 name, maybe the issue ?
$ exiv2 -pt ~/Téléchargements/DSC_3862.NEF | grep -i lens
Exif.Nikon3.LensType Byte 1 D G VR
Exif.Nikon3.Lens Rational 4 28-300mm F3.5-5.6
Exif.Nikon3.LensFStops Undefined 4 5.33333
Exif.NikonLd3.LensIDNumber Byte 1 Nikon AF-S Nikkor 28-300mm f/3.5-5.6G ED VR
Exif.NikonLd3.LensFStops Byte 1 F5.3
the lens tags are using Nickon3 name, maybe the issue ?
Maybe, but not sure. In the example given by @jcubic the first problem is that exiftool extracts different information between nef and jpg, while exiv2 remains coherent.
The other issue is that flicker, according to @jcubic, uses something called LensID which is not provided by exiv2.
@phweyland I'm only guessing that flick is using LensID that was the only full tag in exiftool that showed the lens information like flicker use to show. EDIT I think that didn't pull latest changed from master.
@TurboGit I used master branch from this repo, if it have PR that was not merged then maybe you have problems with the repo.
And general original issue was Tamron lens because this is the lens that I use the most, the other file is my older lens Nikkor that I no longer have, Tamron have issue with exiv2 it seems but this is not related at all, the issue is that exiftool don't show all the tags after export that may cause other solftware not recognize the full lens information.
@jcubic : indeed sorry, I'm reviewing the code for adding the timestamps into the collect module but the timestamps are already in the image information. I was confused.
This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.
I was again looking at some photos on Flicker from someone that edited using GIMP, and they lens information is also missing. If this is not exiv2 issue then maybe there is some other library that write exif data. Or is GIMP have same bug?
@jcubic In #6074 offers a solution.
@esq4 isn't lest ID combo, read only tag? Where the value is taken from multiple fields to generate the proper string.
isn't lest ID combo, read only tag? Where the value is taken from multiple fields to generate the proper string.
?
@phweyland I mean lens id
, I'm not sure but I've read somewhere that it's read only field that take other fields to create the output. Maybe it was something else because I've searched again and it's seems that lens id is a number that is detected by the software to generate actual lens string name (on linux done by exiv2)
@phweyland Accoring to exiftool LensID is composite tag created from other tags.
Got it. Thanks. (Exif.Photo.)LensId is not a standard exif field. Instead that's, as you pinpoint, a specific exiftool (calculated) tag. exiv2 doesn't make this calculation and doesn't have this tag (as far as I know). It just reports the maker's specific LensId field when exists, and may not cover all the existing ids.
As dt uses exiv2 to read or write any metadata (in particular the exif ones), I don't see any standard solution to this problem.
If I'm not mistaken you have not this issue with all your lenses, but only with some of them. So the work around should be to simulate in dt (with exported metadata features) the missing information...
I have issue with every lens, but what's weird is that even when I copy all the tags using exiftool from NEF (nikon raw) file to darktable jpeg I don't get proper lens information on Flickr. So I have no idea what's happening.
This is image with copy exif tags: https://www.flickr.com/photos/jcubic/50263494508 This is another photo with same lens but on canon: https://www.flickr.com/photos/focus1963/50184538322
this is the output for exiftool on before and after tag copy:
kuba@jcubic:~/Zdjecia/Samochody/01/darktable_exported$ exiftool DSC_7256.jpg | grep -i lens
Lens Type : E VR
Lens : 35-150mm f/2.8-4
Lens Data Version : 0204
Lens ID Number : 201
Lens F Stops : 5.00
Lens ID : Tamron 35-150mm f/2.8-4 Di VC OSD (A043)
Lens Spec : 35-150mm f/2.8-4 E VR
kuba@jcubic:~/Zdjecia/Samochody/01/darktable_exported$ exiftool DSC_7256.jpg_original | grep -i lens
Lens Type : E VR
Lens : 35-150mm f/2.8-4
Lens F Stops : 5.00
Lens Spec : 35-150mm f/2.8-4 E VR
Maybe this is issue with Flickr and not Darktable, will send another email to Flickr support will see what they would say.
Describe the bug Exported JPG from NEF file is missing lens model information (in Lens Id Exif tag).
To Reproduce Export jpeg files from NEF files (Nikon)
Expected behavior Lens ID should be in JPG file EXIF data
Platform (please complete the following information):
Additional context Because of this there is missing Lens information on Flickr https://www.flickr.com/photos/jcubic/49844418938/in/dateposted-public/ It was reported to Flickr but it seems that lens information is not in EXIF data that why it's missing on photos. I'm not sure if it's specific to Nikon, I have another compact camera with raw files I can check later and comment in this issue when Lens Id is also missing.