search

Exiv2 / exiv2

Image metadata library and tools
http://www.exiv2.org/
Other
890 stars 279 forks source link

Multi-Page TIFF support to extract DJI Drohnes Meta-Sensor-Data #1460

Closed marsie321 closed 3 years ago

marsie321 commented 3 years ago

This post is related to multi-page support e.g. https://dev.exiv2.org/issues/1124 https://github.com/Exiv2/exiv2/issues/15

Hi we also have some multi-page tiffs. And I am looking into exiftool and exiv2 to export exif info for each page of a multi-page tiff and afterwards write / update exif info.

Does v 0.27 or v 0.28 support multi-page, and if yes how can I access a single page.

exiv2 -pav -g 2 DJI_0018.TIFF | grep -i page # is not working

Warning: Directory Image3 has an unexpected next pointer; ignored. Exif.Image.PageNumber Short 2 0 0 Exif.Thumbnail.PageNumber Short 2 1 0 Exif.Image2.PageNumber Short 2 2 0 Exif.Image3.PageNumber Short 2 3 0

Thank you for any help.

Please find attached some Info / Debug for the tiffs.

exiv2 -pR DJI_0018.TIFF > DJI_0018.TIFF.exiv2-pR

see attachment DJI_0018.TIFF.exiv2-pR.txt

exiv2 -pS DJI_0018.TIFF | grep -i PageNumber | wc -l

3535 # this is correct, there are 3535 pages. DJI_0018.TIFF.exiv2-pR.txt

exiftool -v3 -ee -a DJI_0018.TIFF

ExifToolVersion = 12.14 FileName = DJI_0018.TIFF Directory = . FileSize = 2329576074 FileModifyDate = 1611350267 FileAccessDate = 1611350031 FileInodeChangeDate = 1611350267 FilePermissions = 33188 FileType = TIFF FileTypeExtension = TIF MIMEType = image/tiff ExifByteOrder = II

b) GPSTag is not correctly intepreted with v 0.27 (-pR is KO, but -pa is OK). Why and is there bugfix?

exiv2 -pR DJI_0018.TIFF

1893980318 | 0x8825 GPSTag | IFD | 1 | | STRUCTURE OF TIFF FILE (II): DJI_0018.TIFF address | tag | type | count | offset | value 1893979924 | 0x0001 Version | ASCII | 2 | | N 1893979936 | 0x0002 ISOSpeed | RATIONAL | 3 |1893980012 | 48/1 4/1 32667/1000 1893979948 | 0x0003 ColorMode | ASCII | 2 | | E 1893979960 | 0x0004 Quality | RATIONAL | 3 |1893980036 | 11/1 38/1 28797/1000 1893979972 | 0x0005 WhiteBalance | BYTE | 1 | | 1893979984 | 0x0006 Sharpening | RATIONAL | 1 |1893980060 | 620153/1000 1893979996 | 0x0012 GPSMapDatum | ASCII | 7 |1893980068 | WGS-84

exiv2 -pa DJI_0018.TIFF | grep -i lati

Warning: Directory Image3 has an unexpected next pointer; ignored. Exif.GPSInfo.GPSLatitudeRef Ascii 2 North Exif.GPSInfo.GPSLatitude Rational 3 48deg 4' 31.321" Xmp.drone-dji.RelativeAltitude XmpText 9 50.000000

clanmills commented 3 years ago

Exiv2 does not support multi-page tiff and there is no plan to implement it.

b) GPSTag is not correctly intepreted with v 0.27 (-pR is KO, but -pa is OK). Why and is there bugfix?

There are two tiff parsers in Exiv2. TiffVisitor is the primary parser and used by -pa. The printStructure() parser is used by -pR (and options -pS, -pC, -pX) and can reveal information ignored by TiffVisitor.

I have been writing a book Image Metadata and Exiv2 Architecture which contains a program tvisitor.cpp to explain how Exiv2 works. You don't appear to have provided your tiff file, so I can't run tvisitor on your file. It's very likely that tvisitor shows the information of interest. However, tvisitor isn't a replacement for Exiv2. Its purpose is, in 4000 lines of code, to explain how to parse images and metadata. https://clanmills.com/exiv2/book/

I've been trying for years to retire from Exiv2 and set a goal of my 70th birthday which was last week. Sadly, I don't think anybody will volunteer to maintain Exiv2. My final contribution will be to finish the the book and to ship v0.27.4 on 2021-05-22.

marsie321 commented 3 years ago

Happy Birthday! And thank you for all your efforts. I will have at the source code.

Iterating over all pages is possible. So iterating over all pages is working. exiv2 -pS DJI_0018.TIFF | grep -i PageNumber | wc -l 3535

And meta can be read in general exiv2 -pa DJI_0018.TIFF | grep -i lati

So the steps would be a) where and how is "exiv2 -pS" implemented b) stop/repeat for each page c) what is done for "exiv2 -pa" and finally, d) export and import for each page (exiv2 -eX ... exiv2 -iX)

So I think it might be possible exiv2. Maybe you can help to figure out, where are good points / examples in your source code to do this.

Cheers, Martin

heckflosse commented 3 years ago

@clanmills

I've been trying for years to retire from Exiv2 and set a goal of my 70th birthday which was last week. Sadly, I don't think anybody will volunteer to maintain Exiv2. My final contribution will be to finish the the book and to ship v0.27.4 on 2021-05-22.

Happy birthday. A bit late from my side though. I will need some more time to contribute more. I will give my best to convince other open source developers, which use exiv2, to also contribute.

Can you list all the open source programs, which use exiv2? Maybe this way we can start a thread on pixls.us...

clanmills commented 3 years ago

There's a tiffsplit utility in the libtiff sample applications. Have a look at: https://dev.exiv2.org/issues/1124. It's quite possible that a fairly simple script using tiffsplit and exiv2 will deliver what you want without getting into the Exiv2 C++ code.

The -pS (and -pC, -pX and -pR) parser is src/tiffimage.cpp: 

void TiffImage::printStructure(std::ostream& out, Exiv2::PrintStructureOption option,int depth)

A general solution to multi-page parsing in Exiv2 is a considerable undertaking. The Exiv2 metadata names are of the form Exif.Group.TagName and would require modification to handle page numbers. For example: Exif.Group.TagName[page_number] and the user would need to know how many pages are in the file. What is the metadata inheritance model in a multi-page tiff? I don't know and that needs to be researched. Additionally, there are other multi-page formats such as GIF. So, providing multi-page support in Exiv2 is a big project and that's why it hasn't been done in the past.

In my book, I mention this under Future Development Projects: https://clanmills.com/exiv2/book/#future

A long standing project for Exiv2 is a unified metadata container. There is an implementation of this in the SVN repository. Currently we have three containers for Exif, Iptc and Xmp. This is clumsy. We also have a restriction of one image per file. Perhaps both restrictions have a common solution.

So, I encourage you to look at a script that uses tiffsplit before getting into the Exiv2 C++ code. And I encourage you to discuss this on the chatserver and others may be interesting in working with you. https://matrix.to/#/#exiv2-chat:matrix.org

clanmills commented 3 years ago

Chat Server: #exiv2-chat on matrix.org

I'm very pleased to say that there has been a marked increase in the number of contributors over the last six months. If folks want to develop Exiv2, they will receive my enthusiastic support. However, I want out of the day-to-day effort required to maintain and develop Exiv2. I've made my contribution.

clanmills commented 3 years ago

@heckflosse and @marsie321 Thank you for the birthday greetings. Life is good. I've never been happier.

A list of projects that use Exiv2? I don't have that information. For sure, obvious ones are Gimp, darktable, GwenView, Gnome Desktop and digiKam. However I know there are many more. Somebody who knows apt internal could generate a list of entities which have libexiv2 in their dependencies.

I've personally found working on metadata to be as interesting as anything I encountered in my professional life. And I've worked on compilers, debuggers, interpreters, graphics, printing and many other system level components which run in user space (not kernel). It makes sense for the community to work with me while I am alive and well. If they don't, my book is the hitch-hikers guide to metadata.

clanmills commented 3 years ago 
rmills@ubuntu:~$ apt-cache rdepends libexiv2-27
libexiv2-27
Reverse Depends:
  libgexiv2-2
  libmyth
  viewnior
  pinot
  phototonic
  pdf2djvu
  nomacs
  luminance-hdr
  libqgis-analysis3.10.4
  libkf5kexiv2-15.0.0
  libkf5filemetadata-bin
  libextractor3
  krita
  krename
  kphotoalbum
  hugin-tools
  hugin
  hdrmerge
  gwenview
  gthumb
  gpscorrelate-gui
  gpscorrelate
  gimp-lensfun
  gerbera
  geeqie
  ffdiaporama
  exiv2
  digikam-private-libs
  darktable
  libexiv2-dev
  libexiv2-dev
  libexiv2-dev
marsie321 commented 3 years ago

Thank you Robin,

The link was exactly what I was looking for. And I hope I will find some time to have a deeper look into your source code and your book.

I find a solution, I will post it here.

clanmills commented 3 years ago

Can you attach a test file to this issue, please. I might look at this. And somebody else might be want to investigate this. Let's get everybody on the same page.

If you test file isn't too large, I might deal with multi-page tiff files in my book and add your file to the test suite for the code in the book.

marsie321 commented 3 years ago

Sure, I can upload some files. The file can be viewed with ImageJ, Irfanview or XnView. https://imagej.net/Downloads

marsie321 commented 3 years ago

DJI_0005.zip

marsie321 commented 3 years ago

DJI_0017.zip

clanmills commented 3 years ago

Thanks. I'll look at those files. I think I'll add a section to the book about multi-page tiff files.

In the book, I discuss quite a number of matters which Exiv2 does not support. For example: BigTiff, ISOBMFF (CR3, AVIF, HEIC), JPEG>64k, ICC/XMP support in GIF, PNG Textual Information and other matters that don't immediately come to mind.

Discussing something in the book is usually quite easy and a pre-cursor to full-scale development in Exiv2. I remain hopeful that Exiv2 won't die with my retirement and the book has been written to help future maintainers and developers. While the primary focus of the book concerns Exiv2 implementation for maintenance, I hope discussion of other metadata features will inspire future development.

clanmills commented 3 years ago

Something wrong with the upload of DJI_0017.zip. Perhaps you have to wait for the file to upload before pushing "Comment". Can you try again to attach the file to this issue, please.

marsie321 commented 3 years ago

I just uploaded the file again. I hope it will work. Thank you all your efforts in the past years!

clanmills commented 3 years ago

It has successfully arrived this time, thanks very much.

It's been worth looking at this. I've discovered interesting bugs in tvisitor.cpp. Bugs that I didn't have in $ exiv2 -pR. Testing, testing, testing. If it's not tested, it's probably broken. If it's tested, you only know it's working under defined circumstances. You can never know if code is correct under all circumstances.

If you're happy, I'd like to add DJI_0005.TIFF to the files/ directory in the code in the book (and it'll be parsed by the book's test harness). My aim is to update the section in the book that deals with TIFF to discuss multi-page tiff files.

I'd be fun to have an option on tvisitor to create a GPX with the data in your image. You can upload the GPX to a mapping app (Strava, Google Maps) to create a map of the path of your drone. I'm unlikely to spent time on that.

marsie321 commented 3 years ago

You can add the files to book, thank you. And yes the idea with a GPX track is nice, and it should be easy to implement, after the problems with the tvisitor.cpp / multi-page tiffs are fixed.

clanmills commented 3 years ago

Thanks for agreeing to your files being included in the code base for the book. I've already fixed the issues with tvisitor.cpp/multi-page support and updated that code. Tomorrow I'm add a discussion about multi-page tiff file to the book.

I might implement GPX as it would demonstrate deriving class gpxVisitor' fromclass Visitor`. However the book is about metadata and not about metadata applications. It's probably simple to write a bash or python script to create the GPX by post-processing the output from tvisitor.cpp