Exiv2::XmpProperties::propertyList() doesn't recognize "lr" prefix

[phweyland]

Describe the bug

const Exiv2::XmpPropertyInfo *pl = Exiv2::XmpProperties::propertyList(prefix); returns a valid pointer for all kinds of prefix (dc, xmp, ...) except for "lr" for which the pointer is NULL;

However it doesn't throw any exception (for "1r" I get an exception :)).

Expected behavior

Get the list of tags related to "lr":

• hierarchicalSubject
• privateRTKInfo

Desktop (please complete the following information):

• OS: Debian 10
• in the context of dt 3.x.x
• libexiv2-14/stable 0.25-4 amd64
• libexiv2-dev/stable 0.25-4 amd64
• libgexiv2-2/stable 0.10.9-1 amd64

[clanmills]

I don't know anything at all about this, however that just makes the puzzle more interesting! Can you say something else about your use case here.

[clanmills]

I think this must be a rather elderly version of exiv2. We generate our MetaData webpages using samples/taglist which calls that function. For sure, LR is documented:

https://www.exiv2.org/tags-xmp-lr.html

1430 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build/bin $ taglist lr
hierarchicalSubject,    Hierarchical Subject,   bag Text,   XmpBag, External,   Adobe Lightroom hierarchical keywords.
privateRTKInfo, Private RTK Info,   Text,   XmpText,    External,   Adobe Lightroom private RTK info.
1431 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build/bin $ 

I seem to recall Alan Pater discussing something concerning the lr namespace about 2014 or so. Exiv2 0.25 was released 2015-06-21.

Is there an urgency about this? Is it possible to use the current exiv2 (v0.27.2).

Here's our release history. https://www.exiv2.org/whatsnew.html

[phweyland]

Is there an urgency about this? Is it possible to use the current exiv2 (v0.27.2).

Thanks for the answer. No urgency ! I'll check if I can upgrade exiv2 library on my system.

I don't know anything at all about this, however that just makes the puzzle more interesting! Can you say something else about your use case here.

In dt, at export time the user can fulfill metadata with dt internal information. Of course the lr one is not the more critical here. Here below the user can add metadata (redefined tag) in the table from the list extracted from exiv2 with the above function:

[clanmills]

I've built both v0.25 and v0.26 from the archive source on exiv2.org. Both build. Both build samples/taglist which knows about the lr name space.

Getting a more up-to-date darktable/exiv2 combo seems like the way forward.

I'm going to close this. However if you wish to discuss this further, I'll reopen the ticket.

[phweyland]

Thank you for your help. I'll update you.

[phweyland]

I've updated my system to exiv2 0.27.3-3 including dev.

libexiv2-14/now 0.25-4+deb10u1 amd64 [installed,local]
libexiv2-27/unstable,now 0.27.3-3 amd64 [installed,automatic]
libexiv2-dev/unstable,now 0.27.3-3 amd64 [installed]
libgexiv2-2/unstable,now 0.12.1-1 amd64 [installed,automatic]

I've still the same issue as stated in the first post.

Is there anything I could do to identify better the issue ?

[clanmills]

I'll reopen the issue and re-investigate over the weekend.

[clanmills]

Good News for me, not so good for you. I've replaced the code in exifprint.cpp with:

#include <exiv2/exiv2.hpp>

#include <iostream>
#include <iomanip>
#include <cassert>

int main(int argc, char* const argv[])
try {
    Exiv2::XmpParser::initialize();
    ::atexit(Exiv2::XmpParser::terminate);
    std::cout << Exiv2::versionString() << std::endl;

    if ( argc != 2 ) {
        printf("usage %s prefix\n",argv[0]);
    } else {
        const char* prefix   = argv[1];
        std::cout << "prefix = " << prefix << std::endl;
        const Exiv2::XmpPropertyInfo *pl = Exiv2::XmpProperties::propertyList(prefix);
        if ( pl ) {
            std::cout << "name_  = " << pl->name_  << std::endl;
            std::cout << "title_ = " << pl->title_ << std::endl;
            std::cout << "desc_  = " << pl->desc_  << std::endl;
        } else {
            printf("*** Exiv2::XmpProperties::propertyList returned NUL");
        }
    }
    return 0;
} catch (Exiv2::Error& e) {
    std::cout << "Caught Exiv2 exception '" << e.what() << "'\n";
    return -1;
}

And when run:

602 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build $ bin/exifprint 
0.27.3
usage bin/exifprint prefix
603 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build $ bin/exifprint lr
0.27.3
prefix = lr
name_  = hierarchicalSubject
title_ = Hierarchical Subject
desc_  = Adobe Lightroom hierarchical keywords.
604 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build $ bin/exifprint robin
0.27.3
prefix = robin
Caught Exiv2 exception 'No namespace info available for XMP prefix `robin''
605 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build $ 

Is darktable linked statically with Exiv2?

Can you speak with the darktable people, please? I'm happy to meet with you and them on Zoom (or Skype, or FaceTime) to resolve this.

[phweyland]

Is darktable linked statically with Exiv2?

No, dynamically.

Have you found something ? I've upgraded to 0.27.3 for another reason (=> user's lens identification, which works). I've just checked this point as well but this is not critical for the current dt usage.

If I can do something to resolve this on dt side, I'm happy to meet with you on zoom. If not, just to wait for 0.27.4 ?

[clanmills]

Waiting for v0.27.4 is pointless as far as Exiv2::XmpProperties::propertyList("lr"); is concerned. It is working as documented and your report that it returns NULL cannot be reproduced. I've checked as far back as v0.25 and believe it has always worked correctly.

We need to approach the puzzle through the eyes of darktable about which I know nothing. Open an issue with darktable and describe the issue in the context of their product. Please say that you have spoken to me and I am willing to help resolve this.

However the issue that you have reported here, namely propertyList("lr") returns NULL cannot be reproduced.

[phweyland]

I'll do. Thanks a lot for your time Robin.

[clanmills]

Right. I'm going to close this and will reopen it if/when new information is available.

[clanmills]

Ah, right. Thanks for your time on Zoom. I now understand what's being discussed and you are developing darktable. I'm reopening this. We'll get to the bottom of this.

[clanmills]

Loads of difficulties this morning. Yet, I'm also making positive progress.

I couldn't get darktable to run on my laptop. It was insisting on updating the database and then quiting. I deleted ~/config/darktable and yet he wouldn't start. Anyway got it to run on my desktop (on which darktable has never been installed). Very pleased to see that you are ship exiv2 v0.27.3 It's wonderful to see my work being deployed. And more importantly, when I find out what's wrong, I can modify/build/change the library without waiting for you to build something!

I can't get the code down from GitHub. I have a frequent problem with large GitHub repositories being unable to git clone. It terminates without getting any code!

847 rmills@rmillsmbp:~/gnu/github/dt $ git config --global http.postBuffer $((8*524288000))
848 rmills@rmillsmbp:~/gnu/github/dt $ git clone https://github.com/darktable-org/darktable.git --depth 1
Cloning into 'darktable'...
remote: Enumerating objects: 1967, done.
remote: Counting objects: 100% (1967/1967), done.
remote: Compressing objects: 100% (1847/1847), done.
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
849 rmills@rmillsmbp:~/gnu/github/dt $ git clone ssh://github.com/darktable-org/darktable.git 
Cloning into 'darktable'...
ssh: connect to host github.com port 22: Operation timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
850 rmills@rmillsmbp:~/gnu/github/dt $ 

So, for the moment, building darktable is a challenge. However, it might be quite unnecessary as I will explain. I used GitHub.com on the browser and have a copy of exif.cc In the darktable UI, I don't know where to find the dialog box "edit metadata exportation". However, I believe you populate it with this code:

static void _get_xmp_tags(const char *prefix, GList **taglist)
{
  const Exiv2::XmpPropertyInfo *pl = Exiv2::XmpProperties::propertyList(prefix);
  if(pl)
  {
    for (int i = 0; pl[i].name_ != 0; ++i)
    {
      char *tag = dt_util_dstrcat(NULL, "Xmp.%s.%s,%s", prefix, pl[i].name_, _get_exiv2_type(pl[i].typeId_));
      *taglist = g_list_prepend(*taglist, tag);
    }
  }
}
void dt_exif_set_exiv2_taglist()
{
  if(exiv2_taglist) return;

  Exiv2::XmpParser::initialize();
  ::atexit(Exiv2::XmpParser::terminate);

  try
  {
    const Exiv2::GroupInfo *groupList = Exiv2::ExifTags::groupList();
...
    _get_xmp_tags("dc", &exiv2_taglist);
...
    _get_xmp_tags("lr", &exiv2_taglist);
...
  }
  catch (Exiv2::AnyError& e)
  {
    std::string s(e.what());
    std::cerr << "[exiv2 taglist] " << s << std::endl;
  }
}

It's very strange that 'lr' returns NULL. However, I'm going to copy your code into samples/exifprint.cpp. You'll remember this code above:

int main(int argc, char* const argv[])
try {
    Exiv2::XmpParser::initialize();
    ::atexit(Exiv2::XmpParser::terminate);
    std::cout << Exiv2::versionString() << std::endl;

    if ( argc != 2 ) {
        printf("usage %s prefix\n",argv[0]);
    } else {
        const char* prefix   = argv[1];
        std::cout << "prefix = " << prefix << std::endl;
        const Exiv2::XmpPropertyInfo *pl = Exiv2::XmpProperties::propertyList(prefix);
        if ( pl ) {
            std::cout << "name_  = " << pl->name_  << std::endl;
            std::cout << "title_ = " << pl->title_ << std::endl;
            std::cout << "desc_  = " << pl->desc_  << std::endl;
        } else {
            printf("*** Exiv2::XmpProperties::propertyList returned NUL");
        }
    }
    return 0;
} catch (Exiv2::Error& e) {
    std::cout << "Caught Exiv2 exception '" << e.what() << "'\n";
    return -1;
}

I'm going to "beaf this up" by adding a couple of functions based on your functions dt_exif_set_exiv2_taglist() and _get_xmp_tags(). So, I'm going to "emulate" your code to populate the dialog box and see if that reproduces your situation.

[clanmills]

More good news for me!

// ***************************************************************** -*- C++ -*-
// exifprint.cpp
// Sample program to print the Exif metadata of an image

#include <exiv2/exiv2.hpp>

#include <iostream>
#include <iomanip>
#include <cassert>

/* ------------------------------------------------- */
/* darktable code to populate taglist                */
/* ------------------------------------------------- */

/*
static GList *exiv2_taglist = NULL;

static void _get_xmp_tags(const char *prefix, GList **taglist)
{
  const Exiv2::XmpPropertyInfo *pl = Exiv2::XmpProperties::propertyList(prefix);
  if(pl)
  {
    for (int i = 0; pl[i].name_ != 0; ++i)
    {
      char *tag = dt_util_dstrcat(NULL, "Xmp.%s.%s,%s", prefix, pl[i].name_, _get_exiv2_type(pl[i].typeId_));
      *taglist = g_list_prepend(*taglist, tag);
    }
  }
}

void dt_exif_set_exiv2_taglist()
{
  if(exiv2_taglist) return;

  Exiv2::XmpParser::initialize();
  ::atexit(Exiv2::XmpParser::terminate);

  try
  {
    const Exiv2::GroupInfo *groupList = Exiv2::ExifTags::groupList();
    if(groupList)
    {
      while(groupList->tagList_)
      {
        const std::string groupName(groupList->groupName_);
        if(groupName.substr(0, 3) != "Sub" &&
            groupName != "Image2" &&
            groupName != "Image3" &&
            groupName != "Thumbnail"
            )
        {
          const Exiv2::TagInfo *tagInfo = groupList->tagList_();
          while(tagInfo->tag_ != 0xFFFF)
          {
            char *tag = dt_util_dstrcat(NULL, "Exif.%s.%s,%s", groupList->groupName_, tagInfo->name_, _get_exiv2_type(tagInfo->typeId_));
            exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
            tagInfo++;
          }
        }
      groupList++;
      }
    }

    const Exiv2::DataSet *iptcEnvelopeList = Exiv2::IptcDataSets::envelopeRecordList();
    while(iptcEnvelopeList->number_ != 0xFFFF)
    {
      char *tag = dt_util_dstrcat(NULL, "Iptc.Envelope.%s,%s", iptcEnvelopeList->name_, _get_exiv2_type(iptcEnvelopeList->type_));
      exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
      iptcEnvelopeList++;
    }

    const Exiv2::DataSet *iptcApplication2List = Exiv2::IptcDataSets::application2RecordList();
    while(iptcApplication2List->number_ != 0xFFFF)
    {
      char *tag = dt_util_dstrcat(NULL, "Iptc.Application2.%s,%s", iptcApplication2List->name_, _get_exiv2_type(iptcApplication2List->type_));
      exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
      iptcApplication2List++;
    }

    _get_xmp_tags("dc", &exiv2_taglist);
    _get_xmp_tags("xmp", &exiv2_taglist);
    _get_xmp_tags("xmpRights", &exiv2_taglist);
    _get_xmp_tags("xmpMM", &exiv2_taglist);
    _get_xmp_tags("xmpBJ", &exiv2_taglist);
    _get_xmp_tags("xmpTPg", &exiv2_taglist);
    _get_xmp_tags("xmpDM", &exiv2_taglist);
    _get_xmp_tags("pdf", &exiv2_taglist);
    _get_xmp_tags("photoshop", &exiv2_taglist);
    _get_xmp_tags("crs", &exiv2_taglist);
    _get_xmp_tags("tiff", &exiv2_taglist);
    _get_xmp_tags("exif", &exiv2_taglist);
    _get_xmp_tags("exifEX", &exiv2_taglist);
    _get_xmp_tags("aux", &exiv2_taglist);
    _get_xmp_tags("iptc", &exiv2_taglist);
    _get_xmp_tags("iptcExt", &exiv2_taglist);
    _get_xmp_tags("plus", &exiv2_taglist);
    _get_xmp_tags("mwg-rs", &exiv2_taglist);
    _get_xmp_tags("mwg-kw", &exiv2_taglist);
    _get_xmp_tags("dwc", &exiv2_taglist);
    _get_xmp_tags("dcterms", &exiv2_taglist);
    _get_xmp_tags("digiKam", &exiv2_taglist);
    _get_xmp_tags("kipi", &exiv2_taglist);
    _get_xmp_tags("GPano", &exiv2_taglist);
    _get_xmp_tags("lr", &exiv2_taglist);
    _get_xmp_tags("MP", &exiv2_taglist);
    _get_xmp_tags("MPRI", &exiv2_taglist);
    _get_xmp_tags("MPReg", &exiv2_taglist);
    _get_xmp_tags("acdsee", &exiv2_taglist);
    _get_xmp_tags("mediapro", &exiv2_taglist);
    _get_xmp_tags("expressionmedia", &exiv2_taglist);
    _get_xmp_tags("MicrosoftPhoto", &exiv2_taglist);
  }
  catch (Exiv2::AnyError& e)
  {
    std::string s(e.what());
    std::cerr << "[exiv2 taglist] " << s << std::endl;
  }
}
*/

/* ------------------------------------------------- */
/* Robin's darktable emulator                        */
/* ------------------------------------------------- */
typedef void GList;

static GList* exiv2_taglist = NULL;

static void _get_xmp_tags(const char *prefix, GList** /*taglist*/)
{
  const Exiv2::XmpPropertyInfo *pl = Exiv2::XmpProperties::propertyList(prefix);
  if(pl)
  {
    for (int i = 0; pl[i].name_ != 0; ++i)
    {
      // char *tag = dt_util_dstrcat(NULL, "Xmp.%s.%s,%s", prefix, pl[i].name_, _get_exiv2_type(pl[i].typeId_));
      // *taglist = g_list_prepend(*taglist, tag);
        std::cout << "Xmp." << prefix <<"."<< pl[i].name_ << " __ " << pl[i].typeId_ << std::endl;
    }
  }
}

void set_exiv2_taglist()
{
    const Exiv2::GroupInfo *groupList = Exiv2::ExifTags::groupList();
    if(groupList)
    {
      while(groupList->tagList_)
      {
          const Exiv2::TagInfo *tagInfo = groupList->tagList_();
          while(tagInfo->tag_ != 0xFFFF)
          {
            //char *tag = dt_util_dstrcat(NULL, "Exif.%s.%s,%s", groupList->groupName_, tagInfo->name_, _get_exiv2_type(tagInfo->typeId_));
            // exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
              std::cout << "Exif." << groupList->groupName_ << "."<<   tagInfo->name_ << " __ " << tagInfo->typeId_ << std::endl;
              tagInfo++;
          }
          groupList++;
      }
    }
    const Exiv2::DataSet *iptcEnvelopeList = Exiv2::IptcDataSets::envelopeRecordList();
    while(iptcEnvelopeList->number_ != 0xFFFF)
    {
      //char *tag = dt_util_dstrcat(NULL, "Iptc.Envelope.%s,%s", iptcEnvelopeList->name_, _get_exiv2_type(iptcEnvelopeList->type_));
      // exiv2_taglist = g_list_prepend(exiv2_taglist, tag);

        std::cout << "Iptc.Envelope." << iptcEnvelopeList->name_ << " __ " << iptcEnvelopeList->type_ << std::endl; iptcEnvelopeList++;
        iptcEnvelopeList++;
    }

    const Exiv2::DataSet *iptcApplication2List = Exiv2::IptcDataSets::application2RecordList();
    while(iptcApplication2List->number_ != 0xFFFF)
    {
      //char *tag = dt_util_dstrcat(NULL, "Iptc.Application2.%s,%s", iptcApplication2List->name_, _get_exiv2_type(iptcApplication2List->type_));
      // exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
      std::cout << "Iptc.Application2." <<   iptcApplication2List->name_ << " __ " << iptcApplication2List->type_ << std::endl;

      iptcApplication2List++;
    }

    _get_xmp_tags("dc", &exiv2_taglist);
    _get_xmp_tags("xmp", &exiv2_taglist);
    _get_xmp_tags("xmpRights", &exiv2_taglist);
    _get_xmp_tags("xmpMM", &exiv2_taglist);
    _get_xmp_tags("xmpBJ", &exiv2_taglist);
    _get_xmp_tags("xmpTPg", &exiv2_taglist);
    _get_xmp_tags("xmpDM", &exiv2_taglist);
    _get_xmp_tags("pdf", &exiv2_taglist);
    _get_xmp_tags("photoshop", &exiv2_taglist);
    _get_xmp_tags("crs", &exiv2_taglist);
    _get_xmp_tags("tiff", &exiv2_taglist);
    _get_xmp_tags("exif", &exiv2_taglist);
    _get_xmp_tags("exifEX", &exiv2_taglist);
    _get_xmp_tags("aux", &exiv2_taglist);
    _get_xmp_tags("iptc", &exiv2_taglist);
    _get_xmp_tags("iptcExt", &exiv2_taglist);
    _get_xmp_tags("plus", &exiv2_taglist);
    _get_xmp_tags("mwg-rs", &exiv2_taglist);
    _get_xmp_tags("mwg-kw", &exiv2_taglist);
    _get_xmp_tags("dwc", &exiv2_taglist);
    _get_xmp_tags("dcterms", &exiv2_taglist);
    _get_xmp_tags("digiKam", &exiv2_taglist);
    _get_xmp_tags("kipi", &exiv2_taglist);
    _get_xmp_tags("GPano", &exiv2_taglist);
    _get_xmp_tags("lr", &exiv2_taglist);
    _get_xmp_tags("MP", &exiv2_taglist);
    _get_xmp_tags("MPRI", &exiv2_taglist);
    _get_xmp_tags("MPReg", &exiv2_taglist);
    _get_xmp_tags("acdsee", &exiv2_taglist);
    _get_xmp_tags("mediapro", &exiv2_taglist);
    _get_xmp_tags("expressionmedia", &exiv2_taglist);
    _get_xmp_tags("MicrosoftPhoto", &exiv2_taglist);
}

int main(int /*argc*/, char* const* /*argv[]*/)
try {
    Exiv2::XmpParser::initialize();
    ::atexit(Exiv2::XmpParser::terminate);
    set_exiv2_taglist();
    //  dt_exif_set_exiv2_taglist();
    return 0;
} catch (Exiv2::Error& e) {
    std::cout << "Caught Exiv2 exception '" << e.what() << "'\n";
    return -1;
}

And when I run it:

900 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build $ bin/exifprint  | grep -e GPSTag -e lr -e CountryCode -e DateSent
Exif.Image.GPSTag __ 4
Exif.Thumbnail.GPSTag __ 4
Exif.Image2.GPSTag __ 4
Exif.Image3.GPSTag __ 4
Exif.SubImage1.GPSTag __ 4
Exif.SubImage2.GPSTag __ 4
Exif.SubImage3.GPSTag __ 4
Exif.SubImage4.GPSTag __ 4
Exif.SubImage5.GPSTag __ 4
Exif.SubImage6.GPSTag __ 4
Exif.SubImage7.GPSTag __ 4
Exif.SubImage8.GPSTag __ 4
Exif.SubImage9.GPSTag __ 4
Exif.SubThumb1.GPSTag __ 4
Exif.PanasonicRaw.GPSTag __ 4
Exif.NikonPreview.GPSTag __ 4
Exif.SamsungPreview.GPSTag __ 4
Iptc.Envelope.DateSent __ 65537
Iptc.Application2.CountryCode __ 65536
Xmp.crs.AlreadyApplied __ 65541
Xmp.iptc.CountryCode __ 65541
Xmp.iptcExt.CountryCode __ 65541
Xmp.lr.hierarchicalSubject __ 65543
Xmp.lr.privateRTKInfo __ 65541
901 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/build $ 

So everything seems fine.

I'm going to modify the code in Exiv2::XmpParser::initialize(); by adding my "emulator" and see what happens when I run darktable.

[clanmills]

More good news for me! I've update the library and I execute darktable from the Terminal on macOS, the "darktable emulator" is working correctly:

537 rmills@rmillsmm-local:/Applications/darktable.app/Contents/MacOS $ ./darktable | grep -e GPSTag -e lr -e CountryCode -e DateSent

(process:10155): GLib-GObject-CRITICAL **: 12:32:21.774: g_object_set: assertion 'G_IS_OBJECT (object)' failed
Exif.Image.GPSTag __ 4
Exif.Thumbnail.GPSTag __ 4
Exif.Image2.GPSTag __ 4
Exif.Image3.GPSTag __ 4
Exif.SubImage1.GPSTag __ 4
Exif.SubImage2.GPSTag __ 4
Exif.SubImage3.GPSTag __ 4
Exif.SubImage4.GPSTag __ 4
Exif.SubImage5.GPSTag __ 4
Exif.SubImage6.GPSTag __ 4
Exif.SubImage7.GPSTag __ 4
Exif.SubImage8.GPSTag __ 4
Exif.SubImage9.GPSTag __ 4
Exif.SubThumb1.GPSTag __ 4
Exif.PanasonicRaw.GPSTag __ 4
Exif.NikonPreview.GPSTag __ 4
Exif.SamsungPreview.GPSTag __ 4
Iptc.Envelope.DateSent __ 65537
Iptc.Application2.CountryCode __ 65536
Xmp.crs.AlreadyApplied __ 65541
Xmp.iptc.CountryCode __ 65541
Xmp.iptcExt.CountryCode __ 65541
Xmp.lr.hierarchicalSubject __ 65543   <----- lr is present
Xmp.lr.privateRTKInfo __ 65541
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152
[dt_pthread_create] info: bumping pthread's stacksize from 524288 to 2097152

(darktable:10155): GLib-GObject-WARNING **: 12:32:22.222: invalid cast from 'GtkMenuBar' to 'GtkWindow'

(darktable:10155): Gtk-CRITICAL **: 12:32:22.222: gtk_window_add_accel_group: assertion 'GTK_IS_WINDOW (window)' failed

So, we now know there is almost nothing wrong here. However once your function dt_exif_set_exiv2_taglist() executes, somehow Exiv2::XmpProperties::propertyList("lr") is return NULL. Why? We don't know yet. Something being clobbered in memory, perhaps.

I know how to find the culprit. I've pasted the patch for the "emulator" below. At the moment, it's run exactly once. We can change it to cause it to run every time you call Exiv2::XmpParser::initialize();. Then pepper your function dt_exif_set_exiv2_taglist() with calls to Exiv2::XmpParser::initialize(); and discover when he goes mad!

Several ways to move forward:

1. I'll have to ask you to undertake that work as I cannot build darktable.
2. I have only tried this on macOS. Could this be a platform related issue?
3. Can you suggest a time for us to meet again on Zoom to discuss how to move forward.

diff --git a/src/xmp.cpp b/src/xmp.cpp
index d426c1ef..bc4d8141 100644
--- a/src/xmp.cpp
+++ b/src/xmp.cpp
@@ -29,6 +29,7 @@
 #include "error.hpp"
 #include "value.hpp"
 #include "properties.hpp"
+#include "tags.hpp"

 // + standard includes
 #include <iostream>
@@ -418,9 +419,216 @@ namespace Exiv2 {
     void* XmpParser::pLockData_ = 0;

 #ifdef EXV_HAVE_XMP_TOOLKIT
+/* ------------------------------------------------- */
+/* darktable code to populate taglist                */
+/* ------------------------------------------------- */
+/*
+static GList *exiv2_taglist = NULL;
+
+static void _get_xmp_tags(const char *prefix, GList **taglist)
+{
+  const Exiv2::XmpPropertyInfo *pl = Exiv2::XmpProperties::propertyList(prefix);
+  if(pl)
+  {
+    for (int i = 0; pl[i].name_ != 0; ++i)
+    {
+      char *tag = dt_util_dstrcat(NULL, "Xmp.%s.%s,%s", prefix, pl[i].name_, _get_exiv2_type(pl[i].typeId_));
+      *taglist = g_list_prepend(*taglist, tag);
+    }
+  }
+}
+
+void dt_exif_set_exiv2_taglist()
+{
+  if(exiv2_taglist) return;
+
+  Exiv2::XmpParser::initialize();
+  ::atexit(Exiv2::XmpParser::terminate);
+
+  try
+  {
+    const Exiv2::GroupInfo *groupList = Exiv2::ExifTags::groupList();
+    if(groupList)
+    {
+      while(groupList->tagList_)
+      {
+        const std::string groupName(groupList->groupName_);
+        if(groupName.substr(0, 3) != "Sub" &&
+            groupName != "Image2" &&
+            groupName != "Image3" &&
+            groupName != "Thumbnail"
+            )
+        {
+          const Exiv2::TagInfo *tagInfo = groupList->tagList_();
+          while(tagInfo->tag_ != 0xFFFF)
+          {
+            char *tag = dt_util_dstrcat(NULL, "Exif.%s.%s,%s", groupList->groupName_, tagInfo->name_, _get_exiv2_type(tagInfo->typeId_));
+            exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
+            tagInfo++;
+          }
+        }
+      groupList++;
+      }
+    }
+
+    const Exiv2::DataSet *iptcEnvelopeList = Exiv2::IptcDataSets::envelopeRecordList();
+    while(iptcEnvelopeList->number_ != 0xFFFF)
+    {
+      char *tag = dt_util_dstrcat(NULL, "Iptc.Envelope.%s,%s", iptcEnvelopeList->name_, _get_exiv2_type(iptcEnvelopeList->type_));
+      exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
+      iptcEnvelopeList++;
+    }
+
+    const Exiv2::DataSet *iptcApplication2List = Exiv2::IptcDataSets::application2RecordList();
+    while(iptcApplication2List->number_ != 0xFFFF)
+    {
+      char *tag = dt_util_dstrcat(NULL, "Iptc.Application2.%s,%s", iptcApplication2List->name_, _get_exiv2_type(iptcApplication2List->type_));
+      exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
+      iptcApplication2List++;
+    }
+
+    _get_xmp_tags("dc", &exiv2_taglist);
+    _get_xmp_tags("xmp", &exiv2_taglist);
+    _get_xmp_tags("xmpRights", &exiv2_taglist);
+    _get_xmp_tags("xmpMM", &exiv2_taglist);
+    _get_xmp_tags("xmpBJ", &exiv2_taglist);
+    _get_xmp_tags("xmpTPg", &exiv2_taglist);
+    _get_xmp_tags("xmpDM", &exiv2_taglist);
+    _get_xmp_tags("pdf", &exiv2_taglist);
+    _get_xmp_tags("photoshop", &exiv2_taglist);
+    _get_xmp_tags("crs", &exiv2_taglist);
+    _get_xmp_tags("tiff", &exiv2_taglist);
+    _get_xmp_tags("exif", &exiv2_taglist);
+    _get_xmp_tags("exifEX", &exiv2_taglist);
+    _get_xmp_tags("aux", &exiv2_taglist);
+    _get_xmp_tags("iptc", &exiv2_taglist);
+    _get_xmp_tags("iptcExt", &exiv2_taglist);
+    _get_xmp_tags("plus", &exiv2_taglist);
+    _get_xmp_tags("mwg-rs", &exiv2_taglist);
+    _get_xmp_tags("mwg-kw", &exiv2_taglist);
+    _get_xmp_tags("dwc", &exiv2_taglist);
+    _get_xmp_tags("dcterms", &exiv2_taglist);
+    _get_xmp_tags("digiKam", &exiv2_taglist);
+    _get_xmp_tags("kipi", &exiv2_taglist);
+    _get_xmp_tags("GPano", &exiv2_taglist);
+    _get_xmp_tags("lr", &exiv2_taglist);
+    _get_xmp_tags("MP", &exiv2_taglist);
+    _get_xmp_tags("MPRI", &exiv2_taglist);
+    _get_xmp_tags("MPReg", &exiv2_taglist);
+    _get_xmp_tags("acdsee", &exiv2_taglist);
+    _get_xmp_tags("mediapro", &exiv2_taglist);
+    _get_xmp_tags("expressionmedia", &exiv2_taglist);
+    _get_xmp_tags("MicrosoftPhoto", &exiv2_taglist);
+  }
+  catch (Exiv2::AnyError& e)
+  {
+    std::string s(e.what());
+    std::cerr << "[exiv2 taglist] " << s << std::endl;
+  }
+}
+*/
+
+/* ------------------------------------------------- */
+/* Robin's darktable emulator                        */
+/* ------------------------------------------------- */
+
+typedef void GList;
+
+static GList* exiv2_taglist = NULL;
+
+static void _get_xmp_tags(const char *prefix, GList** /*taglist*/)
+{
+  const Exiv2::XmpPropertyInfo *pl = Exiv2::XmpProperties::propertyList(prefix);
+  if(pl)
+  {
+    for (int i = 0; pl[i].name_ != 0; ++i)
+    {
+      // char *tag = dt_util_dstrcat(NULL, "Xmp.%s.%s,%s", prefix, pl[i].name_, _get_exiv2_type(pl[i].typeId_));
+      // *taglist = g_list_prepend(*taglist, tag);
+        std::cout << "Xmp." << prefix <<"."<< pl[i].name_ << " __ " << pl[i].typeId_ << std::endl;
+    }
+  }
+}
+
+static void set_exiv2_taglist()
+{
+    const Exiv2::GroupInfo *groupList = Exiv2::ExifTags::groupList();
+    if(groupList)
+    {
+      while(groupList->tagList_)
+      {
+          const Exiv2::TagInfo *tagInfo = groupList->tagList_();
+          while(tagInfo->tag_ != 0xFFFF)
+          {
+            //char *tag = dt_util_dstrcat(NULL, "Exif.%s.%s,%s", groupList->groupName_, tagInfo->name_, _get_exiv2_type(tagInfo->typeId_));
+            // exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
+              std::cout << "Exif." << groupList->groupName_ << "."<<   tagInfo->name_ << " __ " << tagInfo->typeId_ << std::endl;
+              tagInfo++;
+          }
+          groupList++;
+      }
+    }
+    const Exiv2::DataSet *iptcEnvelopeList = Exiv2::IptcDataSets::envelopeRecordList();
+    while(iptcEnvelopeList->number_ != 0xFFFF)
+    {
+      //char *tag = dt_util_dstrcat(NULL, "Iptc.Envelope.%s,%s", iptcEnvelopeList->name_, _get_exiv2_type(iptcEnvelopeList->type_));
+      // exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
+        
+        std::cout << "Iptc.Envelope." << iptcEnvelopeList->name_ << " __ " << iptcEnvelopeList->type_ << std::endl; iptcEnvelopeList++;
+        iptcEnvelopeList++;
+    }
+
+    const Exiv2::DataSet *iptcApplication2List = Exiv2::IptcDataSets::application2RecordList();
+    while(iptcApplication2List->number_ != 0xFFFF)
+    {
+      //char *tag = dt_util_dstrcat(NULL, "Iptc.Application2.%s,%s", iptcApplication2List->name_, _get_exiv2_type(iptcApplication2List->type_));
+      // exiv2_taglist = g_list_prepend(exiv2_taglist, tag);
+      std::cout << "Iptc.Application2." <<   iptcApplication2List->name_ << " __ " << iptcApplication2List->type_ << std::endl;
+
+      iptcApplication2List++;
+    }
+    
+    _get_xmp_tags("dc", &exiv2_taglist);
+    _get_xmp_tags("xmp", &exiv2_taglist);
+    _get_xmp_tags("xmpRights", &exiv2_taglist);
+    _get_xmp_tags("xmpMM", &exiv2_taglist);
+    _get_xmp_tags("xmpBJ", &exiv2_taglist);
+    _get_xmp_tags("xmpTPg", &exiv2_taglist);
+    _get_xmp_tags("xmpDM", &exiv2_taglist);
+    _get_xmp_tags("pdf", &exiv2_taglist);
+    _get_xmp_tags("photoshop", &exiv2_taglist);
+    _get_xmp_tags("crs", &exiv2_taglist);
+    _get_xmp_tags("tiff", &exiv2_taglist);
+    _get_xmp_tags("exif", &exiv2_taglist);
+    _get_xmp_tags("exifEX", &exiv2_taglist);
+    _get_xmp_tags("aux", &exiv2_taglist);
+    _get_xmp_tags("iptc", &exiv2_taglist);
+    _get_xmp_tags("iptcExt", &exiv2_taglist);
+    _get_xmp_tags("plus", &exiv2_taglist);
+    _get_xmp_tags("mwg-rs", &exiv2_taglist);
+    _get_xmp_tags("mwg-kw", &exiv2_taglist);
+    _get_xmp_tags("dwc", &exiv2_taglist);
+    _get_xmp_tags("dcterms", &exiv2_taglist);
+    _get_xmp_tags("digiKam", &exiv2_taglist);
+    _get_xmp_tags("kipi", &exiv2_taglist);
+    _get_xmp_tags("GPano", &exiv2_taglist);
+    _get_xmp_tags("lr", &exiv2_taglist);
+    _get_xmp_tags("MP", &exiv2_taglist);
+    _get_xmp_tags("MPRI", &exiv2_taglist);
+    _get_xmp_tags("MPReg", &exiv2_taglist);
+    _get_xmp_tags("acdsee", &exiv2_taglist);
+    _get_xmp_tags("mediapro", &exiv2_taglist);
+    _get_xmp_tags("expressionmedia", &exiv2_taglist);
+    _get_xmp_tags("MicrosoftPhoto", &exiv2_taglist);
+}
+
+
     bool XmpParser::initialize(XmpParser::XmpLockFct xmpLockFct, void* pLockData)
     {
         if (!initialized_) {
+
+            set_exiv2_taglist();
+            
             xmpLockFct_ = xmpLockFct;
             pLockData_ = pLockData;
             initialized_ = SXMPMeta::Initialize();

[phweyland]

Can you suggest a time for us to meet again on Zoom to discuss how to move forward.

I hardly follow you but we can meet when you want.

[phweyland]

Even now ...

[clanmills]

We will solve this. Zoom told me that you invitation had expired! So, I've scheduled for after your lunch and my dinner.

Topic: Another Exiv2/Darktable Debug Discussion Time: Sep 7, 2020 19:00 London

Join Zoom Meeting https://us02web.zoom.us/j/9129667335?pwd=cjFvWHNnMlpwOExYSE44dHFrdWtyZz09

Meeting ID: 912 966 7335 Passcode: 2ngkdc

[phweyland]

Great !

[phweyland]

Thank you again Robin

[clanmills]

The issue is being caused the following code in darktable:

void dt_exif_init()
{
  // preface the exiv2 messages with "[exiv2] "
  Exiv2::LogMsg::setHandler(&dt_exif_log_handler);

  Exiv2::XmpParser::initialize();
  // this has to stay with the old url (namespace already propagated outside dt)
  Exiv2::XmpProperties::registerNs("http://darktable.sf.net/", "darktable");
  Exiv2::XmpProperties::registerNs("http://ns.adobe.com/lightroom/1.0/", "lr");
  Exiv2::XmpProperties::registerNs("http://cipa.jp/exif/1.0/", "exifEX");
}

The lightroom and exifEX namespaces are already known to Exiv2/XMPsdk.

920 rmills@rmillsmbp:~/gnu/github/dt $ exiv2 --verbose --version | grep -e lightroom -e exifEX
xmlns=exifEX:http://cipa.jp/exif/1.0/
xmlns=lr:http://ns.adobe.com/lightroom/1.0/
921 rmills@rmillsmbp:~/gnu/github/dt $ 

The darktable code is 5 years old, so it's remarkable this issue has only surfaced.

I will keep this issue open while I investigate the following:

1. I don't believe the code in Exiv2::XmpProperties::registerNs() has been modified.
2. Registering the same prefix/URI more than once should be without side effect.

[clanmills]

Investigation and Discoveries:

1. Reregistering an existing prefix/uri namespace causesExiv2::XmpProperties::propertyList(prefix) to return NULL.

2. This behaviour is in the library in v0.25 and later (I haven't tested further back)

3. There are very few changes to xmpsdk and src/xmp.cpp and nothing concerting namespace registration.

4. Prefix 'lr' and 'exifEX' were last modified in 0.25 (released 2015-06-21).

   1093 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/src $ git blame properties.cpp | grep -e '"exifEX"' -e '"lr"' 
   1a5e6b6c6 (asp            2015-04-03 15:58:56 +0000  117)         { "http://ns.adobe.com/lightroom/1.0/",           "lr",             xmpLrInfo,        N_("Adobe Lightroom schema")                    },
   3ec3708f9 (asp            2015-08-05 17:39:12 +0000  124)         { "http://cipa.jp/exif/1.0/",                     "exifEX",         xmpExifEXInfo,    N_("Exif 2.3 metadata for XMP")  },
   1094 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance/src $ 

5. The API XmpProperties::registerNs is weak. It enables the user to say that a namespace prefix/uri exists, however it doesn't enable the user to define the XmpPropertyInfo array associated with this namespace.

For this reason adding the darktable namespace from darktable is also "weak". If darktable wish to support tags such as Xmp.darktable.abc, they should consider providing a patch to Exiv2 which defines xmpDarktableInfo.

src/properties.cpp has:

    extern const XmpNsInfo xmpNsInfo[] = {
        // Schemas   -   NOTE: Schemas which the XMP-SDK doesn't know must be registered in XmpParser::initialize
...
        { "http://ns.adobe.com/lightroom/1.0/",           "lr",             xmpLrInfo,        N_("Adobe Lightroom schema")                    },

    extern const XmpPropertyInfo xmpLrInfo[] = {
        { "hierarchicalSubject",    N_("Hierarchical Subject"),    "bag Text",  xmpBag,      xmpExternal, N_("Adobe Lightroom hierarchical keywords.")   },
        { "privateRTKInfo",         N_("Private RTK Info"),        "Text",      xmpText,     xmpExternal, N_("Adobe Lightroom private RTK info.")        },
        // End of list marker
        { 0, 0, 0, invalidTypeId, xmpInternal, 0 }
    };

Conclusion: The obvious reason why this has not been discovered is because it was not been noticed! It has actually been broken in darkroom for years.

Remedy:

Here is the code involved:

    void XmpProperties::registerNs(const std::string& ns,
                                   const std::string& prefix)
    {
        ScopedWriteLock swl(rwLock_);

        std::string ns2 = ns;
        if (   ns2.substr(ns2.size() - 1, 1) != "/"
            && ns2.substr(ns2.size() - 1, 1) != "#") ns2 += "/";
        // Check if there is already a registered namespace with this prefix
        const XmpNsInfo* xnp = lookupNsRegistryUnsafe(XmpNsInfo::Prefix(prefix));
        if (xnp) {
#ifndef SUPPRESS_WARNINGS
            if (strcmp(xnp->ns_, ns2.c_str()) != 0) {
                EXV_WARNING << "Updating namespace URI for " << prefix << " from "
                            << xnp->ns_ << " to " << ns2 << "\n";
            }
#endif
            unregisterNsUnsafe(xnp->ns_);
        }
        // Allocated memory is freed when the namespace is unregistered.
        // Using malloc/free for better system compatibility in case
        // users don't unregister their namespaces explicitly.
        XmpNsInfo xn;
        char* c = static_cast<char*>(std::malloc(ns2.size() + 1));
        std::strcpy(c, ns2.c_str());
        xn.ns_ = c;
        c = static_cast<char*>(std::malloc(prefix.size() + 1));
        std::strcpy(c, prefix.c_str());
        xn.prefix_ = c;
        xn.xmpPropertyInfo_ = 0;
        xn.desc_ = "";
        nsRegistry_[ns2] = xn;

Reviewing this, I remember why I never wanted to touch that code. It calls functions with dangerous names such as lookupNsRegistryUnsafe. I don't understand that code AND it returns NULL for "lr" which causes the destruction of access to xmpLrInfo. The use of malloc() without free() is frightening. However the code has stood the test of time (until now), and I do not want to touch it.

A new API was added in 0.26 to safely obtain a Exiv2::Dictionary of registered namespaces. I've used that to protect the existing code from the foolishness of destroying an existing registration with the loss of access to xmpLrInfo.

diff --git a/src/properties.cpp b/src/properties.cpp
index c6ebd34d..117d9d1e 100644
--- a/src/properties.cpp
+++ b/src/properties.cpp
@@ -2503,6 +2503,11 @@ namespace Exiv2 {
     void XmpProperties::registerNs(const std::string& ns,
                                    const std::string& prefix)
     {
+        Exiv2::Dictionary                          nsDict;
+        Exiv2::XmpProperties::registeredNamespaces(nsDict);
+        if ( nsDict.find(prefix) != nsDict.end() ) {
+            if ( nsDict[prefix] == ns ) return;
+        }
         ScopedWriteLock swl(rwLock_);

         std::string ns2 = ns;

Next Step:

I will do nothing immediately about this to enable feedback from dt engineering. I intend to submit the patch concerning Exiv2::Dictionary to the 0.27-maintenance branch later in September 2020.

[clanmills]

Fixing this is NOT a good idea. It causes major difficulty for the test suite. I'm going to close this, delete branch fix_1206_registerNs_darktable and close #1284. We have to live with this.

I've discussed XmpDarktableInfo with dt engineering. Their call.