Zotero/BBT does not import groups from JabRef 5.1

[bnapoletano]

Thank you for this tool! After more than a decade of using JabRef, I am trying to switch to Zotero, both because importing items seems easier, and because most of the people I collaborate with now use Office and not LaTeX. I probably would not even attempt this changeover were it not for BBT.

Background

Having used JabRef for so long, I have a massive library (over 11,000 entries), and depend heavily on JabRef's grouping features to keep it organized. In particular, I use groups to organize edited compilations, with each group containing the entry for the collection and those for each of its chapters. Unfortunately, it seems that JabRef changed the group format again, and BBT is not reading the groups from the latest version (5.1).

Report ID: D1295549728

Full Bib(La)TeX item you are trying to import: testlib.zip

Version check

Zotero version: 5.0.89 BBT version: 5.2.66 JabRef version: 5.1 Error report ID: D1295549728

Explanation of issue

Expected behavior: Library imported into Zotero with group entries carried over from JabRef. Actual behavior: Groups are not imported into Zotero, or, with some syntax changes, the group tree is imported, but without the entries that it is supposed to contain.

I created a small test library to make sure everything works before trying to import the massive library. This is the BibTeX file I attached to this post. Unfortunately, the groups are not carrying over from JabRef. After some trial-and-error editing of the test BibTeX file, I think I figured out what JabRef changed that is causing the problem.

The new format of the groups tree at the bottom of the BibTeX is as follows: @Comment{jabref-meta: grouping: 0 AllEntriesGroup:; 1 StaticGroup:Ecology_and_Power\;0\;1\;0x8a8a8aff\;\;\;; etc... According to the most recent posts I could find on this issue, it has been raised (https://github.com/fiduswriter/biblatex-csl-converter/issues/84), but I am not sure if the fix has been incorporated. Changing the syntax to an older version seems to work partially, in that it imports the group tree. However, the entries are not imported into the groups to which they belong. After making some changes to the test file based on the syntax of a version of my library I saved as a backup three years ago, from JabRef 2 or 3, I figured out what is going on. In the older syntax, the keys to the entries belonging to the group were listed in the group tree at the bottom of the BibTeX file. Thus, it looks something like this: @comment{jabref-meta: groupstree: 0 AllEntriesGroup:; 1 ExplicitGroup:Edited Volumes\;2\;BeltranRuiz2016\;Blaikie1985\;Echeverria1986\;; 2 ExplicitGroup:A--M\;2\;; etc. In the newer versions of JabRef, however, the groups assignments are included as a line in the entry itself, e.g., groups = {Ecology_and_Power},, and the group tree at the bottom of the file no longer lists the entries. I tried adding a couple entries manually to the group tree in the test file (see the attached .bib file), and they did indeed appear in the proper group collection when I imported the file into Zotero. The problem is that this solution (i.e., looking up the BibTeX tag of each item in each group and manually adding it into the group tree) is not viable given the size and complexity of my entire library. Thus, I am really hoping that BBT can handle the new group format. I would even offer to help with the development, but I am really not a developer. Is there a possibility that this issue will or could be addressed?

Thank you again for providing this tool, and for any help you can offer!

Sincerely, Brian

[bnapoletano]

I did submit an error report vía Zotero. I apologize for not including it here.

Here it is: ErrorReport.zip

[retorquere]

Don't worry about it. In this case, @label-gun was wrong.

[bnapoletano]

Okay, thanks! I was worried that I had missed something...

[retorquere]

What does \a{a} mean? I can't get that to compile.

[bnapoletano]

I think that might have been an error on my part. It is supposed to render as would \aa, but I put the brackets in the wrong place.

[bnapoletano]

Yes, I checked, and it was supposed to render as å. I guess I never noticed that it didn't work... As I mentioned, I have been using JabRef for a long time, and so some of the entries are from before versions of JabRef that converted special characters automatically, when I had to figure out the syntax on my own.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5.2.66.7119 ("why does jabref make this field verbatim?!")

Install in Zotero by downloading test build 5.2.66.7119, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

[retorquere]

I don't use the biblatex-cls-converter parser anymore BTW. I have my own parser now at https://github.com/retorquere/bibtex-parser

[bnapoletano]

Having your own parser makes sense. Thank you for the new build! I tried running the import with the new test build, and it ran smoother, but the groups are still not carried over from the new JabRef format. To make testing easier, I created a new BibTeX file with the groups formatted in the JabRef 5.1 format. I attached it here: testlibnewform.zip

On a side note, I noticed that BBT is clever enough to import the "crossref" field as an "extra" field in Zotero. If reading the new group format as collections turns out to be too much of a hassle, perhaps it could at least add the "groups" field as another "extra" field in Zotero?

[retorquere]

In the first bib file, was MR in UPE one of the missing groups? I see that the Clark2012 entry says it's in that group, but the jabref-meta: groupstree doesn't list that group. Does that mean it's at the top level? But Ecology_and_Power is at the top level in jabref-meta: groupstree and that is explicitly named.

It must be possible to read the new groups format, but it's not pretty.

[retorquere]

When you say

the groups are still not carried over

which are "the groups", specifically?

testlibnewform.bib looks exactly like testlib from a structure perspective. in testlibnewform.bib, all entries except one have groups = {Ecology_and_Power},, one entry has groups = {MR in UPE, Periurbanrift, he2, Ecology_and_Power},. The groups info at the bottom looks like

@Comment{jabref-meta: grouping:
0 AllEntriesGroup:;
1 StaticGroup:Ecology_and_Power\;0\;1\;0x8a8a8aff\;\;\;;
}

And that only lists Ecology_and_Power.

Does 0x8a8a8aff correspond to anything in the JabRef UI BTW?

[bnapoletano]

I apologize, I should have clarified better what I was talking about. I used only one group in the test file (Ecology_and_Power) mainly to see if it was working, but some of the entries belong to more than one group (such as MR in UPE, which is a shorthand reference to a paper I was working on, whereas Ecology and Power is an edited volume). That is another peculiarity of JabRef's new group format. If you copy and paste an entry from one library to another, it carries the "groups" field with it in the entry, but it does not generate the group tree. So, in the test library, Ecology_and_Power is the only group, and it is a top level group. If you want to see the full group structure, I am uploading a copy of the full library that I want to import: reflib51forzot.zip

As you can see, it is quite large, which is why I made a smaller library for testing.

I suppose JabRef's new format could potentially make things easier for BBT in the long run, but at present, it will probably be a major hassle. Or maybe not. It seems that the import would involve two steps. First, BBT would need to read the group tree from the end of the BibTeX file in order to generate the collections in Zotero. Then it would need to reference the "groups" field in each entry to see which group it belongs to. I don't know why the developers at JabRef decided to change the format so radically, although I guess I could see the logic of designating group status as an element of each entry rather than trying to store the information in the group tree. Reading the group designations as an "extra" field would be a helpful short-term fix, as I could then at least query the Zotero database for entries containing that group designation. Ultimately, though, I would be extremely grateful if BBT was able to transfer over the new group format.

I am not sure what the first two numbers (the 0 and the 1) after the group designation refer to, but I believe that the 0x8a8a8aff, or part of it anyway, is a color designation (groups can be assigned different colors now).

[retorquere]

I apologize, I should have clarified better what I was talking about. I used only one group in the test file (Ecology_and_Power) mainly to see if it was working, but some of the entries belong to more than one group (such as MR in UPE, which is a shorthand reference to a paper I was working on, whereas Ecology and Power is an edited volume). That is another peculiarity of JabRef's new group format. If you copy and paste an entry from one library to another, it carries the "groups" field with it in the entry, but it does not generate the group tree. So, in the test library, Ecology_and_Power is the only group, and it is a top level group. If you want to see the full group structure, I am uploading a copy of the full library

No that's fine, let's stick to a single case, otherwise the conversation will get muddy.

Yeah, I'm no fan of the new groups format of JabRef, but we'll get it sorted.

I suppose JabRef's new format could potentially make things easier for BBT in the long run,

Absolutely not. It is a royal pain to deal with, and in the new format, you can't have a collection name twice anywhere regardless of the nesting, because JabRef refers to the group simply by its name. The older format was also a pain to parse, but at least it allowed a name to appear in multiple places of the tree.

First, BBT would need to read the group tree from the end of the BibTeX file in order to generate the collections in Zotero. Then it would need to reference the "groups" field in each entry to see which group it belongs to.

BBT does this already. It should work. There was a bug, which was fixed in 7119, so I'm trying to find out what remaining bug you experience, since you said:

ran smoother, but the groups are still not carried over from the new JabRef format.

But when I import with 7119, I'm seeing the one group present in the bib file created and populated in Zotero.

I don't know why the developers at JabRef decided to change the format so radically, although I guess I could see the logic of designating group status as an element of each entry rather than trying to store the information in the group tree.

Except you can now not express

+ topic-1
  + lit-review
+ topic-2
  + lit-review

because if you now say groups = {lit-review}, it's unclear where the article goes.

I am not sure what the first two numbers (the 0 and the 1) after the group designation refer to, but I believe that the 0x8a8a8aff, or part of it anyway, is a color designation (groups can be assigned different colors now).

Huh. Alright. I don't think it should matter.

With 7119, can you describe what is not working for you?

[bnapoletano]

Yes, it occurred to me as a tried to think through the logic of the whole thing that this would probably make things more difficult for BBT, full stop.

After you brought it up, I did some checking, and (as of 2017) it seems that JabRef was altogether incapable of handling different subgroups with the same name. (I also checked once I discovered that the groups format in JabRef had changed, and it seems that it is impossible to revert to the older format.)

The trouble with 7119 occurs when I change the syntax of the testlib.bib to match that of JabRef 5, i.e.: @Comment{jabref-meta: grouping: 0 AllEntriesGroup:; 1 StaticGroup:Ecology_and_Power\;0\;1\;0x8a8a8aff\;\;\;; }

When I try to import this into Zotero, the group tree (albeit a tree with one branch in this case) is not imported at all. Only when I change testlib.bib back to JabRef's old format is the tree imported. My guess is that 7119 is still looking for jabref-meta: groupstree instead of jabref-meta: grouping, and/or looking for group names indicated by ExplicitGroup instead of StaticGroup. In case it helps, I am attaching a copy of the Debug Log: debuglog.zip

For what it's worth, everything but the new JabRef groups in BBT seems to work perfectly...

[retorquere]

Man I am dense -- you have been trying to tell me they have changed the format again. This shouldn't be too hard to incorporate.

[blip-bloop]

:robot: this is your friendly neighborhood build bot announcing test build 5.2.68.7148 ("update tests suite -- fixes #1641")

Install in Zotero by downloading test build 5.2.68.7148, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

[bnapoletano]

That did it! The group tree is now imported, and the entries appear in their respective groups.

I apologize for not explaining the problem clearly from the onset. In retrospect, I should have uploaded the BibTeX file that was not working, instead of the one that I had been tinkering with. In any case, thank you for the help!

Having confirmed that everything works with the test file, I am now importing my full library into Zotero. I expect this to take a while...

Thank you for the help!

[retorquere]

Meh, point is we got there. I'll cut a new release. You'll be upgraded automatically.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.