Update Bibtex error from zid change (zotero libraryID?)

[tmchartrand]

I recently encountered a failure when running Update Bibtex, and checked and did not have obviously malformed entries in lyz.sqlite. However, I did some tests and noticed the old IDs in my .bib file (zid field in lyz.sqlite) all began with 0, while new ones began with 1. From browsing zotero.sqlite, my guess is these are created from the zotero items as (libraryID)_(key). My guess is that at some point, probably either when I updated zotero to version 5, or added a shared library, the libraryID of my main library changed from 0 to 1, breaking the Lyz lookups.

This may not be a common version of this error, but I'm not sure why lyz can't just use the key field alone to identify zotero items, since it should be unique even for duplicate items in different libraries.

I plan to test this out a fix in my case by editing lyz.sqlite and my .bib file directly.

(Using Zotero 5.0.30 and export through Better Bib(La)tex, but I'm pretty sure my issue is not BBT specific)

[tmchartrand]

I looked a bit more, and I think that in Zotero 4 libraryID was null for the user library, then changed to 1 for Zotero 5, so this issue likely affects everyone who has updated! Also looks like the zid format is directly from zotero's getLibraryKeyHash, not chosen by Lyz.

[tmchartrand]

Confirmed that this can be fixed by manually replacing all '0' with '1' in bibtex .bib file and in lyz.sqlite. You can edit the database using sqlite-manager addon, run the SQL command UPDATE keys SET zid = replace( zid, '0_', '1_')

[wshanks]

Hmm, there is a nice comment in the LyZ source on that:

// current format is 0_XXXXXXX where 0 is "library id", not sure what that is for

I guess the null got cast to an integer 0 at some point, but now the user library is 1. I tested LyZ with Zotero 5, but it's possible I didn't test migrating a database through from 4 to 5.

We could put in something to mangle the 0 into 1 when the zid field is used. It looks like LyZ has some code that is supposed to handle an item whose cite key changes but it's not robust enough to handle this case.

[tmchartrand]

I brought this up at the zotero forums, and it sounds like the 0/1 replacement should work for the user library and for group libraries on a single machine, but the issue is bigger if LyZ is supposed to support sync across machines of citations from group libraries. See https://forums.zotero.org/discussion/comment/294574 The proper way to deal with it may be to switch away from LibraryKeyHash/LibraryKey completely and use URIs, but sounds like even that may have some issues... (or just use key only, and have a fallback in case of duplicates)

[wshanks]

Hmm, we should probably make a release of LyZ that does the 0/1 replacement as a quick fix for anyone affected by this now. I think that gives the same functionality as before (it sounds like the problems with the key hash were already present in 4.0), though we should change to something more robust if we can.

[wshanks]

I put in something to address this in 149a07e65632f9a556523e07d004ada3925edc78. At first, I just used the SQL command you had suggested, but then I realized that this could lead to duplicate keys if someone had already upgraded to Zotero 5 and tried to cite things, so I did a more careful loop over the items to merge the 0 and 1 entries of the same items.