Z5: soft-key init fails when items are not loaded

[dbobak]

When exporting entire library, groups (zotero collections) are listed multiple times (see below). Sometimes they are listed twice, sometimes much more. I have many hierarchically organised collections (but the names do not duplicate), so maybe that is the source of the problem? Report ID: NEKISMT6

@article{NeugebauerMaresch2008Palaeolithic,
  timestamp = {2017-07-18T15:14:13Z},
  langid = {english},
  title = {Palaeolithic Research in {{Austria}}},
  volume = {19},
  journaltitle = {Wissenschaftliche Mitteilungen Niederösterreichisches Landesmuseum},
  shortjournal = {Wiss. Mitteilungen Niederösterreichisches Landesmus.},
  author = {Neugebauer-Maresch, Christine},
  date = {2008},
  pages = {7--18},
  groups = {Austria,Austria,Austria,Austria,Austria}
}

@incollection{Kuhn2003What,
  timestamp = {2017-07-18T15:14:13Z},
  location = {{Lisboa}},
  title = {In What Sense Is the {{Levantine Initial Upper Paleolithic}} a “Transitional” Industry},
  volume = {33},
  booktitle = {The Chronology of the {{Aurignacian}} and of the Transitional Technocomplexes: Dating, Stratigraphies, Cultural Implications},
  series = {Trabalhos de Arqueologia},
  publisher = {{Instituto Português de Arqueologia}},
  author = {Kuhn, Steven L.},
  editor = {Zilhão, João and d' Errico, Francesco},
  date = {2003},
  pages = {61--69},
  note = {00034},
  options = {useprefix=true},
  groups = {Early Upper Palaeolithic,Aurignacian,Early Upper Palaeolithic,Aurignacian,Early Upper Palaeolithic,Aurignacian,Early Upper Palaeolithic,Aurignacian,Early Upper Palaeolithic,Aurignacian,Aurignacian,Early Upper Palaeolithic,Early Upper Palaeolithic,Early Upper Palaeolithic}
}

@article{LloydAtkinson2004Archaeology,
  timestamp = {2017-07-18T15:14:13Z},
  title = {Archaeology and Geostatistics},
  volume = {31},
  number = {2},
  journaltitle = {Journal of Archaeological Science},
  shortjournal = {J. Archaeol. Sci.},
  author = {Lloyd, C. D. and Atkinson, P. M.},
  date = {2004},
  pages = {151--165},
  note = {00020},
  groups = {podręczniki,podręczniki,podręczniki}
}

@article{Mester2002Excavations,
  timestamp = {2017-07-18T15:14:13Z},
  langid = {english},
  title = {Excavations at {{Szeleta Cave}} before 1999: Methodology and Overview},
  volume = {3},
  journaltitle = {Praehistoria},
  author = {Mester, Zsolt},
  date = {2002},
  pages = {57--78},
  note = {00006},
  groups = {Historia badań,Historia badań,Szeletian,Early Upper Palaeolithic,Szeletian,Early Upper Palaeolithic,Szeletian,Early Upper Palaeolithic,Szeletian,Szeletian,Early Upper Palaeolithic,Early Upper Palaeolithic,Early Upper Palaeolithic,Early Upper Palaeolithic,Early Upper Palaeolithic}
}

[retorquere]

Can you right-click your library and hit "Send BBT error" from there? That will get me a copy of your library including those collections.

[dbobak]

There's no such option like "Send BBT error" on the right click on the library...

[retorquere]

My bad -- try 3952 (which has the changes for #698 folded in)

[dbobak]

Sorry for the delay - I'm in the field now with a bit limited access to the Internet. But I'll try my best ;)

I've installed 3952. And now I have an error when trying to export the library ("An error occurred while trying to export the selected file."). When trying to report the error with the right-click-on-the-library option, the content of the report is empty and the "next" button is grayed out.

But I could send the report from the button in the Preferences dialog, the ID is: U464VTWT

[dbobak]

3956 also does not allow export, report ID from 3956 is: JKL4GFLM

And I've just noticed that now (also with 3952) Zotero does not display popup saying, that BetterBibLaTeX is ready for business. I've waited about 5 minutes for it to show.

[dbobak]

With 3981 the errors are more or less the same: cannot export references and right-click error report does not work. The more, when trying to report error by the button in the preferences, I'm getting following error:

Invalid response from repository: 403, expected 204
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Invalid according to Policy: Policy expired.</Message><RequestId>B1F401024D42B14E</RequestId><HostId>1gMsrGW8ZM5Uu4pMIV36Kk2vqxqlyBRHAljmCsIQECobkY+VV9iqHLyyLd79QhV7jS9g2GQa6W4=</HostId></Error>

Succesfully reported with Zotero Help->Report Errors menu option, the ID is: 1366496184 (I do not know, if this could help you).

[retorquere]

I can't access that kind of debug log, they're sent to Zotero private servers. I'm just putting the final touches on the umpteenth database rework; something should be out today which should work better.

[retorquere]

Oh wait crap, the form expired. I've generated a new one so error reporting should work again, but please wait for the next drop.

[dbobak]

Successfully installed 3992 and the multiple collections error still exists. Right-click report ID is: 7BYRZP5X

[retorquere]

Oh that's just fantastic the way collections are imported and exported has changed. On it.

[dbobak]

With 3993 the error still exists. Report ID: UPTQ2QAE

[retorquere]

Can you submit a new error report using 3995? I don't expect the error to be fixed but it will give me more debugging info.

[dbobak]

Well, with 3995 the error is fixed :) Report ID: F9XBIXNW With 3998, on the other hand, I cannot export, getting an error "An error occurred while trying to export the selected file.". Also cannot create report with right click option. Report ID with send from preferences button is: 5MIS3NG2. It looks like BBT could not finish loading - there's no popup saying "BBT ready for business", the last popup is "found two references without BibTeX key".

[retorquere]

Can you try with 3999?

[dbobak]

Still the same: BBT hangs on assigning soft keys for two references. ReportID: UGMIGWZZ When trying manually re-scan citekeys, it says the scan is already running ("Scan is still running, [object Object], [object Object] items left"). If I manually assign keys to the references that does not have it, BBT starts without problems and also the problem of multiple grups does not appear. So the problem now is somewhere during soft keys assignment.

[retorquere]

Can you install 4000 when it drops in a few minutes, stop Zotero, delete the better-bibtex.sqlite file and better-bibtex dir from your data directory, start again and submit a new debug log?

[retorquere]

4000 has dropped

[dbobak]

Installed, deleted the files and restarted Zotero. Now BBT informs that 2 references are updated, but in fact they are not. And still cannot export the library. ReportID: HCJDJYWW

[dbobak]

Update: I cannot export only the references without keys. If I'm choosing to export some references of which all have citekeys, the export succeeds.

[retorquere]

What the... Zotero errors out because apparently the items that I loaded just three lines before are not loaded?! Ugh.

You can't see dynamic keys right now because I'm only storing them in the shadow database (as was the case in Z4). Storing them in the extra field had too many performance problems.

[retorquere]

(the citekey will become visible again in the way it used to be in Z4 -- in the reference details and in the reference list)

[dbobak]

When looking into better-bibtex.sqlite database, I cannot see the dynamic keys. Here is the content of the better-bibtex table in the database:

name    data
better-bibtex.citekey
better-bibtex   {"filename":"better-bibtex","collections":["citekey"],"databaseVersion":1.5,"engineVersion":1.5,"autosave":false,"autosaveInterval":5000,"autosaveHandle":null,"throttledSaves":true,"options":{"env":"NA","serializationMethod":"normal","destructureDelimiter":"$<\n"},"persistenceAdapter":null,"throttledSavePending":false,"throttledCallbacks":[],"verbose":false,"events":{"init":[null],"loaded":[],"flushChanges":[],"close":[],"changes":[],"warning":[]},"ENV":"NA"}

[dbobak]

Sorry, my wrong. The first row of the table has many data (my sql app didn't display it). But there is no dynamic keys for the references without keys. Only pinned keys.

[retorquere]

That's because Zotero errors out before I can create the key. Can you try 4001?

[dbobak]

Now trying 4001. And:

1. Still no automatic dynamic keys assignment on startup
2. Manual re-scanning of the keys works.

Report ID: ZKVN67W4

[retorquere]

Still Zotero claims the item isn't loaded even though I'm loading it right before I use it. Maddening. 4002 adds more debug logging, same procedure as before please (deleting the better-bibtex files will assure all dynamic keys are unset before start.

[dbobak]

Installed 4002, followed the same procedure. Report ID: QRDG2YLI

[retorquere]

Oh, that's just swell. When I ask Zotero to fetch me an item, it will return it to me, but it may or may not have actually been fetched. This is just fantastic. I've put out a question on the Zotero forums, we'll see what that gets me.

[retorquere]

getAsync gets items but doesn't necessarily load them.

[retorquere]

Can you try 4004?

[dbobak]

4004 is working (soft keys are assigned and export proceeds)! Thanks and congratulations 👍

[retorquere]

Super, thanks for the feedback. Couldn't have done it without you as I didn't see the problem at my end.

[retorquere]

merged into master.

[dbobak]

Great, thank you again for your work. I really do appreciate that.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.