search

Juris-M / zotero

Juris-M is a variant of the free and friendly Zotero research platform, with support for legal and multilingual materials.
https://juris-m.github.io
Other
77 stars 12 forks source link

Better BibTeX in Juris-M #16

Closed retorquere closed 8 years ago

retorquere commented 8 years ago

I've tried installing the latest BBT in Juris-M, but I'm getting this error when I import some items (using the debug translator built into BBT). I've tried figuring out what the relevant differences are between Zotero 4.0 and Juris-M, but the code mentioned in the trace seems very similar:

(2)(+0000000): Translate: Translation using BetterBibTeX JSON failed:
string => Error: Invalid field 'false' in ItemFields.isBaseField()
stack => _fieldCheck@chrome://zotero/content/xpcom/data/itemFields.js:410:1
isBaseField@chrome://zotero/content/xpcom/data/itemFields.js:216:3
Zotero.Utilities.itemFromCSLJSON@chrome://zotero/content/xpcom/utilities.js:2342:8
Zotero.Translate.ItemSaver.prototype._saveFields@chrome://zotero/content/xpcom/translation/translate_item.js:674:4
Zotero.Translate.ItemSaver.prototype.saveItems@chrome://zotero/content/xpcom/translation/translate_item.js:132:7
Zotero.Translate.Base.prototype._saveItems@chrome://zotero/content/xpcom/translation/translate.js:1425:1
Zotero.Translate.Sandbox.Base._itemDone@chrome://zotero/content/xpcom/translation/translate.js:150:5
Zotero.Translate.SandboxManager.prototype.importObject/attachTo[localKey]<@chrome://zotero/content/xpcom/translation/translate_firefox.js:527:29
@resource://gre/modules/RemoteAddonsParent.jsm:761:66
Zotero.Item.prototype.complete@resource://gre/modules/RemoteAddonsParent.jsm:761:448
doImport@/tmp/webdriver-rb-profilecopy20160409-28874-rkupna/zotero/translators/BetterBibTeX JSON.js:1697:18
Zotero.Translate.Base.prototype._translateTranslatorLoaded@chrome://zotero/content/xpcom/translation/translate.js:1222:4
Zotero.Translate.Base.prototype.translate/<@chrome://zotero/content/xpcom/translation/translate.js:1189:58
Zotero.Translate.Import.prototype._loadTranslatorPrepareIO/initCallback@chrome://zotero/content/xpcom/translation/translate.js:2051:17
Zotero.Translate.IO.Read.prototype.init@chrome://zotero/content/xpcom/translation/translate_firefox.js:938:3
Zotero.Translate.Import.prototype._loadTranslatorPrepareIO@chrome://zotero/content/xpcom/translation/translate.js:2078:3
Zotero.Translate.Import.prototype._loadTranslator/<@chrome://zotero/content/xpcom/translation/translate.js:2034:3
Zotero.Translate.Base.prototype._loadTranslator@chrome://zotero/content/xpcom/translation/translate.js:1574:16
Zotero.Translate.Import.prototype._loadTranslator@chrome://zotero/content/xpcom/translation/translate.js:2033:2
Zotero.Translate.Base.prototype.translate@chrome://zotero/content/xpcom/translation/translate.js:1189:4
Zotero_File_Interface</_importTranslatorsAvailable/<@chrome://zotero/content/fileInterface.js:355:5
retorquere commented 8 years ago

So the stripping isn't lossy even at sync, it's just that Zotero extensions must become Juris-M-savvy.

Either way works for me. As long as I can get the data.

fbennett commented 8 years ago

After wavering back and forth, I've decided to retain the conversion of hack fields on translator input. With it, we can use the same translator for the main legal types (which require a value in the non-Zotero Jurisdiction field), and get working Juris-M records. That's a bigger win than avoiding the narrow edge case I described above, so keeping the behaviour is worth the candle.

If you run into any further glitches, let me know - this round of fixes has been very beneficial, and I appreciate the time you've put into it.

retorquere commented 8 years ago

I can work with that, but beta16 still strips out {:original-date: XXXX} and doesn't leave the value in any field available to the translators, so the import is lossy from the pov if an export translator.

fbennett commented 8 years ago

It was fixed after, but we ran into the Mozilla signing thing. The service is back up, and I've released the changes made during testing; beta18 should be good.

retorquere commented 8 years ago

Access date

A reference with accessDate: "2014-10-02 00:07:46" yields

"accessed": { "raw": "2014-10-02T00:07:46Z" }

rather than what Zotero exports:

"accessed": { "date-parts": [ [ "2014", 10, 2 ] ] }

Is that because it includes the time?

Editor in extra

{:editor: Xxx} in the extra field gives me a two-part name instead of (what I had assumed would be returned) a one-part name.

fbennett commented 8 years ago

Juris-M will currently export all dates as "raw" in CSLJSON (I think), because by default the client is configured to let the processor handle date parsing from the dumb string. That was done a long time ago, to enable proper date ranges. If the plugin option extensions.zotero.hackUseCiteprocJsDateParser is set to false, you should get what Zotero delivers (in theory, at least).

The parse of editor from the hack syntax needs fixing. I'll try to get that done this evening or tomorrow morning.

retorquere commented 8 years ago

Cool. I now just detect whether I'm getting a T00:00:00.* date and parse that out myself -- The CSL-JSON I output is mainly for Pandoc, and it isn't hip with raw dates I think.

The single-part name is currently the only test that doesn't pass. I'm setting compatibility to v4.0.29.7 or later.

fbennett commented 8 years ago

The date parser is in Juris-M also, so could sent dates in the array format. Seems better to play nice with other processors. On Apr 16, 2016 20:37, "retorquere" notifications@github.com wrote:

Cool. I now just detect whether I'm getting a T00:00:00.* date and parse that out myself -- The CSL-JSON I output is mainly for Pandoc, and it isn't hip with raw dates I think.

The single-part name is currently the only test that doesn't pass. I'm setting compatibility to v4.0.29.7 or later.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/Juris-M/zotero/issues/16#issuecomment-210795037

retorquere commented 8 years ago

The current implementation should be robust given either input.

retorquere commented 8 years ago

One more issue found: {:editor: Xxx} imports to an author field rather than an series editor for report references.

fbennett commented 8 years ago

The editor field is not valid on report, so a fallback to author would be normal. Zotero should (theoretically) do the same. seriesEditor maps to collection-editor, and that should reverse-map to the seriesEditor field okay.

retorquere commented 8 years ago

I will need to vary the test cases for Juris-M in any case, as BBT (at least currently) behaves differently depending on whether you have fields in Zotero fields or as customs in the extra field. I might change that at some point, but I'm close to having the tests go all green.

It's not a Zotero fallback, really, it's mine. I'm going to leave that as-is for now, as specifying vars in the extra fields is something the user does and can correct; I agree with you that your fallback is better though.

fbennett commented 8 years ago

Ah, right, of course - Zotero doesn't do this messiness. Should have a fix for two-field/one-field name modes soon.

retorquere commented 8 years ago

If Zotero would do this, the behavior would be unified across exporters/importers, and all would benefit at once.

Or better yet, if we'd finally have extension-specific reference fields, we could do away with this hack completely.

fbennett commented 8 years ago

I think Dan doesn't like the syntax (which to be fair I just kind of pulled out of the air one day). He prefers colon-separated key:val pairs delimited by newlines. One day we'll figure out how to get everyone's data on the same page, so to speak.

fbennett commented 8 years ago

I do think that custom fields of some sort will be coming after we're past the 5.0 rapids. The Zotero test suite is going to make things so much more flexible.

fbennett commented 8 years ago

I've bumped the betas one patch number, in anticipation of an upcoming Zotero release (necessitated by a processor bug that I let sneak in):

https://github.com/Juris-M/zotero/releases/tag/v4.0.29.8m64beta

Beta19 at that address should keep single-field names as single-field names.

retorquere commented 8 years ago

So is the next release going to be 4.0.29.8 then?

Are there serious movements towards the 5.0 release? Perhaps I should finally try to make BBT compatible. Not looking forward to it TBH -- I hate the promises approach, necessary evil as it may or may not be.

fbennett commented 8 years ago

I think it will be 4.0.29.7, actually - I thought there had been a 7 release already, but current is 6. Faulty memory.

I don't know how close the dev beta of 5.0 is. I'll be putting off the Juris-M migration to the summer. But I did Juris-M patches for an earlier version at Christmas-time, and actually found working with Bluebird a pleasant experience. On Apr 17, 2016 00:04, "retorquere" notifications@github.com wrote:

So is the next release going to be 4.0.29.8 then?

Are there serious movements towards the 5.0 release? Perhaps I should finally try to make BBT compatible. Not looking forward to it TBH -- I hate the promises approach, necessary evil as it may or may not be.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/Juris-M/zotero/issues/16#issuecomment-210835446

retorquere commented 8 years ago

OK, updated.

I reckon I'm just reactionary -- I like sync code. No doubt that bluebird is one of the best in its field, but it's a field I had hoped to avoid altogether. It doesn't help that promisified code usually makes liberal use of constructs like

.then(function() {
  ...
})

which are in practice impossible to monkey-patch, and a good chunk of BBT behavior relies on monkey-patching.

retorquere commented 8 years ago

Well now -- one of the failing tests on Juris-M should in fact have failed on Zotero! I found a bug!

retorquere commented 8 years ago

Boom -- all green!

retorquere commented 8 years ago

Do you have an idea on when you are going to release .7? I'd rather not hard-wire my running tests to fetch a beta as I do now, but if I don't, my builds will start failing.

retorquere commented 8 years ago

For some odd reason, Juris-M seems faster than Zotero. Could just be a fluke.

retorquere commented 8 years ago

Hi Frank,

I'm ready to cut a new release also for other issues I've fixed. Can you give me an approximate idea for a release date of .7? If it's very soon, I'll hold off and include Juris-M support, if not, it'll come in a later release.

fbennett commented 8 years ago

Should be all set.

Apparently I did guess the next Z version right. 4.0.29.8 was released yesterday.

A parallel release of Juris-M that incorporates recent changes is up, as 4.0.29.8m65. I'll make up the Standalone releases for it today (w/in the next 24 hours).

Frank On Apr 18, 2016 22:25, "retorquere" notifications@github.com wrote:

Hi Frank,

I'm ready to cut a new release also for other issues I've fixed. Can you give me an approximate idea for a release date of .7? If it's very soon, I'll hold off and include Juris-M support, if not, it'll come in a later release.

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub https://github.com/Juris-M/zotero/issues/16#issuecomment-211377689

retorquere commented 8 years ago

Superb. My build has just started picking it up, once that goes green (and it should), I'm going to merge to master and cut a release.

Super thanks!

retorquere commented 8 years ago

Everything is go and BBT is out. Thanks for your help.

I'm going to hold off on reaching out to the Plus group from Alan Feldman... nothing against him at all, but I just had a rather nasty rebuttal from the MIT librarians about "plugging" updates to Better BibTeX, and they were already referencing BBT. I'm much better at responding to people that already have an interest.

retorquere commented 8 years ago

BTW, the speed diff between Zotero and Juris-M seems pretty structural, at least on the parts my tests are hitting (containers 2 & 3 are the tests for Juris-M).

fbennett commented 8 years ago

Not sure how to interpret the numbers, sorry - Juris-M is significantly slower? There shouldn't be any thrashing in there, but I'm less skilled than the original designers, and I've always assumed that what I've added drags things a bit.

retorquere commented 8 years ago

On the contrary -- Juris-M is faster. Those bars are just how long each test set takes to run; I've split my tests by hand between 0+1 for Zotero, and the same split for Juris-M in 2+3. Only the long bars in the middle matter, the rest is container setup and teardown, nothing to do with Zotero or Juris-M.

retorquere commented 8 years ago

If there is no "main" in the creator, can I assume it to be the "language" field?