zotero.lua is not producing scannable cite documents with pandoc < 2.9.2

[kjambunathan]

filter broken

**pandocc is installed** ``` ~$ pandoc --version pandoc 2.9.1.1 Compiled with pandoc-types 1.20, texmath 0.12.0.2, skylighting 0.8.4 Default user data directory: /home/kjambunathan/.local/share/pandoc or /home/kjambunathan/.pandoc Copyright (C) 2006-2019 John MacFarlane Web: https://pandoc.org This is free software; see the source for copying conditions. There is no warranty, not even for merchantability or fitness for a particular purpose. ``` **pandoc datadir has `pandoc-zotxt.lua` and `zotero.lua` filters** ``` ~$ ls /home/kjambunathan/.local/share/pandoc/filters pandoc-zotxt.lua pandoc-zotxt.lua-0.3.17 zotero.lua ``` **A very simple `md` file** ``` ~$ cat ww.md @wrigstad2017mastery ``` **Conversion to using `pandoc-zotxt` works fine => citekey is present in Zotero** ``` ~$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain Wrigstad and Castegren (2017) Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like Teaching with Achievements.” In SPLASH-E. ``` **Try using [`zotero.lua`](https://retorque.re/zotero-better-bibtex/exporting/zotero.lua) to produce 'scannable-cite` ODT file** (cf. https://retorque.re/zotero-better-bibtex/exporting/pandoc/) ``` ~$ pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=zotero.lua -o ww.odt ww.md ~$ rm *.xml mimetype ``` **`content.xml` is not OK. I see no scannable cite. The citekey is literally carried over with no translations whatsoever.** ``` ~$ unzip ww.odt Archive: ww.odt extracting: mimetype inflating: meta.xml inflating: META-INF/manifest.xml inflating: content.xml inflating: manifest.rdf inflating: styles.xml inflating: settings.xml extracting: Configurations2/accelerator/current.xml inflating: Thumbnails/thumbnail.png ~$ cat content.xml | tail @wrigstad2017mastery ``` ## Desired output In the Zotero UI, a quick copy of the entry using `scannable cite` format produces this ``` { | Wrigstad, & Castegren, 2017 | | |zu:0:TH5CXWRG} ```

[kjambunathan]

KMFGJF27-apse

[kjambunathan]

Additional context

I have checked out your repo, and looked at the pandoc files in there.

In the Makefile, I see invocation for pandoc-zotero-live-citemarkers.lua (and NOT zotero.lua). I am wondering if you are working on improvements to zotero.lua. This paper document doesn't produce a scannable-cite version.

In the transcript of the bug, I have used one of the keys from biblio.bib in your report, and manually imported it in to zotero client.

paper:
    @rm -f *.docx *.odt *.json
    @pandoc -s --lua-filter=pandoc-zotero-live-citemarkers.lua -o paper$(TIMESTAMP).docx main.md
    @pandoc -s --lua-filter=pandoc-zotero-live-citemarkers.lua -o paper$(TIMESTAMP).odt main.md
    @pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=pandoc-zotero-live-citemarkers.lua -o paper$(TIMESTAMP)-scannable-cite.odt main.md

When I tried to programmatically import the biblio.bib to Zotero, I realized that your repo needs python 3. I am little bit hesitant to move to python3.

start:
    ZOTERO_IMPORT=$(PWD)/biblio.bib npm start

[retorquere]

Scannable cite was indeed broken; try the version at https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/gh-1581/site/content/exporting/zotero.lua

In the Makefile, I see invocation for pandoc-zotero-live-citemarkers.lua (and NOT zotero.lua). I am wondering if you are working on improvements to zotero.lua. This paper document doesn't produce a scannable-cite version.

zotero.lua is a bundled version that includes the json encoder/decoder, utils.lua and zotero.lua to get to a standalone filter, so you don't have to install the dependencies. pandoc-zotero-live-citemarkers.lua is the "main" code that loads and calls the other parts.

In the Zotero UI, a quick copy of the entry using scannable cite format produces this

The lua filter won't produce exactly that; the human-readable part ("Wrigstad, & Castegren, 2017") will just be the citekey. Scannable cite doesn't need that part, it's just a user convenience.

When I tried to programmatically import the biblio.bib to Zotero, I realized that your repo needs python 3. I am little bit hesitant to move to python3.

That's not changing. My test framework has a lot of assumptions expecting Python 3.8. You might want to try pyenv or conda if you really want to use the test runner.

You don't really need all that to test the pandoc stuff though. Just create a new profile (using zotero -datadir profile -P), install BBT in that profile, and import the bib file. After that, whenever you start Zotero with that profile (with -datadir profile), everything you need for the pandoc tests is present. The automated import is mainly because I do all my testing in a dedicated testing profile, and that gets erased very often.

If you don't include -datadir profile on each start of Zotero for these tests, Zotero will point to your main datadir, and the import will land in your regular Zotero library. I'm presuming you don't want that. The test runner automates that, but it's a very heavyweight process; it also rebuilds & installs BBT in the testing profile it creates.

[kjambunathan]

still doesn't work

> Scannable cite was indeed broken; try the version at https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/gh-1581/site/content/exporting/zotero.lua Doesn't work. Here is an annotated transcript: ## See what is in `pandoc` filters ``` ~/.local/share/pandoc/filters$ ls pandoc-zotxt.lua pandoc-zotxt.lua-0.3.17 v0.3.17.zip ww.md ``` ## `pandoc-zotxt` works as expected => `zotero` is running and the citekey indeed exists ``` ~/.local/share/pandoc/filters$ cat ww.md @wrigstad2017mastery ``` ``` ~/.local/share/pandoc/filters$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain Wrigstad and Castegren (2017) Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like Teaching with Achievements.” In SPLASH-E. ``` ## Download the new `zotero.lua` ``` ~/.local/share/pandoc/filters$ wget https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/gh-1581/site/content/exporting/zotero.lua --2020-07-23 08:32:48-- https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/gh-1581/site/content/exporting/zotero.lua Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.152.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.152.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 48420 (47K) [text/plain] Saving to: ‘zotero.lua’ zotero.lua 100%[===================>] 47.29K --.-KB/s in 0.1s 2020-07-23 08:32:49 (406 KB/s) - ‘zotero.lua’ saved [48420/48420] ``` ## Use it ``` ~/.local/share/pandoc/filters$ pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=zotero.lua -o ww.odt ww.md ``` ## Unzip `odt` output ``` ~/.local/share/pandoc/filters$ unzip ww.odt Archive: ww.odt extracting: mimetype inflating: meta.xml inflating: META-INF/manifest.xml inflating: content.xml inflating: manifest.rdf inflating: styles.xml inflating: settings.xml extracting: Configurations2/accelerator/current.xml inflating: Thumbnails/thumbnail.png ``` ## Examine `content.xml`. Not OK -- The key comes out verbatim. No translation whatsoever to `scannable cite` format ``` ~/.local/share/pandoc/filters$ cat content.xml | tail @wrigstad2017mastery ```

[retorquere]

Can you download this, place it on the pandoc directory of the clone you have (replacing the existing Makefile), and run make 1581 (assuming you have imported biblio.bib)? The pandoc call should only output the following:

@sec:theory: not found
@sec:achievement-driven: not found
@sec:the-game: not found
@fig:matrix: not found
@sec:implementation: not found

This downloads the bundled lua filter so that we'll be sure that we're using the same filter, and applies it to the etst file. This reliably gets me a scannable cite odt.

[retorquere]

~Wait, let me retool that to use your citekey so you don't have to import biblio.bib~ never mind, you were already using a key from biblio.bib. Yeah, so test that Makefile please.

[kjambunathan]

still doesn't work

> Can you download [this](https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/master/pandoc/Makefile), place it on the `pandoc` directory of the clone you have (replacing the existing `Makefile`), and run `make 1581` (assuming you have imported biblio.bib)? The pandoc call should _only_ output the following: > > ``` > @sec:theory: not found > @sec:achievement-driven: not found > @sec:the-game: not found > @fig:matrix: not found > @sec:implementation: not found > ``` I am getting a different output, and I am NOT getting a scannable cite odt. **Reset hard to `origin/master`** ``` ~/src/zotero-better-bibtex/pandoc$ git log --format=oneline | head 6cba3f799647710c018ce112cb0538b0d3a4158d Makefile for #1581 abbade7b6b6d89d2a187fcacbff29045a364be0e Merge branch 'gh-1581' 8cf2a21be39e0c94078f1e9b83c5d9a60c2f4ee7 site update 5be8d55edf98b188185d1410ca1cbc60040c05cf ncu de9953283e7436c12503d3d4d8e49e4479f64de3 ncu c1111b0d5adbed41744d58368a9aafee8cb32eb0 Merge pull request #1580 from theworldisnotflat/patch-3 762e25e2713fbefca5fc9a4d4bce3d7847461ae1 main.md #norelease fb5b19cc71b6153346ec1cb185a3784aea8f113f scannable cite was broken #norelease ea9a88edbd54d9c0e555f1aa94c98e25830fbcf6 Update auto.md 93eccea835a73349432eaa3f7b95ef51ba71288a another day, another juris-m ``` **`make 1581`** ``` ~/src/zotero-better-bibtex/pandoc$ make 1581 curl -L https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/master/site/content/exporting/zotero.lua -o zotero-bundled.lua % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 48420 100 48420 0 0 8005 0 0:00:06 0:00:06 --:--:-- 9969 pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=./zotero-bundled.lua -o paper2020-07-23T12:30:46-scannable-cite.odt main.md unzip paper2020-07-23T12:30:46-scannable-cite.odt content.xml Archive: paper2020-07-23T12:30:46-scannable-cite.odt inflating: content.xml grep @wrigstad2017mastery content.xml The course is built around the achievement-driven learning methodology [@wrigstad2017mastery] (more details in [@sec:theory]). This methodology gives students more control of their learning with enough flexibility to engage them in deeper learning [@biggsTang2011]. Yet, year after year, although the course has increased in the number of students, we observed that students do not pursue higher grades. In this course, a higher grade implies the possibility of deeper learning, as students need to justify the design choices of the software-under-construction and they cannot just memorise solutions (Section explains more details). To make these learning outcomes more achievable, comprehensible and accessible, Tobias Wrigstad and Elias Castegren [@wrigstad2017mastery] developed a new assessment technique, called , based on a more fine-grained collection of achievements, which taken together encompass the course’s learning outcomes, but are more achievable when taken individually. The list of achievements makes more explicit what a student needs to master in order to pass the course, but also to achieve higher grades. In a sense, achievement-driven learning refines constructive alignment to the micro level and provides a means for linking learning outcomes with assessment activities—it aims to hit a sweet spot in the constructive alignment design space. Advanced Software Design is a masters-level course taught in the Department of Information Technology at Uppsala University. The course covers topics such as object-oriented analysis and design, domain modelling, software architecture, class and object modelling, behavioural modelling, design patterns, GRASP principles, design evaluation, and design improvement/refactoring. The course does not involve any programming, and instead uses UML, text, and oral presentation as the means for recording and communicating designs. Students taking the course are expected to have a solid background in programming, in particular in object-oriented programming. The course offers 8–10 interactive lectures describing the material and a course-long project. Students form teams on their own (of 4 members and they are encouraged to diversify the team’s skill set) and work together on the achievements. Teams get a fixed teacher assistant (TA) for the duration of the course. Each team has a weekly, thirty-minute-long meeting and feedback sessions with a TA. During the meetings team members are evaluated individually, i.e. some team members may pass an achievement while others will have to try again in the next meeting. The assessment scheme for the project is based on achievement-driven learning [@wrigstad2017mastery], as described in Section [@sec:achievement-driven]. There is a total of 21 achievements: 11 achievements of level 3, 9 achievements of level 4 and a single individual achievement of level 5. There is no exam. The achievement-driven learning [@wrigstad2017mastery] uses gamification to force students to solve a puzzle in order to optimise their time. This work extends achievement-driven learning, introducing explicit gaming elements to foster competition, engagement and the possibility of deeper learning. We have added gaming elements to the course, based on the achievement-driven learning methodology [@wrigstad2017mastery], to motivate students to get higher grades. The gaming elements provide enjoyment and motivational factors. We have built a regression model where enjoyment and motivation explain 55% of the variation in grades, whereby motivation drives the overall effect of the model, meaning that students who were motivated by the game also got higher grades. The link between enjoyment and grades, however, was insignificant. A future research direction could be to further explore the motivational factors that drive higher grades. Are students motivated intrinsically or extrinsically by opt-in gamification elements? What is it that drives motivation when it comes to gamification? If enjoyment, which was insignificant in our model, is indeed an intrinsic motivational factor, as suggested by [@RYAN200054], our results suggest that intrinsic motivational factors may be less likely to influence students’ grades after all. Our results provide a hint that extrinsic motivational factors in form of achievements may be more important in students’ motivation. However, future research is needed to corroborate those suggestions. ``` **Create a simple `ww.md` and ensure that citekey `wrigstad2017mastery`exists** ``` ~/src/zotero-better-bibtex/pandoc$ cat ww.md cat: ww.md: No such file or directory ~/src/zotero-better-bibtex/pandoc$ echo "[@wrigstad2017mastery]" > ww.md ~/src/zotero-better-bibtex/pandoc$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain (Wrigstad and Castegren 2017) Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like Teaching with Achievements.” In SPLASH-E. ``` **`git status`** ``` ~/src/zotero-better-bibtex/pandoc$ git status On branch master Your branch is up to date with 'origin/master'. Untracked files: (use "git add ..." to include in what will be committed) content.xml ww.md zotero-bundled.lua nothing added to commit but untracked files present (use "git add" to track) ```

[kjambunathan]

No luck even after installing 5.2.45.6838.xpi.

This is what I did.

1. installed https://github.com/retorquere/zotero-better-bibtex/releases/download/builds/zotero-better-bibtex-5.2.45.6838.xpi
2. Re-started zotero UI
3. Waited until the initialization is over
4. Re-ran make 1581.

[retorquere]

I have updated the Makefile to only build your minimal ww.md. I still reliably get scannable cite. What version of pandoc are you using?

I appreciate you are trying to include as much relevant info as possible, but your posts are very overwhelming to read for me. Could you use bold instead of ## headings, and maybe a little more selective output? And no all-caps please. I'm an oldschool internet habitant from the usenet days, and allcaps sounds like shouting in my mind, I can't help it.

No need to install 6838, sorry about that post, that was automated. Nothing has changed in BBT, so the normal release will suffice.

[kjambunathan]

I skipped the lua script, and hit the URL directly

I see

TypeError: citationKeys is undefined[1]

The above error is with /zotero-better-bibtex-5.2.45.6838.xpi

~/src/zotero-better-bibtex/pandoc$ curl -s http://127.0.0.1:23119/better-bibtex/export/item?pandocFilterData=true&translator=jzon&citationKeys=wrigstad2017mastery

[1] 16130
[2] 16131
TypeError: citationKeys is undefined[1]-  Done                    curl -s http://127.0.0.1:23119/better-bibtex/export/item?pandocFilterData=true
[2]+  Done                    translator=jzon

~/src/zotero-better-bibtex/pandoc$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain

(Wrigstad and Castegren 2017)

Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like
Teaching with Achievements.” In SPLASH-E.

[kjambunathan]

Quick upate. Scannable cite does NOT work with test build 5.2.45.6841 as well.

[retorquere]

Sorry about those test builds. You can ignore them; nothing has changed in BBT for this. I'm just checking in changes to the Makefile.

[retorquere]

TypeError: citationKeys is undefined[1]

You have to quote the URL. The & in the URL means "execute in background" to the shell when not quoted.

[kjambunathan]

~/src/zotero-better-bibtex$ dpkg -l | grep pandoc
ii  libghc-pandoc-citeproc-data           0.16.4.1-1                         all          Pandoc support for Citation Style Language - data files
ii  pandoc                                2.9.1.1-2                          amd64        general markup converter
ii  pandoc-citeproc                       0.16.4.1-1+b1                      amd64        Pandoc support for Citation Style Language - tools
ii  pandoc-data                           2.9.1.1-2                          all          general markup converter - data files

[kjambunathan]

TypeError: citationKeys is undefined[1]

You have to quote the URL. The & in the URL means "execute in background" to the shell when not quoted.

This is the working version.

curl -s 'http://127.0.0.1:23119/better-bibtex/export/item?pandocFilterData=true&translator=jzon&citationKeys=wrigstad2017mastery'

[kjambunathan]

make 1581 with ww.md still doesn't produce scannable cite documents.

[retorquere]

That's the pandoc version I have. I don't know what to tell you. make 1581 fetches the same filter for you and me, and it gets me a scannable cite on both Mac and Ubuntu.

[retorquere]

Can you try with this as the filter? It will log its workings to stdout.

[kjambunathan]

Can you try with this as the filter? It will log its workings to stdout.

~/src/zotero-better-bibtex/pandoc$ make 1581
curl -L https://gist.githubusercontent.com/retorquere/7c41cea2bd44544571fd5f28ff74eefc/raw/01418802313c481545219b93354cea3aef2e25c4/zotero.lua -o zotero-bundled.lua
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48584  100 48584    0     0   7595      0  0:00:06  0:00:06 --:--:-- 11545
echo "[@wrigstad2017mastery]" > ww.md
pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=./zotero-bundled.lua -o paper2020-07-23T14:27:11-scannable-cite.odt ww.md
{"client":"zotero","author_in_text":false,"scannable_cite":true,"format":"scannable-cite"}
http://127.0.0.1:23119/better-bibtex/export/item?pandocFilterData=true&translator=jzon&citationKeys=
unzip paper2020-07-23T14:27:11-scannable-cite.odt content.xml
Archive:  paper2020-07-23T14:27:11-scannable-cite.odt
  inflating: content.xml             
grep @wrigstad2017mastery content.xml
<text:p text:style-name="Text_20_body">[@wrigstad2017mastery]</text:p>

1581:
    @rm -f *.docx *.odt *.json content.xml
    curl -L https://gist.githubusercontent.com/retorquere/7c41cea2bd44544571fd5f28ff74eefc/raw/01418802313c481545219b93354cea3aef2e25c4/zotero.lua -o zotero-bundled.lua
    echo "[@wrigstad2017mastery]" > ww.md
    pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=./zotero-bundled.lua -o paper$(TIMESTAMP)-scannable-cite.odt ww.md
    unzip paper$(TIMESTAMP)-scannable-cite.odt content.xml
    grep @wrigstad2017mastery content.xml

[kjambunathan]

Can you try with this as the filter? It will log its workings to stdout.

I have a hunch on where the problem could be. May be the Scannable Cite.js is in a location different from what the lua script expects.

Remember cayw url works. But zotero.lua doesn't work. May if you could track down the changes in these code paths, then the fix will be obvious.

zotero-odf-scan-v2.0.43.xpi doesn't install the Scannable Cite.js in to the translators dir · Issue #41 · Juris-M/zotero-odf-scan-plugin

[retorquere]

The lua script doesn't expect Scannable Cite.js to be anywhere -- the lua script makes the scannable cite marker by itself.

[kjambunathan]

Surely, cayw is doing the right thing for me.

[retorquere]

That's because that actually does use the output of Scannable Cite.js. The two don't have anything to do with each other.

[kjambunathan]

That's because that actually does use the output of Scannable Cite.js. The two don't have anything to do with each other.

That explains why I don't see scannable cite in BibTex's translator list. (See screenshot). Are BibTex and Zotero's translators compatible .... May be I can teach Better BibTex about 'Scannable Cite` as a translator ...

[retorquere]

Can you try again with https://gist.githubusercontent.com/retorquere/dbfcc3e51f2aaf82333bddb38cf5e4de/raw/zotero.lua ?

That explains why I don't see scannable cite in BibTex's translator list. (See screenshot). Are BibTex and Zotero's translators compatible .... May be I can teach Better BibTex about 'Scannable Cite` as a translator ...

You don't need it there. If you want drag and drop scannable cites, just set Scannable Cite to be the default format for Default Format under Export in the Zotero preferences. The menu you screencapped above is just to change the behavior of the Better BibTeX Citation Key Quick Copy export translator. They're not translators themselves.

Can we please not do many questions at once? The pandoc scannable cite is entirely separate from the bits you're looking at now.

[kjambunathan]

Is @ part of a citekey. zotero.lua thinks @ is not part of citekey. Does pandoc thnk otherwise.

Is there a way to examine pandoc's AST on my side?

~/src/zotero-better-bibtex/pandoc$ curl -s 'http://127.0.0.1:23119/better-bibtex/export/item?pandocFilterData=true&translator=jzon&citationKeys=wrigstad2017mastery'
{"items":{"wrigstad2017mastery":{"attachments":[],"citationKey":"wrigstad2017mastery","citekey":"wrigstad2017mastery","creators":[{"creatorType":"author","firstName":"Tobias","lastName":"Wrigstad"},{"creatorType":"author","firstName":"Elias","lastName":"Castegren"}],"date":"2017","dateAdded":"2020-07-22T06:12:32Z","dateModified":"2020-07-22T06:12:32Z","extra":"Citation Key: wrigstad2017mastery","itemID":110,"itemType":"conferencePaper","key":"TH5CXWRG","libraryID":1,"notes":[],"publicationTitle":"SPLASH-E","relations":[],"select":"zotero://select/library/items/TH5CXWRG","tags":[],"title":"Mastery learning-like teaching with achievements","uri":"http://zotero.org/users/local/debm4usg/items/TH5CXWRG","version":0}},"zotero":{"wrigstad2017mastery":{"itemID":110,"uri":"http://zotero.org/users/local/debm4usg/items/TH5CXWRG"}},"errors":{}}

[retorquere]

The filter at that gist should output:

Inlines_replace_cites: format=scannable-cite
Inlines_replace_cites: format=scannable-cite
Inlines_replace_cites: format=scannable-cite
Inlines_replace_cites: format=scannable-cite
Inlines_replace_cites: citekeys={"wrigstad2017mastery":true}
scannable-cite: {  | [@wrigstad2017mastery] |  |  | zu:0:PQUXDB37 }

[retorquere]

The @ is not part of the citekey. You can sort of get the AST by doing pandoc ww.md -o ww.json but the field names are not always the same (the json has citationId for example where the AST has id).

The output of the filter from the gist will tell me more. Maybe it's just not fetching the citekeys properly on your system, that would explain the behavior.

[kjambunathan]

It is not really seeing any keys at all. Here is the output.

~/src/zotero-better-bibtex/pandoc$ make 1581
curl -L https://gist.githubusercontent.com/retorquere/dbfcc3e51f2aaf82333bddb38cf5e4de/raw/zotero.lua -o zotero-bundled.lua
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48739  100 48739    0     0  32086      0  0:00:01  0:00:01 --:--:-- 32086
echo "[@wrigstad2017mastery]" > ww.md
pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=./zotero-bundled.lua -o paper2020-07-23T15:23:57-scannable-cite.odt ww.md
unzip paper2020-07-23T15:23:57-scannable-cite.odt content.xml
Archive:  paper2020-07-23T15:23:57-scannable-cite.odt
  inflating: content.xml             
grep @wrigstad2017mastery content.xml
<text:p text:style-name="Text_20_body">[@wrigstad2017mastery]</text:p>
~/src/zotero-better-bibtex/pandoc$ 

[retorquere]

It's not running the filter at all it seems. The Inlines_replace_cites should be present even if it's just a simple text file without citekeys.

[retorquere]

Are you getting it from the right gist? I'm using https://gist.github.com/retorquere/dbfcc3e51f2aaf82333bddb38cf5e4de now

[kjambunathan]

Are you getting it from the right gist? I'm using https://gist.github.com/retorquere/dbfcc3e51f2aaf82333bddb38cf5e4de now

See the make 1581 output

[retorquere]

That's the point. If that doesn't give Inlines_replace_cites output, the filter isn't running at all. Can you check whether the line

print('Inlines_replace_cites: format=' .. config.format)

is present in your filter?

[kjambunathan]

That's the point. If that doesn't give Inlines_replace_cites output, the filter isn't running at all. Can you check whether the line

print('Inlines_replace_cites: format=' .. config.format)

is present in your filter?

Of course. (I am a developer)

[retorquere]

You can skip the unzip and the grep for now. If those lines don't appear with the filter at the gist, the filter is not being ran.

[retorquere]

Can you grab the filter at https://gist.github.com/fe514aa0344c65d28445d8dceb899de1 and see what this outputs:

pandoc -s --lua-filter=./mwe.lua -o ww.docx ww.md

[kjambunathan]

My guess is that pandoc tries to load the module. But module load fails, and pandoc is skipping ahead silently.

Does LUA_PATH look good to you?

~/src/zotero-better-bibtex/pandoc$ luarocks path
export LUA_PATH='/home/kjambunathan/.luarocks/share/lua/5.1/?.lua;/home/kjambunathan/.luarocks/share/lua/5.1/?/init.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;./?.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua'
export LUA_CPATH='/home/kjambunathan/.luarocks/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/?.so;./?.so;/usr/lib/x86_64-linux-gnu/lua/5.1/?.so;/usr/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so'
~/src/zotero-better-bibtex/pandoc$ echo $LUA_PATH
/home/kjambunathan/.luarocks/share/lua/5.1/?.lua;/home/kjambunathan/.luarocks/share/lua/5.1/?/init.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;./?.lua;/usr/local/lib/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?/init.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua

[retorquere]

I have no idea, but the filter doesn't use anything from the path. Everything it needs is bundled into the filter.

[kjambunathan]

https://gist.github.com/fe514aa0344c65d28445d8dceb899de1

~/src/zotero-better-bibtex/pandoc$ cat ww.md
[@wrigstad2017mastery]
~/src/zotero-better-bibtex/pandoc$ cat test.lua
function Inlines_holler(inlines)
  print('inlines')
  return inlines
end

return {
  { Inlines = Inlines_holler },
}
~/src/zotero-better-bibtex/pandoc$ pandoc -s --lua-filter=./test.lua -o ww.odt ww.md

[retorquere]

If that doesn't output anything, pandoc is not running lua filters at all, full stop. There's nothing I can do about that -- for me it does run this filter, and outputs inlines four times. You'll have to get help from the pandoc people to see why this filter is not running.

[kjambunathan]

If that doesn't output anything, pandoc is not running lua filters at all, full stop.

It does run lua filters. My guess is that it is not hitting the inlines callback.

~/src/zotero-better-bibtex/pandoc$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain

(Wrigstad and Castegren 2017)

Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like
Teaching with Achievements.” In SPLASH-E.

[retorquere]

https://pandoc.org/lua-filters.html:

There are two special function names, which can be used to define filters on lists of blocks or lists of inlines.

Inlines (inlines) If present in a filter, this function will be called on all lists of inline elements, like the content of a Para (paragraph) block, or the description of an Image. The inlines argument passed to the function will be a List of Inlines for each call.

[kjambunathan]

Does the AST look good to you. I don't see anything tagged as inlines.

~/src/zotero-better-bibtex/pandoc$ cat ww.md
[@wrigstad2017mastery]
(reverse-i-search)`json': cat ww.^Con
~/src/zotero-better-bibtex/pandoc$ pandoc ww.md -o ww.json
~/src/zotero-better-bibtex/pandoc$ cat ww.json
{"blocks":[{"t":"Para","c":[{"t":"Cite","c":[[{"citationSuffix":[],"citationNoteNum":0,"citationMode":{"t":"NormalCitation"},"citationPrefix":[],"citationId":"wrigstad2017mastery","citationHash":0}],[{"t":"Str","c":"[@wrigstad2017mastery]"}]]}]}],"pandoc-api-version":[1,20],"meta":{}}

[kjambunathan]

If that doesn't output anything, pandoc is not running lua filters at all, full stop. There's nothing I can do about that -- for me it does run this filter, and outputs inlines four times. You'll have to get help from the pandoc people to see why this filter is not running.

Thanks for your time. I will narrow it down further. I have gathered some additional information to proceed ahead.

(Below remark is for benefit of search engines)

Lua filter is not of much importance to me. I will not be using pandoc at all. Instead, I will be using Emacs's Orgmode filters. So, JSON-RPC would be what I need. See https://github.com/retorquere/zotero-better-bibtex/issues/1582.

[retorquere]

Yeah, that AST looks good -- Inlines and Blocks aren't present in the AST, they're called when pandoc sees a certain formation of AST elements.

[kjambunathan]

This is another way to look at the AST.

$ echo 'See [@wrigstad2017mastery].' | pandoc -f markdown -t native

[Para [Str "See",Space,Cite [Citation {citationId = "wrigstad2017mastery", citationPrefix = [], citationSuffix = [], citationMode = NormalCitation, citationNoteNum = 0, citationHash = 0}] [Str "[@wrigstad2017mastery]"],Str "."]]

[retorquere]

Yeah I know but you get the same identifiers, and inside the Lua filters they look differently -- citation nodes in the AST as a Lue filter sees it have an id, not an citationId, and these nodes have a .t property that you find their type name on, and that's not present in the json/native AST dump.

[kjambunathan]

This is a pandoc issue.

zotero.lua doesn't work with this version of pandoc-2.9.1.1, specifically the version that comes with Debian sid

~$ pandoc --version
pandoc 2.9.1.1
Compiled with pandoc-types 1.20, texmath 0.12.0.2, skylighting 0.8.4
Default user data directory: /home/kjambunathan/.local/share/pandoc or /home/kjambunathan/.pandoc
Copyright (C) 2006-2019 John MacFarlane
Web:  https://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.

~$ dpkg -l | grep pandoc
ii  libghc-pandoc-citeproc-data           0.16.4.1-1                         all          Pandoc support for Citation Style Language - data files
ii  pandoc                                2.9.1.1-2                          amd64        general markup converter
ii  pandoc-citeproc                       0.16.4.1-1+b1                      amd64        Pandoc support for Citation Style Language - tools
ii  pandoc-data                           2.9.1.1-2                          all          general markup converter - data files

zotero.lua works with pandoc 2.10, specifically latest binary release from https://github.com/jgm/pandoc/releases/tag/2.10

~$ dpkg -l | grep pandoc
ii  libghc-pandoc-citeproc-data           0.16.4.1-1                         all          Pandoc support for Citation Style Language - data files
ii  pandoc                                2.10-1                             amd64        general markup converter
ii  pandoc-citeproc                       0.16.4.1-1+b1                      amd64        Pandoc support for Citation Style Language - tools
ii  pandoc-data                           2.9.1.1-2                          all          general markup converter - data files

~$ pandoc --version
pandoc 2.10
Compiled with pandoc-types 1.21, texmath 0.12.0.2, skylighting 0.8.5
Default user data directory: /home/kjambunathan/.local/share/pandoc or /home/kjambunathan/.pandoc
Copyright (C) 2006-2020 John MacFarlane
Web:  https://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.

[kjambunathan]

@retorquere As part of the resolution, you may want to check for pandoc version in zotero.lua. Thanks for your support and patience.

I will leave one more comment, for the benefit of people who could be hitting this bug through search engines.

[kjambunathan]

This comment captures the results of a successful run.

~/src/zotero-better-bibtex/pandoc$ make 1581
curl -L https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/master/site/content/exporting/zotero.lua -o zotero-bundled.lua
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48420  100 48420    0     0  12785      0  0:00:03  0:00:03 --:--:-- 12785
echo "[@wrigstad2017mastery]" > ww.md
pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=./zotero-bundled.lua -o paper2020-07-23T18:38:31-scannable-cite.odt ww.md
unzip paper2020-07-23T18:38:31-scannable-cite.odt content.xml
Archive:  paper2020-07-23T18:38:31-scannable-cite.odt
  inflating: content.xml             
grep @wrigstad2017mastery content.xml
<text:p text:style-name="Text_20_body">{ <text:s text:c="1" />| [@wrigstad2017mastery] | <text:s text:c="1" />| <text:s text:c="1" />| zu:0:TH5CXWRG }</text:p>

These are screenshots taken on LibreOffice

Output as produced by zotero.lua

Output after running it through Tools->ODF Scan

Output after after changing Document Preferences to use Chicago Author Date

Output after after inserting Bibliography