Zotero 6 support

[argenos]

Discussed in https://github.com/argenos/zotero-mdnotes/discussions/77

^{Originally posted by **argenos** March 20, 2021} Please use this thread to discuss ideas around the new annotation features in the Zotero beta: [Native PDF annotations](https://www.zotero.org/support/pdf_reader_preview). Quoting the reply I gave to an answer in [another thread](https://github.com/argenos/zotero-mdnotes/discussions/57#discussioncomment-459468) about supporting this: > I do have plans to support this once it's out of the beta. For now, you can try the following (taken from [here]):(https://www.zotero.org/support/pdf_reader_preview): > > > Annotations are stored in the Zotero database, not in the PDF file, which allows for much more advanced functionality as well as fast syncing. The File menu contains a new option to export PDFs, which will convert any Zotero annotations into standard annotations in the exported file. (An upcoming version will let you omit annotations, but in the meantime you can drag files to your filesystem to copy just the file.) Similarly, when exporting metadata from your library, there's a new “Include Annotations” option under “Export Files”. We plan to support other ways to export annotations in future updates. Your annotations will never be locked in Zotero. >

Functionality

Adapted from my reply here: https://github.com/argenos/zotero-mdnotes/discussions/77#discussioncomment-559982

Limitations of the beta

These will likely be addressed at some point, and I haven't tested recently to check their status.

The main limitations I found:

• Limited amount of colours (5 max, predefined)
• It only does highlights (no underlines, which are essential to me, others might use other PDF viewers that support other types of annotations too).
• The extracted annotations when saved as a note don't have the zotero:// link to the page. They have added something similar that works within Zotero, but that makes it useless right now if you want to export/copy-paste that note somewhere else.

• Extra steps required to save your highlights/annotations to your PDF (e.g. if you work on multiple devices or would rather use other apps)

Potential advantages

• image-style clipping - images stored in the DB might not be easy to get out (see https://github.com/argenos/zotero-mdnotes/issues/62)
• individual tagging of the annotations, I however couldn't get Zotero to show my annotations by filtering tags
• annotation searching - this was already possible IMO
• others have mentioned categorization, but I found no way to do that

Potential new features in Mdnotes

Other things missing that Zotfile offers, that might be covered by mdnotes, possibly during export:

• Grouping annotations by colour
• Custom formatting for the note title, highlights and annotations (particularly relevant if you want to make distinctions between the latter two)
• Wildcards
  • Using the colour and colour category wildcards to format the note's contents
  • Formatting the date as you see fit
• zotero:// links

FAQ

Does mdnotes work with the Zotero beta?

Short answer: No. The old workarounds might have stopped working with some of the latest upstream changes

When will you make it compatible?

Originally I had planned to do this once the limitations were addressed in the beta or when the beta was (about to be) released as stable. The amount of time I have to maintain one version, let alone two, is very limited.

I will try to schedule this sooner rather than later, since I don't want to disrupt people's workflows if I can help it. It also seems the Zotero clipper for Mac only works for the beta (and I know how useful that can be). Unfortunately, it's hard for me to say when exactly will that happen, or how long it will take (I depend a bit on the documentation for any new breaking changes upstream).

Related issues

126 #105 #130

Zotero forum discussions:

• Breaking changes for plugins wtih upcoming Zotero beta? - Zotero Forums
• zotero-dev discussion

[danieltomasz]

Does this discussion help in any way? https://forums.zotero.org/discussion/90753/zotero-beta-integration-with-external-note-taking-tools

[argenos]

@danieltomasz Thanks for that link! The link does help for #62! I already know how to get the annotations from Zotero's DB, it's more a matter of available time. I haven't reserved time for this since the viewer is too limited in my opinion, and I was hoping to do this in one go and once it was stable/released. I just don't have time to maintain the beta-compatible version and the stable one right now.

[danieltomasz]

Yes, I think that in the state of undocumented API of beta the idea of implementing that and hoping no change is really risky.

What left is to convince Zotero developers :) that there is a really growing interest for better API for integration with external note taking tools (Obsidian, Logseq, Roam)

[argenos]

Absolutely! I also want(ed) to avoid work that might be covered by the beta, and just add the customization options that mdnotes provides:

Similarly, when exporting metadata from your library, there's a new “Include Annotations” option under “Export Files”. We plan to support other ways to export annotations in future updates. Your annotations will never be locked in Zotero.

[enricllagostera]

An update that the Mdnotes >Export seems to be working again as of Zotero Beta 5.0.97-beta.46+4cfa23050. (Sorry if this is not the best place to give this info).

[vsanchez]

I can confirm what @enricllagostera mentions, but only in Windows, not in Linux :-(

[dingshanliang]

An update that the Mdnotes >Export seems to be working again as of Zotero Beta 5.0.97-beta.46+4cfa23050. (Sorry if this is not the best place to give this info).

I can confirm this on macos. But "create mdnotes file" still does not work on macos. zotero version: 5.0.96.3 MDnotes version: 0.1.3

[panyuz]

Zotero 6 may be coming “soon”. Are there any plans to release an update?

[argenos]

@panyuz yep, working on this. Where did you see an update on the release date? I am not aware of any concrete timeline.

[Shamp0o]

Just a heads up. Zotero is working on a similar feature.

[panyuz]

@panyuz yep, working on this. Where did you see an update on the release date? I am not aware of any concrete timeline.

The zotero development team will not announce the ETA. i just saw in the zotero forum that the developer mentioned that the status of zotero 6 is "soon"

[argenos]

Most likely, people relying on this workflow need to make sure to stop Zotfile from automatically being updated based on this commit: https://github.com/jlegewie/zotfile/commit/773b608706f575d0a6a5772097a859df961e8018

v5.0.16 is the one that is currently compatible with mdnotes

I just wanted to add that I'm sorry it's taken so long! I have seen the announcement about the markdown export in the beta and had some time to test it a few weeks back, but it's still unclear to me when will the beta be released and what exactly is in their roadmap. This makes me hesitant to continue with the migration, since I can't say how much of mdnotes will be actually part of Zotero core and I have a limited amount of time I can devote to this and other responsibilities I need to prioritize. Not sure if @dstillman can shed any light here.

[dstillman]

it's still unclear to me when will the beta be released

Very soon. If you're updating your plugin for Zotero 6, now's the time.

It's already possible to export notes with annotations to Markdown in the beta with optional links back to Zotero, and we're committed to continuing to improve Markdown export, including adding additional export options and converting annotation citations to pandoc-style citations.

[argenos]

Thanks @dstillman, I'll keep that in mind. Any chance you'd have insights on what else is planned for the markdown export on the new version? Particularly this part from the forum announcement:

We'll be adding more customizability to this in future versions, so let us know what sorts of options you'd like to see.

If you are planning to add templating with something like nunjucks [1] or handlebars [2], then that would cover most of the cases mdnotes currently covers and I can most likely deprecate mdnotes, and create a smaller, easier to maintain plugin with what is currently out of scope of or planned for Zotero.

As I've mentioned above and on the zotero-dev group, I'd really like to invest my time wisely and avoid duplicating any features you'll include in Zotero core. I have little experience with JavaScript/Typescript, so digging through code and other repos to find out what needs changing and how to implement it requires time and planning on my part.

[1] https://mozilla.github.io/nunjucks/ [2] https://handlebarsjs.com/

[dstillman]

@argenos: We're working on supporting templates to control how annotations are added to notes, similar to ZotFile — so you could have, say, the quote in a blockquote instead of inline with quotation marks, or put your comment before the highlight text.

It's possible we'll support templates for Markdown note export — which would take control how Zotero notes, with embedded metadata, are converted to Markdown — but I'm less sure about that. We may just add additional fixed options beyond the existing zotero:// link option, like whether to include a textual representation of the color in the output, whether to include pandoc citations, etc. We're not currently planning to support export of item data to Markdown, and so I don't see us including a full templating engine (which would be overkill for note export).

[wangfh5]

In my workflow of using mdnotes, zotero notes are not the most important but the metadata, item links and pdf links are. Can somebody tell me how to collect all the metadata to a markdown file without using mdnotes in zotero beta 6? Including the following information from mdnotes default template

## Metadata

{{itemType}}
{{author}}
{{proceedingsTitle}}
{{date}}
{{dateAdded}}
{{url}}
{{DOI}}
{{citekey}}
{{collections}}
{{related}}
{{tags}}

##  Zotero links

{{localLibrary}}
{{cloudLibrary}}
{{pdfAttachments}}

I would try to go back to zotero beta's early version for now.

[rxy0424]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

[llxlr]

Zotero 6 can export note in Markdown. Zotero 6: Your research workflow, transformed

[llxlr]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

you can change install.rdf and update.rdf in mdnotes.xpi. (If you really need) :

      <Description>
        <em:id>zotero@chnm.gmu.edu</em:id>
        <em:minVersion>5.0.79</em:minVersion>
-       <em:maxVersion>5.0.*</em:maxVersion>
+       <em:maxVersion>6.0.*</em:maxVersion>
      </Description>

[rxy0424]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

you can change install.rdf and update.rdf in mdnotes.xpi. (If you really need) :

      <Description>
        <em:id>zotero@chnm.gmu.edu</em:id>
        <em:minVersion>5.0.79</em:minVersion>
-        <em:maxVersion>5.0.*</em:maxVersion>
+        <em:maxVersion>6.0.*</em:maxVersion>
      </Description>

Thanks for your advice.

[IronShannon]

Zotero 6 can export note in Markdown. Zotero 6: Your research workflow, transformed

Hello! Why I can not see the "Markdown" in the export format? Does it need some extra settings?

[xfap]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

you can change install.rdf and update.rdf in mdnotes.xpi. (If you really need) :

      <Description>
        <em:id>zotero@chnm.gmu.edu</em:id>
        <em:minVersion>5.0.79</em:minVersion>
-        <em:maxVersion>5.0.*</em:maxVersion>
+        <em:maxVersion>6.0.*</em:maxVersion>
      </Description>

Thanks! It works and looks well!

[llxlr]

Zotero 6 can export note in Markdown. Zotero 6: Your research workflow, transformed

Hello! Why I can not see the "Markdown" in the export format? Does it need some extra settings?

Your software version should be 6.0.

[IronShannon]

Zotero 6 can export note in Markdown. Zotero 6: Your research workflow, transformed

Hello! Why I can not see the "Markdown" in the export format? Does it need some extra settings?

Your software version should be 6.0.

Thank you! I have solved the problem. I had updated to Zotero 6.0 before I submited the question. The reason that I could not find "Markdown" in the export format is that I selected the Zotero item instead of the Zotero note.

[ibeBrody]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

you can change install.rdf and update.rdf in mdnotes.xpi. (If you really need) :

      <Description>
        <em:id>zotero@chnm.gmu.edu</em:id>
        <em:minVersion>5.0.79</em:minVersion>
-        <em:maxVersion>5.0.*</em:maxVersion>
+        <em:maxVersion>6.0.*</em:maxVersion>
      </Description>

Thanks! It works and looks well!

Could you please elaborate on how you got it to work? I tried modifying the two files then installing them via Zotero but it still says unsupported version? Thanks in advance

[crystal-nj]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

you can change install.rdf and update.rdf in mdnotes.xpi. (If you really need) :

      <Description>
        <em:id>zotero@chnm.gmu.edu</em:id>
        <em:minVersion>5.0.79</em:minVersion>
-        <em:maxVersion>5.0.*</em:maxVersion>
+        <em:maxVersion>6.0.*</em:maxVersion>
      </Description>

Thanks! It works and looks well!

Could you please elaborate on how you got it to work? I tried modifying the two files then installing them via Zotero but it still says unsupported version? Thanks in advance

I have the same problem. I could not open the updated one because it was still incompatible

[xfap]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

you can change install.rdf and update.rdf in mdnotes.xpi. (If you really need) :

      <Description>
        <em:id>zotero@chnm.gmu.edu</em:id>
        <em:minVersion>5.0.79</em:minVersion>
-        <em:maxVersion>5.0.*</em:maxVersion>
+        <em:maxVersion>6.0.*</em:maxVersion>
      </Description>

Thanks! It works and looks well!

Could you please elaborate on how you got it to work? I tried modifying the two files then installing them via Zotero but it still says unsupported version? Thanks in advance

What？ Maybe I just do it like yours. Don't know why~

Here is what I did:

0. download the mdnotes' xpi file (maybe unzip it or use a compressing soft.) and edit the install.rdf file. (JUST this one)
1. Reinstall the edited xpi file. (don't need to uninstall the previous one)
2. Close Zotero totally. (make sure that there's no task named 'zotero' running. [by ctrl+shift+esc on Windows])
3. Start Zotero to check if it works.

Below is the file edited. (should modify the suffix .zip to .xpi) mdnotes-0.1.3-rdf to 6.zip

[crystal-nj]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

you can change install.rdf and update.rdf in mdnotes.xpi. (If you really need) :

      <Description>
        <em:id>zotero@chnm.gmu.edu</em:id>
        <em:minVersion>5.0.79</em:minVersion>
-        <em:maxVersion>5.0.*</em:maxVersion>
+        <em:maxVersion>6.0.*</em:maxVersion>
      </Description>

Thanks! It works and looks well!

Could you please elaborate on how you got it to work? I tried modifying the two files then installing them via Zotero but it still says unsupported version? Thanks in advance

What？ Maybe I just do it like yours. Don't know why~

Here is what I did: 0. download the mdnotes' xpi file (maybe unzip it or use a compressing soft.) and edit the install.rdf file. (JUST this one)

1. Reinstall the edited xpi file. (don't need to uninstall the previous one)
2. Close Zotero totally. (make sure that there's no task named 'zotero' running. [by ctrl+shift+esc on Windows])
3. Start Zotero to check if it works.

Below is the file edited. (should modify the suffix .zip to .xpi) mdnotes-0.1.3-rdf to 6.zip

Thank you!

[macbikegeek]

Thanks for the detailed instructions. Working for me on Zotero 6 beta! 🙏🏻😁

[llxlr]

Thanks for your great plugin for zotero, I use it almost everyday when reading papers. However, now Zotero 6 is finally released and mdnotes doesn't work with it. It would be selfish to ask you to update, but I'd like to ask what I can do to help adapt this amazing plugin for Zotero 6?

you can change install.rdf and update.rdf in mdnotes.xpi. (If you really need) :

      <Description>
        <em:id>zotero@chnm.gmu.edu</em:id>
        <em:minVersion>5.0.79</em:minVersion>
-       <em:maxVersion>5.0.*</em:maxVersion>
+       <em:maxVersion>6.0.*</em:maxVersion>
      </Description>

In Advanced -> Advanced Configuration -> Config Editor, set up extensions.checkCompatibility.6.0 to false, and You can ignore all plugins compatibility issues in Zotero 6.0 instead of changing the mdnotes.xpi file. Until the plugin resolves the compatibility issue, you can reset to true.

A temporary solution of incompatible of Zotodo

[gitoffdabus]

Hi everyone, thank you for your great help in fixing the Zotero 6 MD notes issue.

I tried uploading the file mentioned above and changed it to a xpi file: Below is the file edited. (should modify the suffix .zip to .xpi) [mdnotes-0.1.3-rdf to 6.zip]

When I restarted Zotero, MD notes did show up but when I tried to export to markdown, it prompted me to open up a new folder rather than extracting the pdf and creating a md file of the pdf annotations in Zotero.

Has anyone else had any luck with the above fix for mdnotes in Zotero 6? Could I convert to an earlier Zotero version? Im on a mac. Thanks for any help - much appreciated. My exam is in under a month so this bug has been stressful.

What it looks like when I click the PDF:

What happens when I click export to markdown:

[wangfh5]

Hi @gitoffdabus , I'm also using Mac. That window is asking for a folder where your extracted markdown file will be located on. You just choose your notes folder, and then press open. Actually, this process is normal to me. I think that it existed before Zotero 6.

[gitoffdabus]

Hi @wangfh5, thanks for following up. When I click open, nothing populates in the folder I selected. I was trying to figure out if there was a problem with my MDnotes preferences (see below) but that was never a problem for me before.

Before the Zotero 6 update, I would click MDNotes, click Create MD file or export to markdown (I honestly can't remember which at this point) and then a file called "Extracted Annotations" would populate as a sub-item listed under my main citation. I pasted an example below. I would really like to be able to replicate this workflow as I am copy pasting these annotations directly into Obsidian. Please let me know if there's any recommendations on how to return to this workflow or fix my issue with the MDnotes fix offered above.

[gitoffdabus]

Further follow up to my above investigation - I experimented with annotating directly in the Zotero 6 PDF viewer. I am able to generate a note if create a note on the right and I drag and drop all of my highlights and comments from the left side (see screenshot below).

This will generate a note with the annotations as a sub-item under the citation (similar to MDnote) but when I attempt to extract into markdown, it doesn't generate the same hyperlinks to the page numbers as in the original (see what it looks like when I copy paste the markdown into Obsidian below). I am not jazzed about having to annotate PDFS in Zotero now.

[gitoffdabus]

Okay, I figured it out. If you upload a pdf with annotations from a third party viewer, do the following: 1) After uploading the annotated PDF into Zotero, Open in the Zotero PDF viewer 2) Create Item Note from Annotations 3) Go back into the main Zotero viewer and you should see a new note listed as a sub-item under your main citation. Click export note as markdown and you should save a file that copy pastes exactly as it used to into Obsidian.

Thanks everyone!

[erazlogo]

@gitoffdabus You can drag and drop annotations directly to Obsidian with links if you press Shift at the same time, or you can disable the Setup > Editor > "Auto convert HTML" in Obsidian. Then the links will show up but you won't be able to convert italics when pasting HTML.

[wangfh5]

@gitoffdabus Since what you want are just the annotations of pdf, I think that the new properties of Zotero 6 will fit your workflow well. And if you want to extract the metadata of any item without this add-on mdnotes, I suggest you try the plugin citaion in Obsidian.

However, I have to say that mdnotes do work for me. Here is my mdnotes preference:

https://user-images.githubusercontent.com/51868839/159293164-9c1e479d-feb4-406c-9d57-943aa852cb74.mov

I have tried split file instead of single file too and it still work.

[argenos]

There's likely going to be stuff not working out of the box, but I've added the changes for #126 and updated the maxVersion(thanks @llxlr) on the last release: https://github.com/argenos/zotero-mdnotes/releases/tag/0.2.0-alpha5

Since I don't know what is broken yet, please let me know if you find something not currently tagged as Zotero 6: https://github.com/argenos/zotero-mdnotes/issues?q=is%3Aopen+is%3Aissue+label%3Azotero-beta

Please note this is currently just fixing broken things, not adding anything new (i.e. the new annotations not in the PDF, etc)

[brimwats]

this is FR and perhaps now out of scope—I am not sure if the annotation changes on Zotero's side dooms #92 — if I could dig up an old version of the zotero installer I'd still love to purge the nonexistent items. It would be great if I knew how to get into the sqlite database and delete all note attachments instead

after thinking about this today, perhaps the best route is actually just to export with files but no notes and then reimport everything!

[argenos]

The latest release is out, so I'll be closing this.