mgmeyers / obsidian-zotero-integration

Insert and import citations, bibliographies, notes, and PDF annotations from Zotero into Obsidian.
GNU General Public License v3.0
996 stars 55 forks source link

Zotero 7 and annotations not importing anymore #385

Open bloosqr opened 3 weeks ago

bloosqr commented 3 weeks ago

With Zotero 6 and the integration plugin, annotations imported perfectly. With Zotero 7 annotations aren't showing up at all. If I use data explorer to look at all the entries annotations are empty. However if I open Zotero 6 and create annotations in Zotero 6, everything works perfectly.

What is odd about this is if I create an annotation in Zotero 7, Zotero 6 sees it but obsidian still will not import it in while connecting to Zotero 6 (e.g. seeing the annotation created in Zotero 7 but trying to import from Zotero 6).

The only way annotations are working currently is if I create new annotations in Zotero 6 and then import from Zotero 6, the obsidian plugin works.

I can't tell if this is a better bibtex bug or a Zotero integration bug but something is not talking to something properly with the new annotation approaches built into Zotero 7.

regards avi

stuarttravers commented 3 weeks ago

I've just noticed this issue as well. No annotations being imported since upgrading to Zotero 7

websurfer93 commented 3 weeks ago

I have the same issue. Annotations and even Zotero links are not importing anymore. (Version: 3.2.1)

wdenton2 commented 3 weeks ago

Same issue here. No annotations since Zotero 7

mgmeyers commented 3 weeks ago

Hi all. Please uninstall and reinstall the better bibtex plugin and see if that fixes the issue for you.

wdenton2 commented 3 weeks ago

Just tried, didn't seem to work. Opened a paper in the data explorer that has annotations and it still returns 0.

Edit: Tried uninstalling the obsidian plugin as well and doing a fresh install. Same issue occurs

mgmeyers commented 3 weeks ago

Hmm strange. Not really sure what could be going on. Annotations seem to be coming through just fine on my end.

bloosqr commented 3 weeks ago

No I have tried this as well. How are you pulling the annotations ? Is there something we can look at in a better bibtex export? The issue right now is data explorer is not showing any annotations that are generated in / with Zotero 7, so the template has nothing to import.

If it is helpful I am on a Mac OS X M2 machine.

sikirebirth commented 3 weeks ago

im not sure if im running into the same issue as everyone else but I'm not able to insert citations image please let me know if I should make a separate issue.

NanamiKubota commented 3 weeks ago

Leaving a comment here as I am running into the same issues after updating from Zotero 6 to Zotero 7. I tried to uninstall and reinstall but the problem persists. The Zotero Data Explorer shows that I have 0 annotations even though the pdf is annotated. The other metadata appears to import just fine (see picture below).

Here are my version numbers for zotero and plugins (on macOS M1):

I also don't see an option to insert citations as mentioned by sikirebirth.

Screenshot 2024-08-22 at 3 08 28 PM
mgmeyers commented 3 weeks ago

@NanamiKubota Are there any annotations listed under either of the attachments?

Screenshot 2024-08-22 at 1 38 55 PM
NanamiKubota commented 3 weeks ago

Thanks for the quick reply! I do not see any annotations listed under the attachments:

Screenshot 2024-08-22 at 4 40 37 PM

mgmeyers commented 3 weeks ago

Hmm, the only other thing I can think of is to check if there are any errors in Zotero's error console after running an import thought the data explorer:

Screenshot 2024-08-22 at 2 24 34 PM

Also check if there are any errors in Obsidian's developer console:

Screenshot 2024-08-22 at 2 25 56 PM
bloosqr commented 2 weeks ago

As I was looking at this thread, I thought I'd try one more thing. I got myself the Zotero API and printed out the full dump of an item that I am trying to import using both Zotero 7 and Zotero 6 e.g. in a simple python script:

from pyzotero import zotero

Initialize with the given library ID

library_id = 'XXXX' library_type = 'user' # Assuming this is a personal library api_key = "XXXXX"

Create a Zotero instance

zot = zotero.Zotero(library_id, library_type, api_key)

Retrieve the item by its ID

item_id ='UVI7FDW8' item = zot.item(item_id)

Retrieve the children of the item

children = zot.children(item_id)

Print the item's data and its children

print("Item Data:") print(item) print("\nChildren:") print(children) ~ ~

What is bizarre is the output of that newly created entry in both Zotero 6 and 7 looked completely sensible to me (and the same) if I create the pdf and highlight the same section in both Zotero 6 and 7.

So then I tried reimporting the same entry from Zotero 7 with obsidian and lo and beyond it works now.

I have tried now to reproduce the lack of importing in Zotero 7 with new Zotero entries and old Zotero entries etc but now everything imports properly. Unfortunately I can't tell what fixed it (me running the script or just something else in my system that changed ) but on my side Zotero 7 and annotation importing seems to work now.

NanamiKubota commented 2 weeks ago

I can now get new highlights made in Zotero 7 (but not old annotation from Zotero 6) imported into Obsidian. Things that worked:

  1. First importing new documents via Zotero connector (Chrome extension) in Zotero 7, create new annotations, then import via Zotero Integration. This is good for new documents but problematic if you want to import old annotations from old documents made back in Zotero 6.

  2. If a document has multiple attachments, deleting attachments until there is only one PDF file remaining (see more detail below).

Things that do not work:

  1. Importing a document with multiple PDF attachments
  2. Importing a document with PDF + 'PubMed Central Link' attachment -> deleting the PubMed Central Link removes the import issue (not sure why this worked). The 'PubMed Central Link' attachment is a file that Zotero connector created back when I was using Zotero 6 and importing documents from PubMed.

Other issues:

flogrows commented 2 weeks ago

I too noticed yesterday that annotations weren't importing anymore. Similarly to @NanamiKubota, I've found it to be related to having multiple attachments to an item. All of my Zotero items have at least one extra attachment created by the Notero plugin.

My workaround is to temporarily move any extra attachments to the Zotero bin while I import my annotations to Obsidian, then restore them. This works fine - the annotations now appear in the Data Explorer and can be imported to a note.

I haven't noticed any issue with old/new annotations, but possibly because I don't tend to annotate within Zotero itself, but using another pdf editor, so the annotations are embedded in the pdf file.

Also: I'm not sure that in my case the issue was caused by the update to Zotero 7, as I noticed the problem yesterday before I had updated. It was in looking around to see what might have changed that I noticed the pending update. There had also been an Obsidian update (via flathub) the day before.

bloosqr commented 2 weeks ago

I can confirm this now what others have said above as I just hit the same problem again and realized @flogrows and @NanamiKubota solution actually works for me. The issue is if there is another attachment in Zotero 7 nothing gets imported. (In fact I was getting hit by the pubmed link issue with the connector which had not been a problem with zot. 6) However if one removes the other attachment the importing works fine. If I delete or temporarily move the attachment during import I can import fine. (The reason I thought the problem was fixed was I was "testing" Zotero by just moving a single test PDF into zotero so did not have the attachments I typically get when I use Zotero connector to import pdfs from a web browser.

As an aside I wonder if we could also fix the multiple attachments issue with regards to annotations. I noticed with Zotero 6 the template would randomly pick one pdf (if there are supplementary pdfs to look at). It would be great if it scanned through all pdfs (or annotatable things) and imported all of those as part of the import.

NanamiKubota commented 2 weeks ago

Adding on to note that I finally was able to import both old annotations from Zotero 6 and new annotations from Zotero 7 using my custom template. Previously, I had issues where Zotero Data Explorer showed both new and old annotations for my custom template but when I went to import using my custom template, only the new annotations came up.

To get the import to work, I had to delete my old Obsidian entry from the Zotero 6 era and make a new entry via Zotero import, rather than appending new annotations to the old Obsidian document using the {% persist "annotations" %} line in my custom template.

mgmeyers commented 2 weeks ago

Ok, finally figured out what's going on here. Non-file attachments are causing errors in better bibtex (my fault). The next version of the Better Bibtex plugin will fix the issue.

bloosqr commented 2 weeks ago

Hey thank you so much for fixing this so proactively! Super appreciated!

NanamiKubota commented 2 weeks ago

Thank you so much! Really appreciate the time and effort. This plugin has been a life-saver!

mgmeyers commented 2 weeks ago

Ok, please update better bibtex and see if that fixes things for you.

sikirebirth commented 2 weeks ago

in regards to me mentioning citation not being available for anyone else wondering image In the options for the plugin I added a citation formatting style and is it now allowing citations image image

I apologize if this was mentioned elsewhere and I missed it.

flogrows commented 2 weeks ago

Ok, please update better bibtex and see if that fixes things for you.

Yes, all good now! Thanks so much :)

stefanpricopie commented 2 weeks ago

I would assume the error below is related to the bibtex issues in this thread. I am getting the following error for any Zotero Integration action in Zotero 7. Zotero 6 on the other hand works fine for me

Screenshot 2024-08-28 at 12 57 20

Better BibTex 6.7.225 Zotero 7.0.2 Zotero Integration 3.2.1 Obsidian 1.6.7 (Installer 1.5.3)

Thank you for your help on this!

AuLaSW commented 2 weeks ago

Will support for annotations on non-PDF files be added? If I annotate, say, a webpage in Zotero 7 the annotations don't show up in Zotero Data Explorer, but the annotations for PDFs do.

EureDudeheit commented 9 hours ago

I have Zotero 7.0.5 and betterbibtex 6.7.227 on Linux. When i want to import the notes in Obsidian, the error console in Zotero says:

`Error: ae:sql:get: undefined is not a string but a undefined 3 better-bibtex.js:17183:48

TypeError: this.gViewSourceUtils is undefined webconsole.js:223:5 viewSource resource://devtools/client/webconsole/webconsole.js:223 onViewSource resource://devtools/client/webconsole/service-container.js:43 onClick resource://devtools/client/shared/components/Frame.js:265 React 19 invokeGuardedCallbackImpl invokeGuardedCallback invokeGuardedCallbackAndCatchFirstError executeDispatch executeDispatchesInOrder executeDispatchesAndRelease executeDispatchesAndReleaseTopLevel forEachAccumulated runEventsInBatch runExtractedEventsInBatch handleTopLevel batchedUpdates$1 batchedUpdates dispatchEvent 1 unstable_runWithPriority interactiveUpdates$1 interactiveUpdates dispatchInteractiveEvent

`