search

mgmeyers / obsidian-zotero-integration

Insert and import citations, bibliographies, notes, and PDF annotations from Zotero into Obsidian.
GNU General Public License v3.0
1.09k stars 55 forks source link

Annotations not coming over to Obsidian for one particular Zotero item #363

Open chlorophilo opened 6 months ago

chlorophilo commented 6 months ago

Hi Thanks for your plugin. It has been a lifesaver. In your FAQ you mention this: "There might be a bug in Bibtex or the Zotero Integration plugin. Zotero has the annotation but its not seen in Obsidian." I am experiencing this issue with one of my Zotero items. The item in question has what amounts to about 25 pages of annotations, and I really need them brought into the Obsidian research note using my usual template. While I have created dozens of research notes using your plugin, something is going wrong with this item and after much playing around I still have no idea why. Unlike the OP in your reference #107, my Zotero item only has the one pdf - no html attachment to confuse things. I have approached retorquere - the creator of Better Bibtex - in the first instance because before I saw this statement in your FAQ I thought it was probably in the BBT mechanism. That exchange is here: https://github.com/retorquere/zotero-better-bibtex/issues/2856. As you can see, retorquere is saying:

  1. Zotero doesn't provide annotations to exporter.
  2. "I don't actually know how the obsidian plugin calls BBT. If the developer of the Obsidian plugin says what it needs I may be able to help, but until the dev does, there's nothing for me to fix. If the obsidian plugin is using BetterBibTeX JSON exports, it is a know problem that you don't get annotations, because Zotero doesn't provide them to exporters."

I hope you can assist.

the1gofer commented 6 months ago

getting the same thing, except with all of my items.

when using the modal i get the integration prompt, searcing for the document doesn't find it, however I can select it using classic view. After selecting, it switches back to obsidian and nothing happens.

Developer view shows no error.

retorquere commented 6 months ago

I think the obsidian integration calls into one of the JSON-RPC methods that BBT offers. If @mgmeyers can replicate your problem, I can work with him get to a diagnosis.

Autoparallel commented 6 months ago

I seem to have this same issue and I've narrowed it down to:

@retorquere when I use BBT to output JSON, I notice that:

I have tried this on Zotero 6.0.37 and Zotero 7.0.0-beta.77+adaa61f2c with the exact same issue both using the BBT 6.7.186 version.

Note that I saw Issue zotero-better-bibtex #2856 which I know you're aware of.

Is there any testing or help I can put in here to resolve this issue?

mgmeyers commented 6 months ago

@Autoparallel Does this happen with all items in group libraries or just some? Do you happen to have duplicate citekeys between libraries? Also, do you annotate in Zotero or with another PDF reader?

mgmeyers commented 6 months ago

@chlorophilo Is the item you're trying to import in a group library? Do the annotations show up in the Data Explorer? Also, are there any errors (red text) in Obsidian's developer console when you run the import / view the item in the Data Explorer? You can open it with command + opt + i / ctrl + shift + i

chlorophilo commented 6 months ago

OK I feel like we are getting somewhere. The data explorer showed no annotations, whereas other items do, so it is not eg a templating issue. But the recent issue I had did involve a file that shared a citekey with an item in a Group Library.

I had taken care not to select the items from the Group Library when using the Zotero item selector when running Zotero Integration. But even so no annotations got imported. Only when I delete the items from the Group Library and run Zotero integration do the annotations then import into Obsidian.

Cc @retorquerehttps://github.com/retorquere

From: Matthew Meyers @.> Sent: Thursday, May 9, 2024 5:03 AM To: mgmeyers/obsidian-zotero-integration @.> Cc: Andrew Newton @.>; Mention @.> Subject: Re: [mgmeyers/obsidian-zotero-integration] Annotations not coming over to Obsidian for one particular Zotero item (Issue #363)

@chlorophilohttps://github.com/chlorophilo Is the item you're trying to import in a group library? Do the annotations show up in the Data Explorer? Also, are there any errors (red text) in Obsidian's developer console when you run the import / view the item in the Data Explorer? You can open it with command + opt + i / ctrl + shift + i

— Reply to this email directly, view it on GitHubhttps://github.com/mgmeyers/obsidian-zotero-integration/issues/363#issuecomment-2101847644, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB2VYK5OO77AL7FXTJRYNEDZBLRNXAVCNFSM6AAAAABHA7J23GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBRHA2DONRUGQ. You are receiving this because you were mentioned.Message ID: @.**@.>>

retorquere commented 6 months ago

@retorquere when I use BBT to output JSON, I notice that

BBT JSON? CSL JSON? Something else?

* For items in "My Library" a PDF is included with annotations

PDF included in the JSON? I don't follow.

* For items in any of my "Group Libraries" has a PDF, but annotations are missing

Missing in the PDF? In the JSON? Which JSON format? None of the export formats include annotations as far as I know.

Note that I saw Issue zotero-better-bibtex #2856 which I know you're aware of.

Yeah but which I've closed because the reporter has gone dark.

retorquere commented 6 months ago

But the recent issue I had did involve a file that shared a citekey with an item in a Group Library.

The JSON-RPC endpoint takes the first item across all libraries that has the given citation key, and "first" is likely by insertion order; I don't honestly know. I can add a parameter to allow selecting a library, but the obsidian side would have to make use of it for that to be useful.

retorquere commented 6 months ago

Please don't CC me in email. If you @mention me here I get notified.

mgmeyers commented 6 months ago

@chlorophilo If you don't mind changing your cite keys, setting this setting may fix the issues for you:

Screenshot 2024-05-09 at 8 02 46 AM

@retorquere Passing along the library ID to the item.* methods would be pretty easy on my end.

Autoparallel commented 6 months ago

@Autoparallel Does this happen with all items in group libraries or just some? Do you happen to have duplicate citekeys between libraries? Also, do you annotate in Zotero or with another PDF reader?

Happens with all of them.

@retorquere What I mean is when I take an item in a library and right click + Export, I am prompted with this:

image

Upon inspecting the output files here, I get Exported Items.json and a directory called files/ that contains another numbered directory with a PDF in it.

When exporting from a file in "My Library", the PDF included in files/[SOME_NUMBER]/[SOME_FILE].pdf includes annotations, but for the file in a group library, there are no annotations included.

I'm not familiar with the full flow here, sorry. My guess is the JSON has a field that points to annotations located here that are stored alongside the PDF and given that they're missing, annotations appears empty to obsidian-zotero-integration. Of course, I may be wrong about this. That's just my guess!

retorquere commented 6 months ago

Passing along the library ID to the item.* methods would be pretty easy on my end.

v6.7.188 will drop in 10 minutes or so that will accept it (name or ID) as an optional parameter. Default is now to search only the user library when nothing is passed. To search all libraries, pass '*'.

When exporting from a file in "My Library", the PDF included in files/[SOME_NUMBER]/[SOME_FILE].pdf includes annotations, but for the file in a group library, there are no annotations included.

Does this also happen when you export with Zotero RDF?

Autoparallel commented 6 months ago

Does this also happen when you export with Zotero RDF?

Using this, I found that the PDF output for both "My Library" and group libraries had annotations (at least for the items I tried this on).

Anything else I can test for you? Happy to keep trying things!

retorquere commented 6 months ago

And this is with "background export" off for the BBT export?

Autoparallel commented 6 months ago

And this is with "background export" off for the BBT export?

Correct

retorquere commented 6 months ago

The weird thing about that is that BBT isn't involved in saving the file - translators don't write anything to disk, they just tell zotero "save this file in that directory", so there should be no difference between the RDF export and the bbt export where it concerns attachments. I'll try to figure out what could explain this difference, but this should simply not be possible.

Autoparallel commented 6 months ago

I appreciate the time and energy. Just ping me if I can be of any help.

mgmeyers commented 6 months ago

Doesn't sound like this will solve everyone's issue, but I've updated things on my end to pass along the library ID when fetching annotations from zotero. This should help in the case of clashing IDs between libraries.

chlorophilo commented 6 months ago

Thank you very much! Much appreciated.

From: Matthew Meyers @.> Sent: Saturday, May 11, 2024 5:15 PM To: mgmeyers/obsidian-zotero-integration @.> Cc: Andrew Newton @.>; Mention @.> Subject: Re: [mgmeyers/obsidian-zotero-integration] Annotations not coming over to Obsidian for one particular Zotero item (Issue #363)

Doesn't sound like this will solve everyone's issue, but I've updated things on my end to pass along the library ID when fetching annotations from zotero. This should help in the case of clashing IDs between libraries.

— Reply to this email directly, view it on GitHubhttps://github.com/mgmeyers/obsidian-zotero-integration/issues/363#issuecomment-2105929158, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB2VYK55GN7BOX6YD4YRUCTZBYYWTAVCNFSM6AAAAABHA7J23GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVHEZDSMJVHA. You are receiving this because you were mentioned.Message ID: @.**@.>>

retorquere commented 6 months ago

If someone can reproduce the problem, I can try to make diagnosis builds available on https://github.com/retorquere/zotero-better-bibtex/issues/2856. I don't currently know where to look specifically, because as far as I know, there's no way for BBT to cause this. But BBT does a lot of crazy stuff, and apparently there is a way for BBT to cause this.

FeralFlora commented 6 months ago
  • trying the suggestion with the cite keys did not fix the issue on my end.

After changing the setting, you have to refresh the citekeys for it to take effect. Otherwise, it only enforces no duplicates across libraries for any new items you add, but not the existing ones.

As far as I know, you have to select all the items, right click on them, then press Better Bibtex -> Refresh Citekey. Maybe someone knows an easier way to refresh the citekeys? But of course, beware that you might lose track of which items you've previously cited, if the old and new citekey are not the same (due to citekey formula changes, or conflict resolution).

But if I understand the new feature (passing along the libraryID), then conflicts across libraries shouldn't be an issue anymore?

retorquere commented 6 months ago

But if I understand the new feature (passing along the libraryID), then conflicts across libraries shouldn't be an issue anymore?

correct