Bringing Crossref, Semantic Scholar, Open Citations and Open Alex lookup + auto-import + redesign to Cita for Zotero 7

[thebluepotato]

Fixes #51, fixes #134, fixes #168, fixes #173

Hi! I adapted the (now stale) PR #139 to the new Zotero 7 branch so it has a chance to be swept up in the new release. The general logic is unchanged from the other PR, but I've made quite a few updates for efficiency, code clarity and type safety as well as fixed a few failing Promises here and there. I've tested quite a bit already, but it could definitely use more in-depth testing.

And I've also added a button to citations to auto-import that reference into Zotero with one click and then link it. It's similar to what https://github.com/MuiseDestiny/zotero-reference does, but I find that addon confusing at best and it doesn't help that all the info is in Mandarin Chinese...

All in all, probably still a WIP, but happy to receive code reviews and have some people test this!

[thebluepotato]

Excellent, will test soon!

For the upload limit, I have no idea, but I'd be surprised if there were more exceptions than notes. Maybe you should ask on the Zotero dev forum?

[Dominic-DallOsto]

Ok, I put the note deleting and saving in transactions and that works.

It was a bit fiddly to get working but it seems this combination of using async function, for loops instead of maps, and await for the save or erase functions is what works.

Let me know if it also works for you

[Dominic-DallOsto]

One thing I noticed though is that rendering all the citations is quite slow when there are 500 of them. Do you get the same behaviour?

Screencast from 2024-11-10 21-38-58.webm

[thebluepotato]

I've updated the citation line counting logic to only trigger when the line is actually visibile + switched from a hash to UUID to keep track of the citations. Should render much faster now, but loading all these notes is still quite intense.

[Dominic-DallOsto]

Nice, that's much faster!

Are there any other things you'd like to add / change or do you think we should start working towards finalising and merging this? Overall it's a huge gain in functionality for Cita!

[thebluepotato]

Yeah, it definitely outgrew its initial goal! I'm very happy with how it is and would love for people to test it a bit more. We can definitely move to finalising/merging, though there's just one final bug I'm unhappy with currently:

When searching PIDs from Semantic Scholar by title, you have to send a first request to get the paper ID (a hash), and then a second one to get the details. I've not been able to figure out a good way to handle the very strict rate limits here and have kept running into 429 errors. I might actually change the order of the requests there. After that, we're good to merge IMO.

[Dominic-DallOsto]

When searching PIDs from Semantic Scholar by title, you have to send a first request to get the paper ID (a hash), and then a second one to get the details. I've not been able to figure out a good way to handle the very strict rate limits here and have kept running into 429 errors. I might actually change the order of the requests there. After that, we're good to merge IMO.

So, even when we try to get one item, these two requests in quick succession are enough to hit the rate limit?

Just noting a few things that I encounter as well:

• I'm not super happy with the button to add a new PID row - do you have any ideas on a nicer option? Do you think we could integrate it within the PID rows themselves like it is for authors?
• If I delete the text in a non-default PID row, it disappears and needs to be added back. I think it would be more intuitive if it just stayed blank and didn't disappear until changing the selected item?
• I find it a bit strange that the fetch button is shown for PID rows that already have values. I think you said this is because, even if we have an openalex id already, we can fetch it again and it could also return some other PIDs that we could fill in? But I think for this use case it would make more sense to click "fetch" on an empty PID, that could still use openalex to get the PID that we fetched?

Let me know what you think! But we can definitely hold over further features / bugs for later if they aren't critical

[thebluepotato]

Re: Semantic and rate limits, the paper search + fetch by ID requests are in quick succession, but they usually happen with adequate delay. However, the bottleneck limiter seemed to have trouble scheduling/delaying requests when the pool was empty. Basically, when it was empty (if the previous paper had been fetched) it would immediately search the next one, triggering the rate limit. I changed the settings a bit and it seems the scheduler now works correctly, but some requests still trigger the limit. I'll probably tweak it a bit by Monday.

[thebluepotato]

Re: PID rows, the better it integrates with Zotero's design, the happier I am. So we can definitely have a plus button within the rows. I also found the disappearance a bit frustrating. Maybe a delete button would make more sense there. And yes, you're right to mention the fetch/search button again. It should only be there when the field is empty.

On that note, the very latest beta of Zotero now includes an API to add custom rows to the item box. Considering that the new design of the PID rows is less intrusive than before, we might consider moving them to the item box. The details are not published yet so this might be something for a future release of Cita, but I have thought more than once that these identifiers are a bit hard to reach when there are lots of citations. Another possibility would be to move them to a distinct section, where we could have a + and refresh button in the header. Up to you.

[thebluepotato]

The PID rows can now be added and removed in a way similar to the author rows. It looks nicer than the previous button IMO, but I'm not a huge fan of the "+" button on every row. It makes sense for authors because you "insert" a new creator at this position, but PIDs don't have an order, so it's a little jarring. I'll try to see what it looks like in a separate item pane.

Regarding custom item rows, it seems to be only for basic use cases, i.e., there would be no "open URL" button or anything.

[thebluepotato]

In this latest version, the identifiers are in a separate pane, but otherwise functionality is more or less the same. There's a section button to add new PIDs, but in the future you could have other buttons there too. It would need another icon though to distinguish from the citations.

The probably last remaining bug is that the item pane seems to "remember" the scroll content height when switching items, leading to a large blank scrollable area. I'll see if I can fix it.

[thebluepotato]

Just found a new bug: when reading an attachment (such as a PDF), the citations are shown. However, the _sourceItem recorded in zoteroOverlay.tsx is not guaranteed to be the currently open reader. Rather, it could very well be the currently selected item in the list. This means that actions relying on that sourceItem (such as adding citations) will happen on the wrong item.

[thebluepotato]

I think that with these latest changes, I just need your input on the design choices and then we're good to go!

[Dominic-DallOsto]

This looks really good, thanks a lot! I think it's a good fit for Zotero's design and potentially paves the way for also displaying an item's citing items in Zotero (if we get around to doing this).

Just one tiny thing - do the brackets on the logos show up as bright white for you when Zotero is in dark mode, instead of the 0.55 alpha white that the rest of the icons appear as?

[thebluepotato]

This looks really good, thanks a lot! I think it's a good fit for Zotero's design and potentially paves the way for also displaying an item's citing items in Zotero (if we get around to doing this).

Just one tiny thing - do the brackets on the logos show up as bright white for you when Zotero is in dark mode, instead of the 0.55 alpha white that the rest of the icons appear as?

Yes, they do! By default, the brackets were black regardless of dark mode, so I tweaked the SVG so that they change with dark/light appearance. Now they're the color of the current text (fill="currentColor"), meaning black in light mode and white (or close to) in dark mode. I tried using the color of the buttons (fill="context-fill") but it was a little washed out, and the bracket corners were overlapping. I'll fix the latter and push a change with context-fill and you'll tell me what you prefer.

[Dominic-DallOsto]

Amazing thanks! Otherwise it all really looks good.

[thebluepotato]

Done! Now the icons blend in with the buttons (which means the brackets are a little paler in light mode than the full black they were using)

[Dominic-DallOsto]

Cool, I think that looks good! And any remaining issues we can iron out separately. It's a huge improvement.

Thanks so much for all your work! I'll merge this in now and make a beta release

[github-actions[bot]]

:rocket: This ticket has been resolved in v1.0.0-beta.9. See Release 1.0.0-beta.9 for release notes.