search

retorquere / zotero-better-bibtex

Make Zotero effective for us LaTeX holdouts
https://retorque.re/zotero-better-bibtex/
MIT License
5.29k stars 284 forks source link

Browser Cannot Conntect to Zotero/BBT Hangs during startup #2887

Closed despdx closed 2 months ago

despdx commented 4 months ago

For people joining this issue, you can submit a log either by going to Help => Better BibTeX Debug log, or Help => Send log to fle.io. I really need logs because I cannot yet reproduce the problem, and I need clues to the cause.

If those don't work, and you're on a Mac or Linux, there's another way to capture a debug log

On MacOS, you can get a log by starting Zotero with

/Applications/Zotero.app/Contents/MacOS/zotero -jsconsole -purgecaches -ZoteroDebugText > $HOME/BBT.txt

or on Linux

zotero -P BBTTEST -jsconsole -purgecaches -ZoteroDebugText > $HOME/BBT.txt

Let Zotero start to the point where you have reproduced the problem, then you can shut Zotero down. In both cases, there will be a file BBT.txt in your home directory. You can attach it here, or email it to emiliano.heyns@iris-advies.com`.

Another option, if you're on Linux or Mac and willing to share a full copy of your Zotero profile, I could try to reproduce the problem with that. I don't have access to a Windows machine unfortunately. And I really need a way to reproduce this. To get this copy:

zip $HOME/Zotero.zip $HOME/Zotero

and mail that to emiliano.heyns@iris-advies.com

What I know so far

There is a Zotero method BBT must call. I call it at a time that Zotero has declared it safe to call it. Yet that Zotero method hangs when I do, hanging BBT in the process. Calling it also appears to take down the connector, when these two things have nothing to do with each other. I currently have a workaround in place that helps for some people, not for others, and I cannot reproduce this problem myself, and it's pretty goddamn hard to fix a ghost bug that I cannot see, and that given where it happens, just shouldn't be able to happen.

related Zotero forums threads

https://forums.zotero.org/discussion/comment/463327#Comment_463327 https://forums.zotero.org/discussion/comment/463326#Comment_463326 https://forums.zotero.org/discussion/comment/463324#Comment_463324

retorquere commented 4 months ago

For people joining this issue, you can submit a log either by going to Help => Better BibTeX Debug log, or Help => Send log to fle.io. I really need logs because I cannot yet reproduce the problem, and I need clues to the cause.

If those don't work, and you're on a Mac or Linux, there's another way to capture a debug log (see below).

On MacOS, you can get a log by starting Zotero with

/Applications/Zotero.app/Contents/MacOS/zotero -jsconsole -purgecaches -ZoteroDebugText > $HOME/BBT.txt

or on Linux

zotero -P BBTTEST -jsconsole -purgecaches -ZoteroDebugText > $HOME/BBT.txt

Let Zotero start to the point where you have reproduced the problem, then you can shut Zotero down. In both cases, there will be a file BBT.txt in your home directory. You can attach it here, or email it to emiliano.heyns@iris-advies.com`.

Another option, if you're on Linux or Mac and willing to share a full copy of your Zotero profile, I could try to reproduce the problem with that. I don't have access to a Windows machine unfortunately. And I really need a way to reproduce this. To get this copy:

zip $HOME/Zotero.zip $HOME/Zotero

and mail that to emiliano.heyns@iris-advies.com

What I know so far

There is a Zotero method BBT must call. I call it at a time that Zotero has declared it safe to call it. Yet that Zotero method hangs when I do, hanging BBT in the process. Calling it also appears to take down the connector, when these two things have nothing to do with each other. I currently have a workaround in place that helps for some people, not for others, and I cannot reproduce this problem myself, and it's pretty goddamn hard to fix a ghost bug that I cannot see, and that given where it happens, just shouldn't be able to happen.

related Zotero forums threads

https://forums.zotero.org/discussion/comment/463327#Comment_463327 https://forums.zotero.org/discussion/comment/463326#Comment_463326 https://forums.zotero.org/discussion/comment/463324#Comment_463324

github-actions[bot] commented 4 months ago

Debug log ID

4TUBZVQC-refs-euc/6.7.192-6

What happened?

A couple days ago, I couldn't import references via my zotero browser extension. It was working just a couple weeks before. I disabled, enabled, and upgraded BBT a couple times, and it worked. You asked me to try version ...190, which worked, but I tried ...192, too, which reproduced the problem. So, I'm here reporting it.

Naturally, it worked this time, but I only tried 192. Next, I tried to recreate my procedure more carefully: Install 190, disable it, restart, install 192 without removing 190, enable the plugin, restart --> bingo.

image

It's not too hard to test, so let me know if I can be of more help. But, I'm going to upgrade in the meantime.

despdx commented 4 months ago

Yeah, my reproducing procedure suggests more to do with the upgrade process than a bug in a particular addon version. I already upgraded to 200. So far so good.

retorquere commented 4 months ago

Can you upgrade to .201 and see if the connector problem is still fixed for you?

lgsmith commented 4 months ago

I'm also having this issue, and my 'add-on' window in the zotero desktop app appears to say I'm on .201. My desktop app's version is 6.0.35. My zotero connector's version is 5.0.123. Disabling the better bibtex extension does restore the zotero connector's function. I'm using firefox for kubuntu 22.04 LTS, version 122.0.

What other information can I provide to help?

retorquere commented 4 months ago

Oh man I hope a log will say something here. Does debug log submission work? Is this reliably reproducible for you?

lgsmith commented 4 months ago

I don't know about that. How do I do it? I can attach a log if you tell me where to find it on my filesystem...

despdx commented 4 months ago

Oh, very interesting. It's broken again after the update to 201. I went to Tools > Add-ons, and BBT was showing an update was available, so I clicked on "restart". Now, I'm getting the same error as before about connecting not working. I see a suck spinner icon next to the words "better bibtex: translators" in the right pane of Zotero desktop app.

despdx commented 4 months ago

I tried to disable BBT and restart. But I got this on the command line:

[18113, Main Thread] ###!!! ABORT: file resource://gre/modules/Sqlite.jsm, line 148
[18113, Main Thread] ###!!! ABORT: file resource://gre/modules/Sqlite.jsm, line 148
/home/<readaced>/local/Zotero_linux-x86_64/zotero: line 13: 18113 Segmentation fault      (core dumped) "$CALLDIR/zotero-bin" -app "$CALLDIR/application.ini" "$@"
despdx commented 4 months ago

Okay, I started Zotero again, enabled the plugin and clicked "restart". It started up and finished initializing BBT. It works, again.

retorquere commented 4 months ago

I don't know about that. How do I do it? I can attach a log if you tell me where to find it on my filesystem...

Help menu, Better bibtex debug log

retorquere commented 4 months ago

I see a suck spinner icon next to the words "better bibtex: translators" in the right pane of Zotero desktop app.

If the aforementioned debug log doesn't work, there will be an entry "send to file.io" in the Help menu. Please send that. Wonder what's broken now. I must call a method in Zotero before start, and it just hangs sometimes since recently.

retorquere commented 4 months ago

/home//local/Zotero_linux-x86_64/zotero: line 13: 18113 Segmentation fault (core dumped) "$CALLDIR/zotero-bin" -app "$CALLDIR/application.ini" "$@"

A core dump is always a Zotero problem. I'll add it to the report I filed with the Zotero crew.

retorquere commented 4 months ago

Okay, I started Zotero again, enabled the plugin and clicked "restart". It started up and finished initializing BBT. It works, again.

If you do see any of these problems again, please try one of the debug log methods. I am traveling tomorrow so I may not always respond immediately.

lgsmith commented 4 months ago

When I restarted to activate logging I stopped seeing this issue. I'm not sure what 'moved' underneath me, but I seem to have something that works now on my laptop.

On my workstation for work, the issue is still present even with the .201 upgrade. In fact it's behaving how my laptop was behaving yesterday. In any case, I managed to send this debug report from my workstation (but not sure that it has anything useful in it): D790041850

retorquere commented 4 months ago

Man I had people for which the start issue and the connector issue were linked and I had hoped I had killed two birds with one stone... does the better bibtex debug log or the file.io log from the help menu work? I can forward D<number> log id's to Zotero but I can't access the logs behind them. The logs I can access will have euc, apse or fio in the ID.

despdx commented 4 months ago

... does the better bibtex debug log or the file.io log from the help menu work? ...

I don't think so. I clicked on it, and nothing happened. But, I don't know if that is correct behavior or not.

despdx commented 4 months ago

Oh, interesting. It's broken again. Maybe that's why the "Send debug log" didn't work.

despdx commented 4 months ago

Hmm, looks like there was another update. It was automatic; I didn't do anything to initiate the update. I'm at .202 now.

It's petty clear that the bug I'm reporting is that BBT hangs or crashes on start up, and Zotero doesn't notice. It shows up as enabled in the Add-ons window, but, in the right pane, I again see the stuck spinner icon and the words "better bibtex: translators".

Perhaps I should change the name of this issue to something like "BBT Hangs after Update, No Logged Errors, Breaks Connector, Zotero Works Fine Otherwise".

retorquere commented 4 months ago

The top message describes a way to send a log that will always work on Linux or Mac. I've moved the messages around so newcomers know the current state from the top post. I'm using this issue now to track the problem for all those that are experiencing it. If you have no privacy-sensitive stuff in your zotero library, I'd love to get a copy, in the hopes that I can reproduce the problem with that. I haven't been able so far.

despdx commented 4 months ago

Woah woah. I'm on your side! I'm trying to understand the problem, too, so I just realized that BBT could hang without causing all of Zotero hang; I didn't know that could happen. I wasn't complaining or suggesting you do anything.

I have the STDOUT for zotero, and I see the same thing there as I previously reported. But, I changed my startup procedure to include the debug version, so, if it happens again, I'll be able to get you a nice log of the whole session. I quit and used those options to restart zotero and didn't have any problem this time. We're poised for the next step. Evidence suggests it will happen after the next upgrade.

retorquere commented 4 months ago

Yeah, I apologize for my earlier rant. It's been stressful but that's no reason to take it out on others.

retorquere commented 4 months ago

So it's no longer reproducible? 😭

despdx commented 4 months ago

No problem. I'm sorry I'm not able to provide more insight.

Well, I cannot reproduce it by doing anything specific...oh wait, sorry, I could downgrade and upgrade again, right? Okay, let me try...hmm, no I cannot reproduce it myself. I tried removing BBT and installing .190, then installing .192 over that, then I tried upgrading .192 to .202 via manual "check for updates and that didn't work. And I did it again, too. I couldn't get it to break, again. According to my old notes, this process reproduced the problem in the past, so I'm confused again.

retorquere commented 4 months ago

transient errors are the worst

retorquere commented 4 months ago

If anyone hits the error again, please also try an earlier Zotero beta.

LambdaP commented 4 months ago

I have been pretty consistently unable to get papers using the Zotero Connector extension lately. Tried restarting everything etc.

I started Zotero using the diagnostic command line in the first comment in this thread, then tried using the Connector, and it worked fine. Restarted Zotero normally, still worked fine.

Aren’t computers marvelous.

retorquere commented 4 months ago

Gaah, these transient errors! Are you on Linux or macos by any chance.

despdx commented 4 months ago

Linux

On Wed, May 29, 2024, 10:44 Emiliano Heyns @.***> wrote:

Gaah, these transient errors! Are you on Linux or macos by any chance.

— Reply to this email directly, view it on GitHub https://github.com/retorquere/zotero-better-bibtex/issues/2887#issuecomment-2137947640, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBZWI654DIGTSST3ODQBDLZEYHWNAVCNFSM6AAAAABIBAVYM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZXHE2DONRUGA . You are receiving this because you authored the thread.Message ID: @.***>

retorquere commented 4 months ago

Would you be OK with sharing your zotero profile (which, to be clear, would contain all your Zotero data) privately? I have to find a way to replicate the problem, and with a profile that at least sometimes has the problem, I could just hammer that profile. I have a private Linux box that I could use for tests, this wouldn't leave my personal systems, not shared with anyone, not sent to github. Are you on zotero 6 or 7?

github-actions[bot] commented 4 months ago

:robot: this is your friendly neighborhood build bot announcing test build 6.7.202.2887.6349 ("put wait for translators in start")

This update may name other issues, but the build just dropped here is for you; it just means problems already fixed in other issues have been folded into the work we are doing here. Install in Zotero by downloading test build 6.7.202.2887.6349, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

retorquere commented 4 months ago

Can someone please try build 6349?

retorquere commented 4 months ago

I've rolled this into a new release. I'd appreciate feedback on whether release .203+ structurally solves the issue.

mjthoraval commented 4 months ago

It is working nicely for me so far on Windows 10 and Ubuntu 24.04 LTS, with Zotero 7.0.0-beta.83+066eda731 (64-bit) and BBT 6.7.203. I have tried torturing it for some time, but it has always recovered. I will add an update here if the problem comes back.

I can see this popup message between the loading of the header of the Item Pane and the display of the "Citation key" column: image

Then the column would appear correctly. When I was observing the bug, the header of the Item Pane would not load until I would change item in the Items Tree and the "Citation key" column would never appear.

The closest torturing steps I could get were:

-> During that time, and for a few minutes after, it looked like the Citation key column would not load. But after waiting a few minutes, it still came back and worked properly. So if you keep Zotero busy, you can temporarily prevent BBT from loading.

I just got this error a few times during the recovery time:

[JavaScript Error: "TypeError: item is undefined" {file: "jar:file:///C:/Users/mjthoraval/AppData/Roaming/Zotero/Zotero/Profiles/4o6yeo4t.default/extensions/better-bibtex@iris-advies.com.xpi!/content/better-bibtex.js" line: 144486}]

I have tried also installing an old beta (Zotero-7.0.0-beta.53) and then let Zotero auto-update. BBT still came back working after the update.

So I have tried my best to get the bug back, but BBT seems to be very robust.

Thanks!

retorquere commented 4 months ago

It looks like there's now two signals I need to await before I can start BBT, unfortunately zotero takes 5-8 seconds to clear the 2nd. After that BBT starts in 3 seconds max, usually less, but until that 2nd signal clears, BBT sits there doing nothing, and the UI update comes last, so after some 6-12 seconds in total.

In principle it should be safe to start BBT after the 1st signal clears, which is what I've always done, as there's just one particular part of BBT that needs the 2nd signal to clear, so I've always let BBT start in parallel, but apparently there's a race condition to a deadlock now. Serializing everything was honestly just a shot in the dark, but I'm happy it works.

retorquere commented 4 months ago

Minutes is pretty long though. If that's repeatable I'd love a log. But if that time is just spent waiting for the 2nd signal (the log has information on whether this is the case), there isn't anything I can do about that.

mjthoraval commented 4 months ago

Here are the details. I have followed the same steps:

After releasing the keys, I waited some time, during which Zotero was still very busy. Then this appeared: image

I pressed Continue.

It took 2 minutes 45 seconds after releasing the keys to get the header to appear in the Item Pane. Then another 2 minutes 40 seconds to get the Citation key column appear, so 5 minutes 25 in total after releasing the keys.

Here is the Debug log ID: H7QPQ4VH-euc/6.7.203-7

mjthoraval commented 4 months ago

I have tried also pressing Stop script. The Citation key column still appears after waiting for 7 minutes.

retorquere commented 4 months ago

Here is the Debug log ID: H7QPQ4VH-euc/6.7.203-7

Yeah that shows what I already suspected:

orchestrator: start took 00:07:59.336

which is insanely long but that is the time Zotero takes to clear the 2nd signal on your system. I'll see whether the Zotero crew wants to dive into this, but between "mysterious breakage of core Zotero functionality" and "BBT takes forever to start up", I'm picking the latter.

retorquere commented 4 months ago

Still -- the BBT database init independently takes over 2 minutes, which is ridiculous. A new build is incoming, please send a new log.

github-actions[bot] commented 4 months ago

:robot: this is your friendly neighborhood build bot announcing test build 6.7.203.2887.6364 ("log DDL times")

This update may name other issues, but the build just dropped here is for you; it just means problems already fixed in other issues have been folded into the work we are doing here. Install in Zotero by downloading test build 6.7.203.2887.6364, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

despdx commented 4 months ago

Would you be OK with sharing your zotero profile (which, to be clear, would contain all your Zotero data) privately? I have to find a way to replicate the problem, and with a profile that at least sometimes has the problem, I could just hammer that profile. I have a private Linux box that I could use for tests, this wouldn't leave my personal systems, not shared with anyone, not sent to github. Are you on zotero 6 or 7?

Sure, that would fine. How can I send it to you? What do I send?

mjthoraval commented 4 months ago

I got similar results with the test build 6.7.203.2887.6364.

-> So 4 minutes 25 seconds in total to load the Citation key column. Debug log ID: WQPIDRRP-euc/6.7.203.2887.6364-7

But to be fair here, the test protocol is a bit extreme, and Zotero itself does not react very well to it. So BBT is probably not to blame here.

For comparison, here are the times if I do not press Ctrl+Page Down for 5 minutes after starting Zotero:

-> So 29 seconds in total to load the Citation key column. Debug log ID: VHY9RXTS-euc/6.7.203.2887.6364-7

This is in line with what I had reported to Zotero before: (see this post) . And the PDF Reader does not seem to handle very well a continuous flow of actions in recent versions (see this post), especially in Troubleshooting Mode.

retorquere commented 4 months ago

These builds add only logging, so they won't be any faster than they were. A new build is incoming, I'd appreciate a new log. The 2.5 minutes wasn't in the part where I added logging.

github-actions[bot] commented 4 months ago

:robot: this is your friendly neighborhood build bot announcing test build 6.7.203.2887.6370 ("more logging")

This update may name other issues, but the build just dropped here is for you; it just means problems already fixed in other issues have been folded into the work we are doing here. Install in Zotero by downloading test build 6.7.203.2887.6370, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

mjthoraval commented 4 months ago

Here it is: NEBZAWGL-euc/6.7.203.2887.6370-7

-> So 3 minutes 5 seconds in total to load the Citation key column.

github-actions[bot] commented 4 months ago

:robot: this is your friendly neighborhood build bot announcing test build 6.7.203.2887.6378 ("Merge branch 'master' into gh-2887")

This update may name other issues, but the build just dropped here is for you; it just means problems already fixed in other issues have been folded into the work we are doing here. Install in Zotero by downloading test build 6.7.203.2887.6378, opening the Zotero "Tools" menu, selecting "Add-ons", open the gear menu in the top right, and select "Install Add-on From File...".

mjthoraval commented 4 months ago

The first test has nearly killed Zotero: image

image

Wait, just recovered after 12 minutes!

A new error message at the end: image

After clicking on "Check for Update", I get a mysterious window: image I cannot expand it.

After clicking on the cross to close the tiny window, it closed Zotero completely... So I did not get the debug log...

retorquere commented 4 months ago

But can you do a plain start and post a log? I've added more timing logging. Something really strange is going on -- there are large chunks of time (over a minute in the last log) where BBT does not appear to doing anything but there isn't anything else in the logs.

mjthoraval commented 4 months ago

The second try was successful: QE2WV8GQ-euc/6.7.203.2887.6378-7

-> So 6 minutes 11 seconds in total to load the Citation key column.

I guess it was getting longer due to the additional logs, and opening the Debug Output window was the thing that killed Zotero...

And a log for a normal restart without doing anything during the restart: RUY8E96J-euc/6.7.203.2887.6378-7