De-ACSM got overwhelmed with multiple ACSMs then deadlocked

[p1nkyy]

Bug description

Several books went through fine but the rest suddenly didn't. The bar for processing stuck. I aborted and tried again to no avail. When the ACSM is processed in other device, it's said it's already fulfilled. Processed fine in ADE.

Calibre error.txt

Operating system

Windows

Which version of Calibre are you running?

6.11

Which version of the ACSM Input plugin are you running?

v0.0.17

Import type

Dragging-and-Dropping the ACSM file into the Calibre window

Further information

Version I use: https://github.com/Leseratte10/acsm-calibre-plugin/actions/runs/3689432064

[Leseratte10]

Interesting bug ...

• Did you add one ACSM at a time or did you drag-and-drop a lot of ACSMs into Calibre at once? If so, how many? (approximately).
• If it's only one, did you try to restart Calibre and/or Windows?
• Can you still fulfill the Adobe example books or is the plugin now completely stuck even after restarting Calibre?

Calibre is designed to process all imported book files in parallel batches, not strictly sequencial. For ACSM files that's not a good idea since ADE doesn't do that either and it can confuse the Adobe servers, so I added a bunch of code to the plugin that tries to make sure only one file is fulfilled at once even though Calibre executes the plugin multiple times in parallel.

Looking at the logfile, that code seems to not have worked properly and the plugin is "stuck" waiting for other, nonexistant plugin executions to finish.

[p1nkyy]

First I add many at once. It stopped working at 4/7 or something. So some files did go through. Then I aborted. Started again with a lot (3 or 4) but different files. No progress. Then I added 1 only. No progress. I restarted with debug mode & process just 1 to get the error file. No progress. I restarted normally & process just 1 file. No progress.

Up until now still can't use ACSM Input ☹️ please help 😭 Oh, haven't tried to restart the PC yet though.

[Leseratte10]

Looks like it softlocked completely, then.

In Calibre, go to Preferences -> Advanced -> Miscellaneous -> Open calibre configuration folder -> plugins. Once you have that folder open, close Calibre.

Then check the files "deacsm.json" and "ACSMInput/ACSMInput.json" (they may both exist, or only one of them might exist). Open the JSON file(s) in a text editor and set the "loan_identifier_token" to 0 in both files (if it exists).

Then save the file and restart Calibre and the plugin should work again. I'll try to figure out A) what caused that and B) how I can prevent it.

[p1nkyy]

OK done. ACSMInput/ACSMInput.json doesn't have "loan_identifier_token" in it so I changed just the other one. Thank you for this!

EDIT: Yes it works now.

[p1nkyy]

Hi, I can't open Show Loaned List now.

This is what I got from Copy to Clipboard

ACSM error.txt

[Leseratte10]

Whoops, missed that comment. Can you open the JSON file again and post the whole contents of the loaned list?

[Leseratte10]

Thanks. Can you test the most recent version a9519fbd20026fad3ac3a55965717951490ebabf , this issue should be fixed with that.

https://github.com/Leseratte10/acsm-calibre-plugin/suites/11091879230/artifacts/564641561

[p1nkyy]

Perfect, thanks. Also lately the plugin takes forever to load a book (even when processing one by one and not many at once), especially if the same title has already been in Calibre (UK vs US version of a novel, for example). Do you have any idea why? And can it be fixed?

[Leseratte10]

Not sure why, only thing I could think of is the server being slow.

If you run Calibre in debug mode (calibre-debug -g on the console) you'll see the log "live" as it's being generated; maybe you can check if it's stuck on one particular step for a long time?

[p1nkyy]

OK, I will try that next time and send you the log again. Do I need to enable the Verbose Log in the plugin?

[p1nkyy]

I chose not to enable 'parallel fulfillment' in the hope of not encountering this error again but today it crashed again. I have since reset the .json file and made the plug in works.

ACSM error again.txt

[p1nkyy]

Hi, is there any update or solution for this? At the moment I've resorted to manually processing it one by one but it's distracting and takes time when time is of essence. I'd appreciate if you can help. Thanks.

[Leseratte10]

I'm not sure if there is a good solution for the issue. Calibre is designed in a way to be able to run multiple plugins at once when importing multiple books, but when a plugin needs to A) contact a server and B) write its own logs like the loan records that can get messy.

I'll try asking Kovid Goyal if he can add an API in Calibre or something to force sequencial operation for a given plugin.

[p1nkyy]

Thanks for responding. After a while and reading issue #46 I got an idea and tried that method instead. So now my Calibre will auto process ACSMs I put in a certain folder. It works well so far (2 weeks in) and hasn't deadlocked again.

[Leseratte10]

Okay, with the new version I've just committed this should hopefully be completely fixed: 999354dde9963a2c5685e936718a1776c9bd7a38

A workaround for the old plugin versions (if necessary) should be to go to Preferences -> Advanced -> Miscellaneous and set "Max. simultaneous ... jobs" to 1 (default: 3). But with the new update this should no longer be necessary.

[p1nkyy]

Wow, thanks so much. Yes indeed I changed the max simultaneous jobs to 1 after getting desperate and ever since that things has been perfect. I will try the newest version.

[p1nkyy]

Hi, one problem with the newest version, 0.1.0, none of the books I borrowed are registered in the Loaned Book List, therefore I can't return from there 😨

[Leseratte10]

Did you not already use one of the beta version(s) before? So the beta shows your existing authorization, but doesn't show the loaned books? Interesting. If you now loan a new book with the new version, does that show up?

[p1nkyy]

Did you not already use one of the beta version(s) before? So the beta shows your existing authorization, but doesn't show the loaned books? Interesting. If you now loan a new book with the new version, does that show up?

Yeah, I've been using beta ver for a while. Previously it was this one: https://github.com/Leseratte10/acsm-calibre-plugin/actions/runs/4262742735 current ver I think this one: https://github.com/Leseratte10/acsm-calibre-plugin/actions/runs/4902419270

Nothing showed up in the loaned list, everything that was left there was from previous version. This current version doesn't generate any at all. I tried processing the automatic way and manual way (dragging them to the Calibre), no luck. Books got processed though.

Do I need to provide something, maybe?

[Leseratte10]

Oops. Should now be fixed with 53c1d762246b601aa9706f94c1265910381e5e5c

[p1nkyy]

OK everything looks beautiful at the moment, been testing it with 5-7 acsms in one batch for several instances, trying returning a book, etc. I will close this for the time being. I truly hope no more issues will arise. Kudos to you, you're an awesome and responsive author/developer. Thanks for creating this very useful plugin.