search

BlafKing / sd-civitai-browser-plus

Extension to access CivitAI via WebUI: download, delete, scan for updates, list installed models, assign tags, and boost downloads with multi-threading.
GNU Affero General Public License v3.0
289 stars 44 forks source link

[Bug]: Several models not updated preview #289

Closed silveroxides closed 2 months ago

silveroxides commented 2 months ago

Describe the bug.

Multiple models fail preview update due to one or more(but not all) in a chunk call to api resulting in no response from api.

Steps to reproduce the problem.

  1. start updating preview with models that does not exist on site but did previously

Expected behavior

I would have expected the extension to self correct for such situations by partitioning the chunk of multiple models called into smaller ones until there only is one upon which it throws an error and prints to console and writes error loig file.

System info

Console logs

CivitAI Browser+: Request timed out for https://civitai.com/api/v1/models?ids=100000&ids=100435&ids=100561&ids=100935&ids=101149&ids=101233&ids=101312&ids=102575&ids=102579&ids=103467&ids=103473&ids=103599&ids=10364&ids=103666&ids=103696&ids=103724&ids=103848&ids=104260&ids=105053&ids=105108&ids=105487&ids=105670&ids=105796&ids=105813&ids=105976&ids=105987&ids=106004&ids=106186&ids=10676&ids=107313&ids=109262&ids=109745&ids=109818&ids=109892&ids=110917&ids=111237&ids=11126&ids=111918&ids=111955&ids=111962&ids=111986&ids=112064&ids=112198&ids=112299&ids=112594&ids=112619&ids=112669&ids=112750&ids=113064&ids=113427&ids=113542&ids=113823&ids=113916&ids=114051&ids=114487&ids=114949&ids=116197&ids=116407&ids=116420&ids=117393&ids=117603&ids=118126&ids=118153&ids=119026&ids=119032&ids=119201&ids=119388&ids=119399&ids=119427&ids=119494&ids=119892&ids=119899&ids=120464&ids=120829&ids=121027&ids=121094&ids=121303&ids=121567&ids=121777&ids=121801&ids=122183&ids=122200&ids=122279&ids=122316&ids=122518&ids=122539&ids=123687&ids=123714&ids=123950&ids=124023&ids=124347&ids=124436&ids=124701&ids=124731&ids=124797&ids=125187&ids=125794&ids=126551&ids=126596&ids=127526&ids=127533&ids=127548&ids=127854&ids=128272&ids=12859&ids=128836&ids=130075&ids=130866&ids=131143&ids=13117&ids=132407&ids=132454&ids=132774&ids=133554&ids=133622&ids=133658&ids=135936&ids=137409&ids=137511&ids=137711&ids=138025&ids=138087&ids=138917&ids=138919&ids=139061&ids=139157&ids=139203&ids=13941&ids=139460&ids=139625&ids=139889&ids=140376&ids=14153&ids=142014&ids=142153&ids=142411&ids=143592&ids=143636&ids=143637&ids=144089&ids=144828&ids=146315&ids=146348&ids=146662&ids=14674&ids=147279&ids=147556&ids=148465&ids=149000&ids=150021&ids=15132&ids=151594&ids=151894&ids=152877&ids=152986&ids=15365&ids=154142&ids=154589&ids=155051&ids=155165&ids=155176&ids=155566&ids=155768&ids=156891&ids=158249&ids=160073&ids=161580&ids=161593&ids=164142&ids=165069&ids=16651&ids=16775&ids=168174&ids=17302&ids=17391&ids=174566&ids=174930&ids=17504&ids=175513&ids=17739&ids=17875&ids=18299&ids=18639&ids=18671&ids=19006&ids=195519&ids=198359&ids=200140&ids=200937&ids=200990&ids=20124&ids=202595&ids=203651&ids=203701&ids=203707&ids=20720&ids=20736&ids=211356&ids=214442&ids=215373&ids=21728&ids=217477&ids=219166&ids=221845&ids=22231&ids=22256&ids=22437&ids=228016&ids=22889&ids=22989&ids=23699&ids=238419&ids=23885&ids=239851&ids=242825&ids=24380&ids=245198&ids=24542&ids=246523&ids=248856&ids=25396&ids=263195&ids=264290&ids=265314&ids=26642&ids=268007&ids=26806&ids=27079&ids=27317&ids=275796&ids=276675&ids=28212&ids=28248&ids=285908&ids=286044&ids=287556&ids=288949&ids=291748&ids=295019&ids=295100&ids=296700&ids=297027&ids=299146&ids=30020&ids=302870&ids=303683&ids=30575&ids=305832&ids=306441&ids=310512&ids=310964&ids=31197&ids=315703&ids=31618&ids=317171&ids=318850&ids=320203&ids=320780&ids=32117&ids=32395&ids=32420&ids=32510&ids=326320&ids=326799&ids=326901&ids=328567&ids=330184&ids=330816&ids=331812&ids=332133&ids=332681&ids=33314&ids=333386&ids=33504&ids=338068&ids=339129&ids=339413&ids=33964&ids=339690&ids=341867&ids=345739&ids=345909&ids=346158&ids=348070&ids=349220&ids=350450&ids=353150&ids=353214&ids=35930&ids=360567&ids=361174&ids=361869&ids=362011&ids=362788&ids=365106&ids=365320&ids=365756&ids=36861&ids=371783&ids=371867&ids=374601&ids=374620&ids=374787&ids=375210&ids=378306&ids=378428&ids=380458&ids=381045&ids=381829&ids=382344&ids=383758&ids=383936&ids=384754&ids=385236&ids=38551&ids=385974&ids=388308&ids=388810&ids=388887&ids=389465&ids=389677&ids=390201&ids=390732&ids=392111&ids=39300&ids=393047&ids=393601&ids=39424&ids=402412&ids=402462&ids=407627&ids=409043&ids=409737&ids=410755&ids=41791&ids=421662&ids=42214&ids=423853&ids=423988&ids=42412&ids=424138&ids=424144&ids=424163&ids=424593&ids=430939&ids=432778&ids=433116&ids=436857&ids=437039&ids=438354&ids=4391&ids=439171&ids=439645&ids=440382&ids=441057&ids=44276&ids=444650&ids=444691&ids=44530&ids=44638&ids=447020&ids=44833&ids=449775&ids=450852&ids=453813&ids=454173&ids=45597&ids=458527&ids=46278&ids=46491&ids=466467&ids=46655&ids=466696&ids=467190&ids=468146&ids=468265&ids=471536&ids=471694&ids=475679&ids=475764&ids=4784&ids=48979&ids=49008&ids=49044&ids=51221&ids=51345&ids=52023&ids=53614&ids=54925&ids=54934&ids=55054&ids=5634&ids=56374&ids=57944&ids=58012&ids=58059&ids=58912&ids=5919&ids=59574&ids=59639&ids=60284&ids=60472&ids=60648&ids=60732&ids=61316&ids=62379&ids=63378&ids=63556&ids=63886&ids=65040&ids=65603&ids=65894&ids=6619&ids=67829&ids=68624&ids=68719&ids=69652&ids=70282&ids=70387&ids=7076&ids=71953&ids=72451&ids=73412&ids=73756&ids=73782&ids=74593&ids=7468&ids=75127&ids=75165&ids=75265&ids=76006&ids=77203&ids=77579&ids=77890&ids=78090&ids=78241&ids=78897&ids=79125&ids=79274&ids=80113&ids=81079&ids=81163&ids=81353&ids=81597&ids=8179&ids=8217&ids=82219&ids=82630&ids=82926&ids=8328&ids=84146&ids=84148&ids=84319&ids=84382&ids=84600&ids=84778&ids=84938&ids=8679&ids=8730&ids=87378&ids=87528&ids=87575&ids=87584&ids=87901&ids=88402&ids=88479&ids=88581&ids=89030&ids=89325&ids=89437&ids=89544&ids=89925&ids=90316&ids=90495&ids=90687&ids=90738&ids=90758&ids=90903&ids=91032&ids=91969&ids=92022&ids=92557&ids=92704&ids=92706&ids=92936&ids=93399&ids=94239&ids=94351&ids=94951&ids=9502&ids=95035&ids=95284&ids=95802&ids=96002&ids=96490&ids=9651&ids=97167&ids=98054&ids=99911&ids=99930&limit=100&nsfw=true&cursor=865%7C7937%7C92557. Skipping...

Additional information

Also you do not have to print to console every single time a model has or has not been found. You can simply assign it to a variable which you then concatenate upon and print the entire thing once at the end. Every print to log is unnecessary especially if user wants to use their webui while it downloads.

BlafKing commented 2 months ago

Not really a bug imo. You're reporting a timeout error, which is really just something caused by your connection or by the CivitAI servers.

When you include a model ID in the API request that no longer exists, then the API request still works fine and automatically excludes the missing model, partitioning the chunked models into smaller ones won't resolve that issue.

There might currently not be a retry function in place that retries when a request times out, which might help solve this issue so I'll look into that.

About the model not found prints being obnoxious, feel free to turn them off in the settings, or if you'd like to see different behavior regarding the prints, feel free to make a pull request as I personally like the current behavior of prints.

Closed for not being a bug.

silveroxides commented 2 months ago

I am reporting a timeout error that happens at the same point every time which results in all of the other models (brand spanking new ones) not getting updated. So yes it IS a bug. Or why would it miss several newly downloaded model. Note that I download them externally from Civitai and not from the extension download tab since I find that too clunky to use effiecently and since it does not support tabbing(not that it could) for looking at models later. I usually scroll through about 50 models and open a bunch in tabs which I go through in batches.

But after going through your code I can see why some things work as they do. Rare to see that many 'if' statements nowadays.

BlafKing commented 2 months ago

You're free to not use this extension if you don't like it :,) It's my first python project, so yeah, the code is an absolute mess 🧠

Tried the URL you mentioned in the bug report, which is working fine for me which is why I assumed the extension to not inherently be at fault, of course the other behavior when it fails is far from ideal. I'll try to experiment some more, but I assume the solution will be increasing the timeout time or retrying the URL if it times out instead of moving on after it fails once, which I'll implement if I have some free time to work on this extension again.

silveroxides commented 2 months ago

image Is this the option you meant. Cause that was off. The only other one I see is the one that disables model not found but that is in the browser options section. I get the timeout at API request 6 of 14. My max downloads was set to 36. Changed to 32 and it worked. So something about that specific batch was the issue and looking at it, it might be some edge case of not being able to handle the modelid of the first one or something.

EDIT: Yup all previews updated. Either it was the thing I wrote above about edge case or I was rate limited by civitai.

BlafKing commented 2 months ago

Good to know, thanks for the extra info.

I was indeed referring to the "model not found print" option in the browser options location, still trying to guess which setting should go in which category, so not exactly sure if it's current place is intuitive😅

Also the Number of connections to use for downloading a model option only applies to Aria2, which is only used for downloading the actual model itself, not for preview updates, so afaik changing that setting shouldn't affect the updating of any previews.

I'm assuming it might be a rate limit by CivitAI, or something else to do with CivitAI their servers taking longer than usual, so I'll implement some more checks regarding unresponsive servers, so that it either retries or waits a bit longer for a response 👍🏻