search

BlafKing / sd-civitai-browser-plus

Extension to access CivitAI via WebUI: download, delete, scan for updates, list installed models, assign tags, and boost downloads with multi-threading.
GNU Affero General Public License v3.0
289 stars 44 forks source link

[Bug]: Extension silently overwrites models with the same filename. #288

Open NonaSuomi opened 2 months ago

NonaSuomi commented 2 months ago

Describe the bug.

When attempting to download, if a file with the same name already exists in the designated folder, the extension will simply overwrite the pre-existing file. Given that it is particularly common for creators to just use "artistname.ext" with style LoRA's, for example, this can cause significant problems.

Steps to reproduce the problem.

  1. Have a file in the folder that Browser+ will save to
  2. Download a file with the same filename

Expected behavior

Prompt the user for how to handle the file conflict and/or rename the new file being downloaded. Ideally, it could prepend something unique such as the the the model's hash, or the ID portion of the model's AIR onto the downloaded filename.

System info

Console logs

N/A

Additional information

No response

BlafKing commented 2 months ago

Thanks for reporting this issue!

I'm not exactly sure how this could occur, since the extension checks if each model from the search results is installed by doing the following:

  1. It first looks for a saved hash, which gets saved if you download a model with this extension so that it can reliably detect it as being installed later.
  2. If a hash has not been found, it then tries to find a matching filename compared to the filename of the selected model.

So, when you attempt to download a model with the same filename as another model, it should show up as already being installed and because of that it should not be downloadable.

Could you please clarify in more detail which model(s) you had saved and which you tried to download? as I'm not completely sure how this issue could occur.

Regardless, I'll add an extra check to ensure that a model does not get overwritten. I'll append something unique to the filename, as you suggested. 👍🏻

NonaSuomi commented 2 months ago

Okay, so I've done a little more poking around and have found that there's two presumably-related issues that I'm running into here:

To begin with, some of the checkpoints, LoRAs, etc. I have in my models directory are from sites other than CivitAI. When I installed the extension I first went to the settings and input my API key, then I went to the extension tab and the "update models" subtab, checked the "All" box and hit "Load models" to let it scan my files to match what it could, then ran "update info & tags" followed by "update model preview" operations. From there I went to the Browser subtab.

To begin with, I checked the console output for a model that it couldn't find a match for, and tried searching for a CivitAI model of the same artist- in this case Afrobull. The model I had already can be found on PixAI as model 1619930580734245232. It is 72MB and has a CRC32 of 3DFADF71. The version I downloaded with the extension was CivitAI model 6285, which is 72MB and has a CRC32 of 56D140A2. Something that might be worth noting here is that for some reason the hashes that CivitAI displays do not match what I get when checking the downloaded files in either 7zip or in hashtab. Regardless, from here things proceeded as my original post- I told the extension to download model 6285, and when I headed back to txt2img, I noticed that my old LoRA was missing, and found that the file had been overwritten.

The next issue, which I assume is probably related, is that if a model was downloaded directly from within the extension itself, it will show any other model with a matching filename as installed as well. You can see this using the same model above (6285). It uses the filename "Afrobull.safetensors" which is the exact same filename used by model (442032). If you download one through the extension, the other one will then also marked as [Installed] in the version field, and you won't have the option to download the second model without hitting "Delete model" first.

Another odd thing, and this appears to get to the heart of it, but this filename-matching appears to be case-sensitive, which can cause the extension to overwrite its own downloads. Again, this can be seen using the Afrobull models as an example- model (348816) has two versions, and v1.0 uses the filename "afrobull.safetensors" note the lower-case 'a'. If either model (6285) or (442032) has already been installed, model (348816) will not be detected as installed, but when downloaded it will overwrite the preexisting model file.

Given this info, it seems that the extension only looked for a case-sensitive filename match, and when the file I already had was a case-INsensitive match, it was overwritten. I'm not sure if this is perhaps due to some misconfiguration on my part, but on my install at least it doesn't appear to be doing any kind of hashing at the point of model download/installation.

onyxlee commented 1 month ago

We need a different identifier for each model. The original file name, the lora name can all be duplicated. Now, when the file with the same name exists, ALL the other loras with the same name all appear to be installed. This issue has been there for a very very long time. Hope you can fix this. Thank you.