search

civitai / civitai-link-desktop

Manage your Civitai models for your Stable Diffusion Web UI models folder
16 stars 7 forks source link

Link not scanning for new files #36

Open Brennok opened 4 months ago

Brennok commented 4 months ago

Updated to 1.15.1 yesterday and I have yet to see Link try and find new files. I don't see anything in the logs either. The newest file from download date shown is just before upgrading from 1.15.0 to 1.15.1. My latest file found is timestamped 4/22/2024 7:22 pm est and I downloaded files after this.

I also had the PC off overnight and started fresh this morning with still no new files detected,

main.log

cmoody commented 4 months ago

How is your directory structure setup? I made a change that stopped scanning the root directory but only scans the model folders. Either the default ones for the type you are using like A1111 or Comfy or if youve changed them then it only scans those. Wondering if that maybe messes up something.

Brennok commented 4 months ago

I use a folder named Stable Diffusion which is correctly set as my root folder in settings. I then have separate folders for Checkpoints, Lora, Embeddings, etc. I do have sub-folders in the various model folders separating the models by type, base, etc

For example D:\Stable Diffusion\Checkpoints\SD 1.0 D:\Stable Diffusion\Loras\SD 1.5 D:\Stable Diffusion\Loras\SD 1.5\Style

Then I symlink those into the various SD webUI model folders.

I did find in the settings just now that it was no longer pointed at my folders so not sure if the 1.15.1 update changed that. Lora for example was pointed to D:\Stable Diffusion\Lora now instead of D:\Stable Diffusion\Loras

I think it is ignoring sub-folder now. I moved a Lora out of my SDXL folder and into my main Loras folder and that file was found. It still hasn't picked up any of my other new models in the sub-folders. main.log

cmoody commented 4 months ago

Ah I think I know whats wrong. When I was scanning the root folder I was doing it recursively however I changed that when doing individual folder scans. Everyones setup is different so still learning on that front. Im adding it back in so should be good in the next version today.

I did find in the settings just now that it was no longer pointed at my folders so not sure if the 1.15.1 update changed that. Lora for example was pointed to D:\Stable Diffusion\Lora now instead of D:\Stable Diffusion\Loras

This one Im not sure about. I updated some folder stuff so I could have messed something up there. If you update it does it save/scan correctly, aside from the sub-folders?

Brennok commented 4 months ago

I did find in the settings just now that it was no longer pointed at my folders so not sure if the 1.15.1 update changed that. Lora for example was pointed to D:\Stable Diffusion\Lora now instead of D:\Stable Diffusion\Loras

This one Im not sure about. I updated some folder stuff so I could have messed something up there. If you update it does it save/scan correctly, aside from the sub-folders? It did save correctly when I changed it, I think it scans correctly since I did it before moving the one Lora to test.

Interestingly enough the count didn't go down when I noticed the folder change or I would have caught it when it happened so not sure if it was just displaying that way but actually remembered the correct folder or it didn't try to rescan when it updated the default folder.

cmoody commented 3 months ago

@Brennok checking on the status of this. We made a bunch of changes around folder watching and scanning. Starting in 1.18.0 I believe.

Brennok commented 3 months ago

@cmoody Downloading via link, models are detected immediately, and deleting models are detected pretty quickly, at least one downloaded via Link. I still need to test other files.

Manually downloading though still waiting for the new LoRA to be detected. I downloaded 6 LoRAs and will monitor to see how long it takes for them to show up. It has only been five minutes, but as of yet nothing.

Brennok commented 3 months ago

It has been an hour and still nothing. I don't know if I just have too many subfolders and it only monitors the main after initial scan or what triggers it that a new file is downloaded if you don't use link to download. Civitai under account does have a download history but not sure if there is a way to get a trigger from that to link that a new file was downloaded and to prompt a scan.

cmoody commented 3 months ago

They should show up immediately so thats a bit worrisome. The way it should work is it sets up watchers for all of the folders in your settings. It should watch them recursively so I dont think the number of subfolders should matter. We did notice an issue if you named it like X.Pony where it wouldnt work on Windows but that is resolved. That said looking at one of your examples D:\Stable Diffusion\Loras\SD 1.5 I wonder if theres anything I need to do with spaces thats maybe getting messed up.

Brennok commented 3 months ago

I would think it would cause an issue during scan if spaces were the issue, but maybe it is a different scan. I even tried just saving one in the main D:\Stable Diffusion\Loras in case it didn't go into subfolders for new models.

I am using the default naming from Civitai so nothing renamed.

Brennok commented 3 months ago

Hmm I think there might be something going on with scanning. I updated to 1.20.1 and my file count dropped. It still didn't find the new files either.

GothicSkate commented 2 months ago

I am having the same issue, starting with version 1.20.1. Once I updated, my model count dropped by 2000 and it has not added any new loras, dowloaded any new images, or created any .json files (specifically for loras) to the index for several weeks now, despite me adding 1000+ new files to the folder. It still updates embeddings and checkpoints, but no matter what troubleshooting I do, such as reinstalling, changing folder location from linked to actual directory, etc. nothing has worked, rending the program stuck with whatever was indexed prior to the update (minus the 2000 it initially stopped counting). I also have many subfolders inside my main lora directory for organization, but this was not a problem before the current update. The only exception is for loras downloaded using the link seem to still be indexed.

pryingopen3rdi commented 2 months ago

I'm having the same issue. I have reinstalled a few times on Kubuntu Linux and it is showing only 232 files. I have a lot more than that and it just isn't finding them anymore. However it has identified models in the correct folders. It just isn't scanning and adding them anymore. I tried deleting the config directory and reinstalling, but still no luck.

pryingopen3rdi commented 1 month ago

Anyway to completely uninstall Civitai link and all former db's etc so that I can get a clean install and see if this issue persists? Instructions for linux please. I already tried deleting the config directory as well as the app.

Brennok commented 1 month ago

Anyway to completely uninstall Civitai link and all former db's etc so that I can get a clean install and see if this issue persists? Instructions for linux please. I already tried deleting the config directory as well as the app.

I found the only way was to use RevoUninstaller. It lets you wipe all the install folders that the regular uninstall misses. Then reboot and do a fresh install. Just saw Linux, not sure on Linux though if you google revouninstaller for Linux someone might have suggestions or previously asked how to replicate it.

Might also consider emailing support. I haven't downloaded much lately so just went back to using the SD extension.

cmoody commented 1 month ago

Sorry for the delay responding to some of these. Been focused on a few other things. Im taking a look at making sure it removes app data when uninstalling. For now you can manually delete it based on your platform here https://www.electronjs.org/docs/latest/api/app#appgetpathname

Im going to test out again with adding/removing models to see if I can identify anything. The one thing I will say that could be part of this difference is we only display models that are on Civitai. So if you are creating your own or downloading from other sites and they are not on Civitai, they wont show up. One of the reasons the app is resource intensive for the first time install is because we hash the model file and use that to identify it against the Civitai API. If the hash isnt found then we dont have any data to display and wont count it towards the total.

cmoody commented 1 month ago

One other thing is this is an open source project so contributions are welcome, especially for Linux. Im happy to schedule some time to chat and walkthrough things if yall are wanting to take a crack at any of these issues. Id have to check with the team but we can probably work out a buzz bounty around that.