Closed NUROISEA closed 1 year ago
etherealxx
commented
1 year ago The thing is, sometimes gdown fails to download from google drive, idk why. Weird.
Something like this. Even though the link is public.
I'll try to add the other websites. You can list the most used website for these.
Catbox is banned in my country, i'll figure things out.
etherealxx
commented
1 year ago for discord is it this link? https://cdn.discordapp.com/attachments/*
NUROISEA
commented
1 year ago The thing is, sometimes gdown fails to download from google drive, idk why. Weird.
Heavily guessing here, but maybe the file has a delay on being updated to the servers?
Catbox is banned in my country, i'll figure things out.
Even if catbox is banned in your country, pretty sure that Google's servers are in not in the countries that have strict internet censorship. IIRC, the servers are in US, EU, and SG. catbox links are wget-able and will most likely work with any CLI downloader.
for discord is it this link? https://cdn.discordapp.com/attachments/*
Yes it is!
I'll try to add the other websites. You can list the most used website for these.
I think having a special case for websites would be a bad idea and will lead to code bloat, I propose that having a toggle of downloading from anywhere would faster to implement (also easier?).
But regardless, these are the sites that provide direct links to files that I can list at the top of my head:
*.s-ul.eu/*github.com/*/*/raw/*/* or raw.githubusercontent.com/*/*/*cdn.discordapp.com/attachments/*/*/*files.catbox.moe/*gitgud.io/*/*/-/raw/*/*etherealxx
commented
1 year ago pretty sure that Google's servers are in not in the countries that have strict internet censorship. IIRC, the servers are in US, EU, and SG. catbox links are wget-able and will most likely work with any CLI downloader.
wget etc. doesn't work on catbox even on colab, i tried. Maybe i'll try again with vpn.
NUROISEA
commented
1 year ago pretty sure that Google's servers are in not in the countries that have strict internet censorship. IIRC, the servers are in US, EU, and SG. catbox links are wget-able and will most likely work with any CLI downloader.
wget etc. doesn't work on catbox even on colab, i tried. Maybe i'll try again with vpn.
Can confirm, catbox does not work. It's just stuck in an eternal halt.
Apologies, I must've jumbled up my memory 😅
etherealxx
commented
1 year ago @NUROISEA try this branch
!git clone -b ariasubprocess https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webuii added aria2 downloader, discord attachment, changed os.system to subprocess, more accurate logging system, controlnet path, and github extension clone. I'll merge to main if it's stable enough.
Still uncertain about the other websites. Maybe you can give me model link examples for every website?😊
Imma sleep for now. It's midnight on my timezone🌃
NUROISEA
commented
1 year ago Did a quick test with aria2. It seems like the downloaded files are not being placed to their respective folders.
This is my input:
#embed
https://cdn.discordapp.com/attachments/1064919066420203551/1064919069519773757/EMT.pt
https://cdn.discordapp.com/attachments/1066570375653244928/1066570434746785882/Moomers.pt
#hypernet
https://cdn.discordapp.com/attachments/1070489470127841381/1070489471964954684/MomopocoV3.pt
https://github.com/fkunn1326/openpose-editorUI Output:
NUROISEA
commented
1 year ago gdown test, same inputs, download fails
UI output:
NUROISEA
commented
1 year ago wget test, same inputs (only removed github link), download fails
UI output:
NUROISEA
commented
1 year ago curl test, same inputs (no github), downloads correctly
UI output:
etherealxx
commented
1 year ago Sorry, i think the extension update breaks everything. You can git checkout to the previous commit (on the same branch). I'm currently not in my house, I'll check that later.
etherealxx
commented
1 year ago Try this one
!rm -rf /content/stable-diffusion-webui/extensions/batchlinks-webui
!git clone https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webui
!cd /content/stable-diffusion-webui/extensions/batchlinks-webui
!git checkout ariasubprocess
!git checkout 0e0c37844eef9be0be91d5dddba758c94931005e
etherealxx
commented
1 year ago @NUROISEA whoops sorry, wrong command. Here's the one:
!rm -rf /content/stable-diffusion-webui/extensions/batchlinks-webui
!git clone -b ariasubprocess https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webui
%cd /content/stable-diffusion-webui/extensions/batchlinks-webui
!git checkout 0e0c37844eef9be0be91d5dddba758c94931005e .okay, i noticed the same problem also happens on that commit. I know the problem and will fix it soon
etherealxx
commented
1 year ago @NUROISEA sorry for the inconvenience🙏, i pushed a new commit.
!rm -rf /content/stable-diffusion-webui/extensions/batchlinks-webui
!git clone -b ariasubprocess https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webuiIt fixes most of the bugs, all downloader working, extension texted, haven't tested discord attachment yet. Thank you by the way, for debugging my previous code, really appreciated😁✨
Currently i'm planning to update it once more before merging to main. Someone give me codes to directly download from civitai model page link, instead of from civitai direct link.😊
NUROISEA
commented
1 year ago Hi @etherealxx, I tested your branch against the latest version of the webui and against my fork (previous versions of the webui) and I saw no problems with your extension!
Test case:
#model
https://huggingface.co/NoCrypt/SomethingV2/resolve/main/somethingv2_1.safetensors
#vae
https://huggingface.co/NUROISEA/av3-mirror/resolve/main/anything.vae.pt
#embed
https://cdn.discordapp.com/attachments/1064919066420203551/1064919069519773757/EMT.pt
https://cdn.discordapp.com/attachments/1066570375653244928/1066570434746785882/Moomers.pt
#hypernet
https://cdn.discordapp.com/attachments/1070489470127841381/1070489471964954684/MomopocoV3.pt
#lora
https://huggingface.co/YoungMasterFromSect/Trauter_LoRAs/resolve/main/LoRA/Misc.%20Characters/Aponia/Aponia_Softpruned.safetensors
https://github.com/fkunn1326/openpose-editorTested with all 4 download methods gdown, wget, curl, aria2 with both versions of the web ui and it works as expected.
Test environment (a colab I quickly wrote for this purpose)
Output:
Test limitations:
etherealxx
commented
1 year ago Thank you!😊✨ MEGA is working fine, so does Civitai. However, a friend of mine implemented a new way to download Civitai stuff using the api. So instead of using the direct download link (example, https://civitai.com/api/download/models/5616), it uses the model page link, (example, https://civitai.com/models/4823/deliberate), and it will automatically download the best model, download the thumbnail, and put it as as a preview image on webui. Also, it automatically knows the type of the model (it will be located properly automatically without using hashtag thing). If that works perfectly, then sd-civitai-browser would be kinda useless😅
So i'll try to implement that on the next commit.
Also that person commented about the UI. I wanna hear from you what do you think about the UI honestly. I try to make it as easy-for-beginner as possible (as i was months ago).
NUROISEA
commented
1 year ago So i'll try to implement that on the next commit.
Yo, let's go!
I wanna hear from you what do you think about the UI honestly. I try to make it as easy-for-beginner as possible
The UI is intuitive, at a glance you kinda know what you are dealing with already, so no problems on the UI side of things. :D
etherealxx
commented
1 year ago hello @NUROISEA 👋
After hours of staring at codes, finally i pushed this (almost) final commit. Here's the list of all new features
you can test it with this:
!git clone -b ariasubnewcivit https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webuii tested most of it works (haven't tested Windows yet). If it's stable enough, i'll merge it with main. (I'll also ask some of my friend on discord to test it too i think)😊
NUROISEA
commented
1 year ago Hi @etherealxx! here's my testing:
| Results: Tested on the latest version of the webui and a slightly older one | gdown |
wget |
curl |
aria2 |
|
|---|---|---|---|---|---|
| huggingface | ✅ | ✅ | ✅ | ✅ | |
| civitai | ✅ | ✅ | ✅ | ❌ | |
| discord | ✅ | ✅ | ✅ | ✅ | |
| mega | ✅ | ✅ | ✅ | ✅ | |
| github | ✅ | ✅ | ✅ | ✅ |
Civitai seems to fail on both methods on aria2, direct /api/models/ and /models/, error seems to be the same to both instances:
Another bug I saw is the output box seems to not display anything after finishing the task, successful or not. Granted, I am not using --gradio-queue in my tests.
etherealxx
commented
1 year ago Hi @etherealxx! here's my testing:
Thank you so much for the findings!🤗✨
that error log is actually comes from the debug text i made for, well, debugging purposes.
print("debug folderforsavestate: " + folder)
TypeError: can only concatenate str (not "PosixPath") to strshould be like this
print("debug folderforsavestate: " + str(folder))Also yeah, i haven't tested this on non queue environment and the old version, i might check that. More bug fixing to come🛠️🪲🏃♂️💨💨
etherealxx
commented
1 year ago hi @NUROISEA , it's been a while😅 I did more fixing and adding stuff. I'm hoping that this is the last update before i merge to main.😁 Here's all the new features. (and yes, github raw and catbox is supported)
!git clone -b ariasubnewcivit https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webuiI tested it on camenduru's v2.0. Also with gradio queue on, as queue off might breaks some things. (camen's colab is queue on by default anyway)
I also has this version that was built solely for camenduru's v1.6 colab. (won't work properly on others)
!git clone -b onedotsix https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webuiAnother bug I saw is the output box seems to not display anything after finishing the task, successful or not.
this one seems to be an issue with timeout and websocket (i didn't quite understand that), but to mitigate that problem, this 'onedotsix' variant will cut the download session every 2 minutes, and there will be a resume button to resume the cutted download session.
Just wanna say thank you for your support all this time, really means a lot to me🙏✨
NUROISEA
commented
1 year ago Hi hello @etherealxx o/
I'm quite short in time so I only tested your extension on a 1 month old commit of the webui since I figured there's no major changes between this and the latest version of the webui anyway 😄
Alas the test went smooth, no problems whatsoever!
Tested with --gradio-queue
All links work on all download methods, even catbox!
Test limitation is I didn't test the raw github links since I cannot find TIs or anything that is hosted in github.
Since stuff works flawlessly, I'll just suggest things. These can be done on a later date, whenever you feel like it 😆
You can maybe change this so people know they can use catbox links
The UI gets very squished on mobile phone resolutions.
Catbox downloads files that have gibberish file names (the same as in the URL), maybe utilize the ## comments to it? An idea:
#lora
https://files.catbox.moe/uarze8.safetensors ##r neurosamaNotice the ##r, this would be downloaded as neurosama.safetensors instead of uarze8.safetensors. This is not an elegant approach, but I'm pretty sure that you can do something better.
Google drive support? This could be a higher priority since a lot of people use drive to store models since not everyone can use HuggingFace.
You did well on this 👏👏👏
I added your extension earlier on my colab collection, so kinda so it's being battle tested right now 😄
etherealxx
commented
1 year ago Alas the test went smooth, no problems whatsoever!
Thank you! Since this is stable enough, i'll do some tweak then push straight away! (it's been pain in the ass holding the update long enough😅) Then i'll start adding more features.
You can maybe change this so people know they can use catbox links
Will do! forgot about that😅
The UI gets very squished on mobile phone resolutions.
This one is kinda hard. If i remove the drop file here on the right, the UI would look really ugly on my ultrawide monitor😅. But yeah, it would help mobile user. I'll try to improve the UI after the v2.0.0
Catbox downloads files that have gibberish file names (the same as in the URL), maybe utilize the ## comments to it? An idea:
While i really like customization, it might get more confusing, as this hashtag syntaxing thing is, while designed to be really beginner friendly, i can see some people still getting confused to it. I would, figure out some stuff to allow easier way to renaming things.
Google drive support? This could be a higher priority since a lot of people use drive to store models since not everyone can use HuggingFace.
Let's say it's complicated right now (#43 #26 #146). Again, I'll add it if i figured things out.
You did well on this 👏👏👏
Lastly, thank you so much 🙏✨ Couldn't thank you enough.
etherealxx
commented
1 year ago Hello @NUROISEA , long time no see😊
For the past couple weeks i've been developing on the next update of this extension. Some of the feature are the one that you requested: renaming and google drive support.
In fact, the rename function already available since weeks ago i think😅 works something like this:
#lora
https://files.catbox.moe/uarze8.safetensors > neurosama.safetensorsfor current version that i'm developing, it's aiming to support more and more website. Currently:
This update will also use different method for downloading civitai from direct links, speeding up the download speed by up to 40x (looks comical but the current method can sometimes goes really slow like 5Mbps). Also there's a lot going on in the backround, related to how civitai model page link is downloaded (takes a lot of my time to fix it 🙃)
Another feature i'm implementing is called 'sdless', where you can run the extension without even launching the sd-webui (more here). It makes my developing progress much quicker! (though, this is the toughest update since i'm still learning things 😅)
Next feature that someone suggested is custom hashtag with custom path. Works something like this
@new #temp /content/temp
#temp
<link>Another qol feature is @extract to extract every 7z file on curent directory.
Lastly, the UI got updated a bit, and that's it.
if you want to test it:
!git clone -b sdless https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webuiI might spend another day or two to write some stuff on the readme page, and then release the update straight away.
I would like to reach you in discord if you have one, so we can talk or something👀. Or maybe just join camenduru's server since i'm there too.✨
NUROISEA
commented
1 year ago Hi @etherealxx! Cool update!
I have tested the extension with the latest A1111 commit with the same text file as my previous comments and there are some bugs, unfortunately:
The progress bar doesn't have any indication? the bar colors, the text percentage seems to work as intended.
Is this intentional? The progress bar appears on the hashtags table:
Do note that my test is limited since I have limited time and do not have the proper links to the new sites but I would assume that they work, catbox in an exception since I am getting unlucky with the servers and I cannot download from that site. This is it for my bug reports. (sorry if this feels quite lazy I'm also doing something and I'm just testing on my downtime 😅)
I tried breaking your extension, by mixing up the hashtags, a simplified example would be:
#lora
<link>
#model
<link>
#lora
<link>
...and it would just download it correctly to the appropriate places!
The rename feature is such a nifty QoL one, thank you for that. The custom path works as intended too!
I haven't tested @extract and I assume it only works with .7z?
I like what you have done to the UI, hiding stuff and stretching the UI seems pretty nifty (although stretched looks dank on wide resolutions 😂)
Overall, good job on this one! Seems pretty solid by the looks of it! 🎉
And yes I do have discord, I'll just DM you with this screenshot of this response in the editor. I think you can verify if you try to edit this.
etherealxx
commented
1 year ago Okay, time to address the problem.
Progress bar seems to be stuck to -1.0% if the download fails
i don't know how to replicate it, honestly. Can you give me the example/links where it happens? all i know for 1% is, i put that as placeholder to trigger the progressbar to run (idk but at one point the progress bar wont show up without that placeholder). I think i'll make sure the progress bar goes to 100% whenever the download is complete.
The progress bar doesn't have any indication? the bar colors
It's supposed to be orange, gradio's color. But i think the auto1111's code messed it up. When i launch the UI standalone-ly, the progress bar works fine.
Is this intentional? The progress bar appears on the hashtags table:
so the thing is, the way gradio works is an UI change only could happen by a return value passed into the components. I want it so that after the download is finished, the table become automatically updated, so that when ppl use custom hashtag, it will shows after the download is finished right away. (i can't update the UI while the code is running, unless it use gradio every like the log text, which blinks every now and then).
In order to update the table right after the download is complete, i need to assign the table as the output for the button, just like the output box. But how the progress bar works in gradio is, it will make every output as a progress bar, hence the table also appears as progress bar. (sucks honestly, but i can't find a good workaround currently)
Do note that my test is limited since I have limited time and do not have the proper links to the new sites but I would assume that they work
I do test all of them, but you seem have a skill to somehow breaks my extension sometimes, and that's useful😆
sorry if this feels quite lazy I'm also doing something and I'm just testing on my downtime 😅
Don't worry! I'm not forcing you at all. I'm just asking for a little help in your spare time😋 don't feel burdened, consider it a little activity when there is nothing to do.
I haven't tested @extract and I assume it only works with .7z?
yeah i made it bcoz one of the link i downloaded from pixeldrain is a .7z file. Would be very handy coz the syntax for extracting a simple single item is pretty long. I'll add .zip and .rar support too i guess...
I like what you have done to the UI, hiding stuff and stretching the UI seems pretty nifty (although stretched looks dank on wide resolutions 😂)
The reason i haven't got rid that useless huge drop your txt here on the old version is because, just like you said, it looks jank on my ultrawide monitor. So i instead add it as an option. The strech UI thing works great on mobile. Unstretched is the default, as it's how the UI looks like before.
Again, thank you for spending your time on this passion project of mine, really really appreciated😊
etherealxx
commented
1 year ago The update is now live! Thank you for every of your previous supports on this project!💖✨
Release note (every new feature): https://github.com/etherealxx/batchlinks-webui#release-v301
A lot of embeddings and loras are uploaded to catbox.moe and related sites from users on 4ch.
Maybe an edge case, I tend to store my TIs in Discord since its quite convenient.
Since this extension is already utilizing
gdown, downloading from Google Drive also?Maybe add this as a toggleable option since supporting every site will lead to unexpected things.