search

x-CK-x / Dataset-Curation-Tool

A tool for downloading from public image boards (which allow scraping) / preview your images & tags / edit your images & tags. Additional tabs for downloading other desired code repositories as well as S.O.T.A. diffusion and auto-tag/caption models for your purposes. Custom datasets can be added!
GNU General Public License v3.0
33 stars 7 forks source link

Remove aria2c #3

Closed jorektheglitch closed 1 year ago

x-CK-x commented 1 year ago

did you plan on having aria2 & wget replaced in the webui,py & helper_functions.py also, or just the batch downloader for now?

jorektheglitch commented 1 year ago

For now it's just about batch downloader. Feels free to use FliesDownloader to replace aria2c everywhere it used.

x-CK-x commented 1 year ago

also was that gitignore auto-generated or something? there's a lot of things in that file

jorektheglitch commented 1 year ago

Yes, I use slight edited Python.gitignore from gitignore collection. It covers a lot of things and simplifies developing process.

x-CK-x commented 1 year ago

I still need to test on Linux, but while the files were downloaded. The program after ## Checking tag search query on windows started to loop the download of the tag csv.gz file again and then tried to reload it a second time before actually downloading images. /proxy_test/Dataset-Curation-Tool/tags-2023-05-20.csv.1.gz

That's really the only issue happening atm. I'll let you know if it also happens on linux.

x-CK-x commented 1 year ago

Yes, I use slight edited Python.gitignore from gitignore collection. It covers a lot of things and simplifies developing process.

I moved all the webUI downloading to helper_functions.py, can you add your proxy feature to those?

https://github.com/x-CK-x/Dataset-Curation-Tool/blob/9b98e1cdf5d930a222130e2ae0b332e35f49b9c6/helper_functions.py#L372

jorektheglitch commented 1 year ago

Surely. Will update PR asap.

x-CK-x commented 1 year ago

The webui has reached a somewhat steady-state. If you want to update the batch downloader script & the image board download functions in the helper_functions script to be compatible with using a proxy just let me know. Here or discord works.