search

jhc13 / taggui

Tag manager and captioner for image datasets
GNU General Public License v3.0
777 stars 37 forks source link

Duptool #126

Closed yggdrasil75 closed 7 months ago

yggdrasil75 commented 7 months ago

marking draft instead of completing and full pr. dont want to spend a lot of time on this if its out of scope.

goal is to make it easier to dedup an automatically downloaded dataset. lots of issues right now, will clean them up before marking as a proper pr. literally does nothing as the widget hasnt been hooked up yet.

jhc13 commented 7 months ago

Nice work, but this does seem to be out of scope:

  1. TagGUI was designed as a tool for managing tags and captions of datasets, and not the images themselves.
  2. There are multiple popular preexisting tools for image deduplication, like Czkawka.
yggdrasil75 commented 7 months ago

yeah, that is what I was thinking the response would be, hence why its nowhere near done before I make spend the next week making it run well and get declined. I just wanted to be able to easily pull tags from image a downloaded from source a to image a from source b where both images are the same, but tagged differently because of how the sites operate differently.

my biggest issue with using any existing options is they dont handle the caption file that goes with them, while this option would, and would potentially allow (via feature matching option) comparing the tags of 2 images with the same character and seeing what might be missing from one that was added to the other.

jhc13 commented 7 months ago

I understand, but I think it's a slightly niche use case. You could create your own tool for your purposes.

geroldmeisinger commented 5 months ago

I agree that this is already implemented in other tools (I used fastdup btw) On the other hand I had the usecase of preparing an image dataset for fine-tuning a stable diffusion model. when de-duplicating images I had to decide which caption to take from up to 3000 duplicates (for one image). I think this is something that taggui should handle ("merging tags") even though the duplication info doesn't have to necessarily come from taggui. a "tag compare and diff"-view maybe.