search

finegrain-ai / refiners

A microframework on top of PyTorch with first-class citizen APIs for foundation model adaptation
https://refine.rs
MIT License
585 stars 43 forks source link

will anyone implement this for comfyui? #396

Closed rafstahelin closed 1 week ago

rafstahelin commented 1 month ago

Seems to work quite well sometimes for removing text, logos, and titles on photos on the covers of magazines. would be very useful

limiteinductive commented 1 month ago

Hey @rafstahelin, are you talking about the Object Eraser?

rafstahelin commented 1 month ago

Hey @rafstahelin, are you talking about the Object Eraser?

Yes, the eraser

kadirnar commented 3 weeks ago

@rafstahelin @limiteinductive I can do it. Is there any technical documentation for Object Eraser? If you want, I can add ComfyUI support for all models in the refiners library.

rafstahelin commented 3 weeks ago

@rafstahelin @limiteinductive

I can do it. Is there any technical documentation for Object Eraser? If you want, I can add ComfyUI support for all models in the refiners library.

No idea. I couldnt see any

kadirnar commented 3 weeks ago

I can use LaMa to delete objects. I can use Sam-v1 to mask. I need to do some research and review the codes. I will get back to you in 1-2 days.

LaMa : https://github.com/advimman/lama

rafstahelin commented 3 weeks ago

Cool.

I already use lama cleaner that does a good job in removing some logo and magazine titles but it usually leaves smudges over larger text areas.

Would be great if this lama you refer to could still clean up the image behind the removal

And, using natural language rather than drawing would be a game changer

Thanks for your interest

kadirnar commented 3 weeks ago

Thanks. Can you send me a sample image? Do you want to delete the background or the object?

rafstahelin commented 3 weeks ago

image

I mean the text on top of images as one use case.

It's very useful for cleaning datasets for training or for running inference on the reference image with controlnet and Ipadapter without these visual distractions

Of course removing an actual physical object is another use case

kadirnar commented 3 weeks ago

Have you tried using ClipSeg(prompt="text") + LaMa? I'll create a pipeline and test tomorrow.

rafstahelin commented 3 weeks ago

I actually didn't try it on text but good point. Have used both for other stuff. Will try tomorrow

deltheil commented 3 weeks ago

I can use LaMa to delete objects. I can use Sam-v1 to mask. I need to do some research and review the codes. I will get back to you in 1-2 days.

@kadirnar I’m afraid you won’t get the same quality vs. our specialized model. FYI, we are considering open sourcing it, stay tuned!

kadirnar commented 3 weeks ago

I can use LaMa to delete objects. I can use Sam-v1 to mask. I need to do some research and review the codes. I will get back to you in 1-2 days.

@kadirnar I’m afraid you won’t get the same quality vs. our specialized model. FYI, we are considering open sourcing it, stay tuned!

I like contributing to open source projects. Do you have a roadmap? Do you want to add models like aurosr, laMa, clipseg...? Do you want to add only diffusion models like diffusers?

catwell commented 3 weeks ago

To clarify, Refiners (the framework itself and the implementations of models it contains) itself is completely open source. The Object Eraser mentioned by the OP is implemented using Refiners but it is not Open Source.

Regarding Refiners implementing other models with it is completely fine! We do not only want diffusion models, we already have e.g. SAM and we'd even like some LLMs. We even have a list of bounties you can check out.

kadirnar commented 3 weeks ago

To clarify, Refiners (the framework itself and the implementations of models it contains) itself is completely open source. The Object Eraser mentioned by the OP is implemented using Refiners but it is not Open Source.

Regarding Refiners implementing other models with it is completely fine! We do not only want diffusion models, we already have e.g. SAM and we'd even like some LLMs. We even have a list of bounties you can check out.

I looked at this documentation. Only train projects are empty. Will you add different models? Can I add different models to this library? Do you have any plans for the ComfyUI tool?

deltheil commented 3 weeks ago

Can I add different models to this library?

Contributions are more than welcome! E.g. FLUX or SAM 2 would be awesome, in particular.

Do you have any plans for the ComfyUI tool?

We don't have anything in the making at the moment: if you think about interesting parts to wrap into ComfyUI, please go ahead!

rafstahelin commented 3 weeks ago

flux would be awesome indeed

On Mon, Aug 19, 2024 at 3:26 PM Cédric Deltheil @.***> wrote:

Can I add different models to this library?

Contributions are more than welcome! E.g. FLUX or SAM 2 would be awesome, in particular.

Do you have any plans for the ComfyUI tool?

We don't have anything in the making at the moment: if you think about interesting parts to wrap into ComfyUI, please go ahead!

— Reply to this email directly, view it on GitHub https://github.com/finegrain-ai/refiners/issues/396#issuecomment-2296576052, or unsubscribe https://github.com/notifications/unsubscribe-auth/APNXTFXRPICIISRF2NRRRP3ZSHXAFAVCNFSM6AAAAABL3OMCKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJWGU3TMMBVGI . You are receiving this because you were mentioned.Message ID: @.***>

-- thanks

best regards, raf

rafstahelin commented 3 weeks ago

To clarify, Refiners (the framework itself and the implementations of models it contains) itself is completely open source. The Object Eraser mentioned by the OP is implemented using Refiners but it is not Open Source.

Regarding Refiners implementing other models with it is completely fine! We do not only want diffusion models, we already have e.g. SAM and we'd even like some LLMs. We even have a list of bounties you can check out.

I looked at this documentation. Only train projects are empty. Will you add different models? Can I add different models to this library? Do you have any plans for the ComfyUI tool?

Hey how's it coming along?

kadirnar commented 3 weeks ago

@rafstahelin

I tested CLIPSeg + LaMa model. CLIP model doesn't detect text well. That's why the output is bad. You can use VLM models for text detection. But vlm models run slowly.

CLIPSeg:

image

Dino + Sam:

image
dnsbrl commented 3 weeks ago

There are detectors designed for text, like https://github.com/clovaai/CRAFT-pytorch. Maybe worth trying.

kadirnar commented 3 weeks ago

This can be done using this library. I want to add generative ai models to this library.

Github: https://github.com/fcakyon/craft-text-detector

rafstahelin commented 2 weeks ago

This can be done using this library. I want to add generative ai models to this library.

Github: https://github.com/fcakyon/craft-text-detector

Would love to test your extension for comfyui if you've got anything going yet ?

kadirnar commented 2 weeks ago

This can be done using this library. I want to add generative ai models to this library. Github: https://github.com/fcakyon/craft-text-detector

Would love to test your extension for comfyui if you've got anything going yet ?

I am not working on this subject at the moment. My first goal is to learn the refiners library and add new models to this library. I am also training controlnet for flux.

rafstahelin commented 2 weeks ago

This can be done using this library. I want to add generative ai models to this library. Github: https://github.com/fcakyon/craft-text-detector

Would love to test your extension for comfyui if you've got anything going yet ?

I am not working on this subject at the moment. My first goal is to learn the refiners library and add new models to this library. I am also training controlnet for flux.

That sounds really cool. I think IP Adapter is also in very high demand.

totally fine. I guess I ll just just bbox and flux inpainting

thanks

Laurent2916 commented 1 week ago

Hey @rafstahelin and @kadirnar

We made a couple of comfy custom nodes, see #421. We only support our new Box Segmenter at the moment, but we're thinking of adding more nodes since there seems to be a demand for it.

We leverage the new (beta) Comfy Registry to host our nodes. They are available at: https://registry.comfy.org/publishers/finegrain/nodes/comfyui-refiners. You can install them by doing:

comfy node registry-install comfyui-refiners

Or by unzipping the archive you can download by clicking "Download Latest" into your custom_nodes comfy folder.

kadirnar commented 1 week ago

Hey @rafstahelin and @kadirnar

We made a couple of comfy custom nodes, see #421. We only support our new Box Segmenter at the moment, but we're thinking of adding more nodes since there seems to be a demand for it.

We leverage the new (beta) Comfy Registry to host our nodes. They are available at: https://registry.comfy.org/publishers/finegrain/nodes/comfyui-refiners. You can install them by doing:

comfy node registry-install comfyui-refiners

Or by unzipping the archive you can download by clicking "Download Latest" into your custom_nodes comfy folder.

Will you create a Github repo?

Laurent2916 commented 1 week ago

Will you create a Github repo?

They are inside the main repo: https://github.com/finegrain-ai/refiners/tree/main/src/comfyui-refiners