search

Lightning-Universe / lightning-flash

Your PyTorch AI Factory - Flash enables you to easily configure and run complex AI recipes for over 15 tasks across 7 data domains
https://lightning-flash.readthedocs.io
Apache License 2.0
1.74k stars 212 forks source link

GAN tasks in Flash #270

Closed aniketmaurya closed 3 years ago

aniketmaurya commented 3 years ago

🚀 Feature

Support for GAN models - image2image translation, image generation.

Motivation

Flash already has text, vision, video, and tabular. It will be awesome to have a module for GANs too! We can start with models already present in pl_bolts and another repository like lucidrains

PS: I can work on implementing this feature in Flash!

ethanwharris commented 3 years ago

Yeah, this would be really cool, can maybe share some components with the pystiche integration being added in #262 :smiley:

edgarriba commented 3 years ago

Should we group all those those tasks under image_generation ? basically to cover all those that inputs/outputs images /cc @pmeier

pmeier commented 3 years ago

Should we group all those those tasks under image_generation ?

IMO that is a good idea, but image_generation is not general enough: for example style transfer or cycle GANs do not generate image but perform an image translation.

aniketmaurya commented 3 years ago

Agree @pmeier . I would suggest to have a Task for GAN and subtasks can be image2image translation, image Generation, image to text Generation, etc.

Inside image translation also we have multiple category - paired and unpaired image2image translation

edgarriba commented 3 years ago

@aniketmaurya My point was more that to me, a GAN is a technique to solve a Task but not a task itself.

aniketmaurya commented 3 years ago

I created a quick and dirty poc for gan module in Flash. Pls have a look and let me know if I am proceeding right or not

https://github.com/aniketmaurya/lightning-flash/blob/gan_poc/flash/gan/image_translation/models/paired_translation.py

edgarriba commented 3 years ago

@aniketmaurya I suggest to go for one task at a time and create just flash.vision.image_translation - the task should be just ImageTranslation.

You can check on other tasks like segmentation and try to follow the same structure of files (think as each task a mini-project inside flash):

In this particular case, I would do something like backbone = pix2pix (or similar known method) and then make somehow a PairedImageTranslationBackbone class that handles the generator and discriminator during the forward pass.

Regarding, the training - do you have any implementation in mind yet ?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.