Open patrickvonplaten opened 2 years ago
Docs are now live :tada:
Thanks to @natolambert we have a rough template of diffusers docs: https://huggingface.co/docs/diffusers/index
In two days, we want to kick-start a doc-sprint where we add as many doc-strings as possible. Here the rough draft: https://docs.google.com/document/d/1TwqMlQrjWXi5a0y-C-Wbzx70R4ZQP8f8mq-ARSqB5fI/edit?usp=sharing
Tomorrow we will share more details on what exactly what could be worked on. Let us know if you'd like to participate in the doc-sprint on Wednesday with a :+1:
The general doc structure (which is defined here) should look more or less as follows:
"1. Get started" I think the following subsections for "1. Get started" would be nice: "1.1" Diffusers (as is there already), but needs to be adapted "1.2" Quicktour (as is there already), but needs to be adapted “1.3” Installation
“2. Using Diffusers” We only have vision at the moment so let’s maybe just focus on this:
2.1 Loading pretrained instances (explain from_pretrained for all pipelines, models, schedulers)
2.2 Pipelines for Inference 2.1.1 Unconditional Generation 2.1.2 Conditional Generation 2.1.3 Image2Image 2.1.4 In-painting 2.1.5 Create a custom pipeline
"3. Custom Hardware / Optimizing"
pipelines for inference (fp16, autocast, onnx, mps, OpenVino)
“4. Training Diffusers”
3.1 Unconditional Training 3.2 Text-inversion 3.3 Conditional Training (TODO)
“5. Conceptual guides”
"4.1" Here I'd maybe now make a whole section for "Stable Diffusion" with subsections like ethics, optimization, the Hub, ... "4.2" Philosophy (as is there already), but would need to be adapted "4.3" Contribution (we can more or less copy what we have here: )
“5. API”
This is pretty self-explanatory - all public methods need some nice docstring. For now I’d put all models (UNet2D, UNet2DCondition) in one doc string file and also put all schedulers in one doc string file. The each pipeline should get a “Pipeline card” just like we do for Transformers models -> we could just copy-paste the format from: https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/stable_diffusion (which could be the stable_diffusion.mdx model card) to all other models.
For section 5.
We need to adapt both the code files, e.g. add nice doc-strings to each public function and class as well as add some .mdx
files for the main API classes.
In general section 5 will look as follows:
API
- Main classes
- Models (one file with info and docstrings about `ModelMixin`, `Unet2DModel` ,`Unet2DConditionModel`, `VQModel`, `AutoencoderKLModel`
- Schedulers (one file with info and docstring about all scheduler classes - cc @natolambert)
- DiffusionPipeline (one file - only with `DiffusionPipeline` class)
- Logging (can copy from transformers: https://huggingface.co/docs/transformers/main_classes/logging)
- Config (one file - only with `ConfigMixin`) and `register_to_config`
- Outputs (one file with all `BaseOutput` classes)
- Pipelines (similar in spirit to Transformers models): https://huggingface.co/docs/transformers/model_doc/albert
- Stable Diffusion (one file which can be a copy of: https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/stable_diffusion)
- DDPM (we should create a README similar to https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/stable_diffusion
Communication tomorrow will happen over a discord and we'll also hold a call at 9h30am Paris time. Please make sure to join discord to join the call and to participate :-)
To join: Follow this link https://discord.gg/bYA2G9yJtP, then click on the 🎨 emoji to get access to the diffusers group. There you'll find the "core-contributions" channel where we'll post all the important info!
Working on scheduler updates in #376.
For anybody who wants to join - ideally we start with 5. API docs for pipelines. Just pick one of the following pipelines and follow the PR: https://github.com/huggingface/diffusers/pull/377/files
latent_diffusion.mdx
_toctree.yml
: https://github.com/huggingface/diffusers/blob/main/docs/source/_toctree.ymldef __init__
and def __call__
Pick one of the following pipelines to get started :rocket: & please note below which pipeline you would like to work on with an open PR :-)
Taking latent diffusion in this PR: https://github.com/huggingface/diffusers/pull/377/files
I can take stable diffusion if that's ok.
I can take 2 ddpm and pndm
I'd like to take latent_diffusion_uncond.
Template is ready: https://github.com/huggingface/diffusers/pull/385
=> We can start filling it out :-) (Note that I copy-pasted quicktour.mdx
to all doc pages and models.mdx
to all API pages to be faster. Make sure to delete / overwrite those :-)
First example you can copy from: https://github.com/huggingface/diffusers/pull/377#pullrequestreview-1098963887
i can do ddim karras and score_sde_ve next
Okay, seems like there is some interest from North America timezone folks! Here's a list of pages that could use some contributions by category. These are a little more open-ended than the Pipelines which focus on one system.
You can see the latest docs here where one can easily find blank pages.
Additionally, doc strings are very important! You can choose some code that no one mentioned above and write docstrings for every public method (I have a lot unmerged yet for schedulers).
UNet2DModel
: https://github.com/huggingface/diffusers/blob/1a431ae886b516d13f49c1f8a1e1e68d3159eab4/src/diffusers/models/unet_2d.py#L25UNet2DConditionModel
: https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/unet_2d_condition.py#L25VQModel
: https://github.com/huggingface/diffusers/blob/1a431ae886b516d13f49c1f8a1e1e68d3159eab4/src/diffusers/models/vae.py#L370AutoencoderKL
: https://github.com/huggingface/diffusers/blob/1a431ae886b516d13f49c1f8a1e1e68d3159eab4/src/diffusers/models/vae.py#L450These are conceptual guides covering different applications -- could be based off previous colabs released with diffusers:
These may be a bit harder unless you have experience with them
(All except fp16
are available, fp16 covered in #404 )
_toctree
had a typo that made the API
heading undefined.
Fix here #406 .
Thanks!
Large schedulers push done here in #376, ready for review.
Did a quick scan for typos in #408
@patrickvonplaten Can you check if #413 and #414 look good?
Hi, there is a minor mistake in https://github.com/huggingface/diffusers/blob/main/docs/source/installation.mdx #402
git clone https://github.com/huggingface/diffusers.git
cd transformers
pip install -e .
The user should cd to diffusers not transformers ! Don't know where to report this, so mentioning this here.
@utsavnandi Thank you for pointing it out; I've created a PR to address it :) #453
There is still more TODO, to see open options, see this comment
Let's add them docs.
cc @anton-l