search

mindspore-lab / mindone

one for all, Optimal generator with No Exception
Apache License 2.0
328 stars 62 forks source link

[Features] New modules in mindone/diffusers/models #468

Closed wtomin closed 3 weeks ago

wtomin commented 2 months ago

Added new modules in mindone/diffusers/models/`. They are listed below:

module torch diffuers mindspore mindone
ImagePositionalEmbeddings Link Link
SinusoidalPositionalEmbedding Link Link
CombinedTimestepLabelEmbeddings Link Link
PixArtAlphaCombinedTimestepSizeEmbeddings Link Link
PatchEmbed Link Link
AdaLayerNorm Link Link
AdaLayerNormZero Link Link

These modules are strictly aligned, except the way to handle self.register_buffer .

For example, torch code :

self.register_buffer("pos_embed", torch.from_numpy(pos_embed).float().unsqueeze(0), persistent=False)

is changed to ms code:

self.pos_embed = ms.Parameter(ms.Tensor(pos_embed).float().unsqueeze(0), requires_grad=False)

When loading parameter dict, pos_embed can be loaded perfectly. However, if the user accidently set all parameters (including self.pos_embed to trainable, this can be a problem. Because in torch, self.register_buffer("pos_embed", xxx) is not trainable at all. It is not a nn.Parameter.

If change it to:

self.pos_embed = ms.Tensor(pos_embed)

It may raise error or warning when loading parameter dict. Do you have better suggestion?