Hardware auto-optimizing

[bghira]

Is your feature request related to a problem? Please describe. The current Diffusers library does not provide an out-of-the-box functionality to automatically apply system and model-level optimizations.

I often find myself manually tweaking system-level optimizations in PyTorch as well as model-specific ones such as VAE tiling and CPU offloading, which can be a tedious and error-prone task.

Over time, a lot of interface names or methods for optimization have come and gone, or no longer apply in certain situations. This leads to inconsistencies in the documentation, and published examples in tutorials. A single auto-optimize function could be applied in these cases, and the behaviour of that would transparently evolve over time.

Describe the solution you'd like I would like the Diffusers library to include a new module or helper function that can intelligently select and apply the most effective optimizations to both the system itself (using the relevant torch.* methods) and to a given pipeline.

For instance, a helper function could take in an instance of a pipeline and kargs containing a set of parameters, and then apply the selected optimizations. The parameters could include settings for things like enabling or disabling VAE tiling, whole model or sequential CPU offloading, gradients, torch_dtype, and others. This would make the optimization process more streamlined and user-friendly.

Describe alternatives you've considered One alternative is to manually apply these optimizations, as is currently done, but this can be complex and time-consuming, and it also risks inconsistent application of the optimizations across different parts of the code.

Another alternative could be the development of individual helper functions for each type of optimization, but this could lead to a bloated and less cohesive codebase.

Additional context This feature would be beneficial to both newcomers to the library, who may not be fully versed in the intricacies of these optimizations, and to experienced users, who would appreciate the time saved and the consistency ensured by this feature.

It would contribute to making the Diffusers library more accessible and efficient to use, and could potentially improve the performance of the models built with it.

Any new optimizations that make obvious sense can be enabled by default in this tool, which would seamlessly improve the experience for users, as they update.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[bghira]

this is still an active request

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[bghira]

not stale, you pesky bot