Concerns about Transformer2DModelOutput and future censorship of models.

[311-code]

This may be a little pre-mature but want to get ahead of it. I've been following the recent changes in the diffusers library here, especially decisions to move to standardize output classes like Transformer2DModelOutput. You have all probably seen the deprecation warnings about this. While I see the benefits of consistency and it being easier maintain, I have some concerns about potential downsides. Let's start with the cons since that's the main point here.

Cons:

Potential Censorship: Centralized control by "standardizing" leading to certain models or outputs being restricted or deprecated based on subjective criteria determined by repo maintainer's here. This could limit the diversity of AI models, which is worrying. It's like the" frog in boiling water" scenario, small changes may seem useful now, but they could gradually lead to significant restrictions.

Inflexibility: Harder to implement custom solutions or pipelines that don't fit the new standardized format.

Forced Updates: Deprecation of features or output classes forces updates that may not benefit many use cases.

While these features seem beneficial at first glance, they could slowly lead to censorship. AI models are already censored enough imo, and we all know what excessive censorship can do:

Lessening community control in this area could lead to future restrictions stifling innovation and creativity. And as we know diffusers is a backbone and affects all models including .safetensors and so many platforms running on this.

Ok now finally for the Pros.

Pros:

Consistent interfaces Simplified integration Reduced errors Easier maintenance

My suggestions:

Backward Compatibility: Maintain backward compatibility as much as possible to ease transitions.

Custom Output Options: Allow custom outputs to integrate seamlessly with the framework without having to rewrite everything if you made a custom pipeline and suddenly something was axed by diffusers.

Transparent Decision-Making: Engage with the community about deprecations and updates to mitigate any negative impacts of this new standard and prevent unwarranted censorship. Do any of you remember the safety_checker in the beginning of this? I sort of feel like this could end up mostly like that but in slower form.

I believe it's important to strike a balance between standardization and flexibility to keep the framework innovative, but without imposing unnecessary restrictions (or the potential for it in the future). We should work together to try to maintain an open and flexible ecosystem.

Addressing Forking:

It's true that developers can fork the project if they disagree with the direction taken by the maintainers here. However, forking is not always a practical solution.

Community Fragmentation: Forking fragments the diffusers community, leading to duplicated efforts/divided resources. This fragmentation can slow down progress.

Maintenance Burden: Maintaining a forked version requires significant effort. It involves not only keeping up with upstream changes but also ensuring compatibility and fixing bugs independently.

Resource Constraints: Smaller teams or individual developers might not have the resources to maintain a fork effectively. They rely on the collective effort and support of the broader community and maintainers.

Ecosystem Support: The broader ecosystem, including plugins, integrations, and tools, is often built around the main project. A forked version might lack the necessary support and compatibility with these ecosystem components.

Forking is a valuable in open source development, but not a cure all. I think it's important to strive for balanced approach within the main project and address concerns to maintain a unification/creativity in this space.

I did review thediffusers GitHub repository discussions on this, and it seems that standardization of output classes like Transformer2DModelOutput is not really talked about right now.. and I just wanted to get ahead of it. I think this may not be widely recognized yet here as a potential for future censorships that could maybe happen. But by bringing the points to light we can encourage discussing it and make sure the framework evolves in a way that supports many of use cases and also maintains the flexibility needed for creative innovation.

A flexible and open framework that supports diverse use without imposing restrictive updates that are also convenient updates (actually pretty convenient though.. which makes this more difficult) but we also don't want this to hinder things. Again, I may be getting ahead of myself here.. but I just felt it was worth bringing up and is a warranted concern.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.