amotile / stable-diffusion-studio

An animation focused workflow frontend for Stable Diffusion
MIT License
478 stars 30 forks source link

Suggestion for how frames and transitions could be presented. #5

Open Petri3D opened 2 years ago

Petri3D commented 2 years ago

Seems like the prompt is behaving oddly. The original "fantasy landscape" may linger there in the start, even if removed. Also, It seems like the prompt get stored into both the prompt and negative prompt at keyframes.

amotile commented 2 years ago

Yea, I messed something up. will release a new version tomorrow.

mischaschaub commented 2 years ago

Great. Yeah, you forgot to include separate categories for "prompt" and "negative prompt " in the workshop windows, contrary to the playback tab. Was thinking while walking.

1) Basically your solution seems rather closer to the powerpoint/keynote approach than to video-editing apps, because you have just stills and transitions. Important are the slides which have been created in the workshop. The SD-calculated in-betweens are just a sophisticated transition.

2) So they are just a nuisance in the interface, much too extended for a real-world use. It would be super great to just have the slides with some kind of standardized time for in between's, let's say one or two seconds, what just could be represented by a number-input for the desired amount of frames. So the user has enough screen space to arrange his slides in the best order for his story, to remove or insert slides, single ones or as a group. And he could fine-tune the numbers for the in-between-frames, and the whole movie would adapt accordingly.

3) The user could hover over such a number field to preview the in-between-clip.

4) Most important, the app should permanently be working on rendering the in-betweens, the machine should never wait for your orders, because this means that you will have to wait while it does its tricks.

5) I think that such an app would be rather future-proof, as there will evolve a whole set of transition tools to be added, for instance a zoom tool, or a rotation around the y-axis, and certainly a time function, where the user could design a curve for the transition timing.

6) You could import a soundtrack and then position your slides to relevant moments of the soundtrack, and the program would fill out the transitions.

So I hope you are not vexed by my ideas from the forest! I am just unable to do these with my background as a designer.

amotile commented 2 years ago

The prompt bugs is hopefully fixed now. I will retitle this issue and relabel as a suggestion.

I do admit I am a bit confused but the suggestion, perhaps some simple mockups would help.

Petri3D commented 2 years ago

The binary seems to work, In the workshop it is showing the negative prompt keying as well (top-left).

However, I tried first with the source package, but the problem exists there, I took the zip file, did npm install, npm start:

Loading AUTOMATIC1111 config from: http://localhost:7860/config Socket listening on ws://localhost:4001/ Server listening on http://localhost:4000/

started ok, but the UI did not have the negative prompt keying.

Maybe user error, I dont know.

mischaschaub commented 2 years ago

I do admit I am a bit confused but the suggestion, perhaps some simple mockups would help.

Bildschirmfoto 2022-10-23 um 09 10 17

So with this mockup I tried to show how much more useful info you could squeeze in the window. Once you start composing a longer movie, it becomes critical to have as much as possible of the grand view of the full project. The user could pull a keyframe to another position in the timeline, and the old transitions would be automatically erased, and new ones would be generated automatically in the new location of the keyframe. The user could change the qualities of the transition, such as duration, etc. The user could click on a transition and the related sequence of starting keyframe and transition and ending keyframe would be shown. To fine-tune the timing of a movie, it would suffice to just change the duration of one of the transitions, and the later frames would follow along.