Analyse user study results

[rachanahegde]

Prototype Testing Analysis & Results

Note: I haven't added the comments from the sixth testing session but I'll add it directly to my dissertation draft instead.

Grouping comments users made while completing the tasks

Generating plot ideas

• P4: “[Deciding between the 2 plot ideas that were generated] Think I’m gonna go with the enchanted forest chase because I feel like it kind of progresses the story forward a little bit more. Whereas the library vault draws more from the first part of the story…I was thinking about kind of going back to the library but it doesn’t really unlock magic for everyone.”
• P4: [Referring to the first plot idea card] I feel like it concludes a little bit quickly on this one. Just retrieving the book and then returning to the forest. Whereas this one [second plot idea] they have the spell book even though they shouldn’t have it yet. But it does set up kind of a final confrontation between the priest and the orphan. I think I’ll go with this first prompt [referring to the first idea card]. I’ll go with this one and then kind of revise the ending a little bit. And then there’s a little bit of a heist here so that’s fun.

Grouping Interview Responses under Questions and/or Themes

Storyboarding feature pros

• P4: “It’s fun!” In reference to the storyboarding process.
• P7: User: I already wanna write dialogue now. (Participant expressing desire/urgency to start working on the story.)
• I plotting ideas when it comes to short stories or even stories in general, so I wouldn't be surprised even if I wanted to continue this story even after this video call. (User expressing desire to continue working on story.)
• This is phenomenal even if it's like plain, mundane and simple, it's sometimes as simple as the best way to get through. (Likes the simple interface)

Storyboarding feature helping to organise ideas and outline story effectively

• P6: “Feature is organised well, it’s clear, there’s a space for a title and a longer description and photo which I like. It’s not overly cluttered. It’s pretty simple and intuitive to use.”
• P6: “I also like that there’s templates too like the hero’s journey template and the five act structure - I think that could be helpful when you’re plotting a novel or short story.”
• P4: “I feel like it did. It was pretty easy to get my thoughts in order for each of them and it was nice to have it broken down so you can delve into the details in segments.”
• P5: A: Yes and no. I think it’s a good thing that you can divide your themes and have different story cards – it’s like having prompt cards so you can follow through with your story while you’re writing. Maybe you could benefit if you could move it around as well as because when you’re writing a story, sometimes you like to move around plot points. Q: Yes, you can re-arrange it. A: My bad, I didn’t see that. Then, it’s really helpful!
• P7 expressed that coming up with interesting ideas is hard because the ideas don’t always readily come to mind. This tool is helpful because it provides ideas which can be modified to fit the writer’s needs and they can identify useful elements in those initial ideas even if they aren’t perfect.
• P8: Ohh, I personally feel like this storyboard has definitely helped with outlining. If not given me more inspiration that I began with because they're like knowing that there are places where you can plot ideas or to test how far you want to create those ideas or experimenting, it's like you just feel so, like free spirited with it, because, for example, if I were to just write in a Google Docs and not have to use a storyboard, I'm obligated to making these ideas work. But over here the storyboard or to the generating plot idea, it's like, OK, there's so many possibilities towards these characters that it doesn't have to be just one way, one thing or like there's like so many it gives us so many possibilities to look forward to you that it's like OK, like, I'm here to have some fun and I'm here to enjoy the process. I feel like the storyboard helps us or helped me focus more on the process as opposed to the outcome of the story. Obviously there are outcomes in this at the end of the like at the end of the scene at the end of plot generating quite like it's like they're also part of the process because I get to decide what I feel most comfortable with in the story. So overall, I'd say, yeah, for sure it is definitely effective because you get to focus on the process a lot more than the outcome, which I'd say is really good for is well, it's kind of mindset I'm working on when it comes to creating art in any way, shape or form, so yeah.

Storyboarding feature helping to address writer’s block

• P6: “I think it would help address writer’s block.”
• P5: A: I think it could help with writer’s block in the sense that writers can put forth their ideas and there is no stress, especially since you can move around the story cards. In that sense it helps with writer’s block. Whatever comes to mind, you just put it on screen. Because of the AI and because you can generate plot ideas, it might help you think through stuff.
• P7: I would say it definitely can't hurt. I think there's a good chance it would help. Possibly dependent on the particular writing process of different authors. If there's someone who, OK, I'm just spit balling here, but if there's someone who just kind of writes off the top of their head and stuff just comes to them and they don't plot out stuff much in advance, I think this could be quite a significant help possibly. But if they generally work off of having to know their entire plot in advance for like 50 chapters. Like maybe for the mystery or something, for example, where everything has to be internally consistent. Then maybe it might be less helpful, but.

Plot idea generation feature helping to address writer’s block

• P6: “I think being able to generate plot ideas by putting in a prompt or having a couple of scenes and asking the form to generate more scenes – I think it’s a good way to overcome writer’s block. The two scenes that it came up with for this story were both pretty detailed and made sense with the flow of the story. They’re pretty good ideas.”
• P6: “Yes. If I didn’t have any ideas at all – if I just put in the genre and some simple premise. The fact that it could come up with pretty good plot ideas would help with writer’s block. And it’s at least something to start with that I can build on or change a bit if I wanted to.”
• P4: I think it would. I think the length [of the idea] was good. I think my concern if I fell into writer’s block is that I would maybe rely on it a little bit too heavily to shape the rest of the story. But I felt like the prompts that were generated weren’t so long that I felt like it was shaping the entire narrative but it did target specific sections that I felt stuck on – like kind of knowing where to go from the forest chase and then encountering a magical creature. I think in terms of like making those connections that was helpful.
• P8: Umm, I think that depends on the person. Regardless of AI being there to solve, being there to help us rejuvenate or being there to refresh our ideas, I think that's still depend on the person because few of us can be picky on some certain ideas and few of us may be a lot more flexible or maybe more comfortable with like what they get.Umm, but personally in my opinion I personally feel like it refreshes ones mind and it keeps us thinking about different things. It could definitely help with writers block. But then again, it all just depends on the person. For me, definitely. I haven't had like writers block yet, touch wood. But I do think would come across as really helpful when I do and I would definitely still feel like it refreshes my mind. So yes, the answer is yes either way.

Idea generation feature positive

• P5: One [first plot idea] is good.
• All of the participants added two of the generated ideas to the storyboard as two additional scenes.
• The ideas made sense for the story.
  • P6: “they’re both cool ideas” about the two generated ideas
• Participants made minimal modifications to the generated ideas they added to the storyboard.
• P4 expressed satisfaction after editing the plot idea that she added to the storyboard.
• P7: “ The first one is amazing. Let's see second one. OK, the picture is amazing.”
• P8 - However, the user did like the art style of a few of the images in the plot ideas generation section and screenshotted the images and ideas to reference them after the user testing session.
• P8 - She liked that the AI was imperfect because she felt it added a human touch and it made her anticipate the results for both image and plot idea generation.

Idea generation feature negative

• P5: But I suppose if someone is actually writing, they’d be frustrated because now it is as though they’ve got so many ideas. Maybe it’ll sort of get them to deviate from the storyline they thought of.
• P5: I don’t think there’s a reason why it would not help with writer’s block unless it’s something very specific – i.e. part of a plotline that’s troubling you. Then it might be a bit of a problem because AI is just going to fill in any plot holes. If you’re looking for something specific, it might take you five or six tries to get exactly what you were thinking of [from the AI generated ideas]. But on a whole, I think it’s a great help, yes.
• P8 - The images that illustrated the plot ideas misgendered one of the characters so the user was initially surprised by this.

Plot idea generation feature benefits

• P6: “The part where it came up with the idea that the orphan has rare abilities wasn’t really something I was originally thinking of but when I saw the idea, I was like that makes sense. Typical chosen one [narrative]. I do like the ideas it came up with. They were all positive and made sense and relied on the typical tropes without being predictable. At the very least, it’s a good starting place and I could build of off whatever idea it gave me.”
• How do you feel about the slight lag [for generating plot ideas]? P4: “I feel like it’s manageable because as I’m thinking about how I would piece through the story – I mean, I don’t think it’s a rushed process. If you were in a hurry for it, maybe it would be a little bit of a lag but I feel like as it is in the drafting process, I don’t think it the pause is disruptive.”

Plot idea generation influencing storyboarding/outlining process

• P4: “When I got to the first one I used – it was helpful because it built on what I had so far with the first half of it. I think that was entering the forest itself. But then it introduced new plot ideas like magical creatures or later – like the cat kind of being a comedic feature. Then I was able to go back to earlier characters or ideas I had and kind of reshape those AI plot points a little bit so maybe making interaction between creatures or the cat…then I think in the last section, I would say I feel like the conclusion piece – it kind of just wrapped up the story a little bit too quickly for me personally in terms of the AI prompt [referring to plot idea]. But in terms of key scenes, it was helpful in terms of coming up with some ideas on where to go next. But I feel like the conclusion – maybe I’d have to put that in as a specific prompt outside of the main plot.” – It affected the direction their story went in.
• P5: Yes. For example, one of the story cards right before the final plot – it was a scene set around a lighthouse. So it would be a confrontation scene around a lighthouse. That image I had in mind – stormy night and lighthouse. But I went through the [idea] generation, it changed the whole thing. The two options it gave me did mention the lighthouse but it made me think oh there’s one more plotline I could add right before the confrontation. In a way I guess it also helped the story. But I can see how it could deviate you from the idea you initially had because it would be a case of having too many options and not knowing what to choose. That could maybe make your writer’s block worse too. So it could go either way. Q: Right, so the person could become overwhelmed? A: Yes. It could either get you to become overwhelmed or maybe you would agree with it and go that’s a great idea, I should have thought of that! But if you’re asking, does it get you to deviate from your initial plan then yeah there’s a high chance it will.
• P7: Generated ideas were much more comedic which he didn’t originally have in mind. But he specified comedy in the form field so the AI leaned into that. He also said: “And the way it's sort of already gathering an adventuring party, it's kind of a very natural thing to do in terms of a fantasy plot. And definitely something you want to do in your story. You want more than just two characters to bounce off each other, especially if you're trying to make it funny.”
• So I like the inclusion of the witch and how it introduced her to the group. Felt felt pretty natural.
• Q: Do you think that the ideas that came from the AI were ones that you hadn't originally considered at all? P8: Yeah, definitely. Like I was just like, Oh my God. Wow, I did not even consider this like out of the words that I mentioned, the fact that they actually got the gist and they were able to, the AI was able to even take out even like sometimes to read between the lines. So yeah, adorned in a casual chic get up [reading from the generated prompt]. Like I didn't mention that exactly like those words in the prompt, but they were able to really read between the lines like this. Exude an aura of warmth and hopefulness. Like I didn't mention that on the prompt with the fact that the I got these little bits in details like bits and bobs just add up to that experience a lot more than I thought they were able to read. And I'm really glad they're able to figure out that it's a like a balance between traditional and modern look, despite being the 1990s, early 2000s, so on and so forth. So yes.

Q: Did you find the images that accompanied plot ideas useful or helpful?

• P6: “I think so. Even though they ended up changing the gender of the characters. When I was putting everything on the storyboard, I didn’t have a firm idea of what I wanted the characters to look like. Overall, the look of it and the vibe is what I was looking for even if the specific characters themselves weren’t what I ended up going with. The overall look was pretty good.”
• P4: “The ones that I remember – I guess I was kind of indifferent on the images that were generated there. But I think it was helpful to spark a little inspiration by seeing the picture and the text. For this one that I changed [referring to scene card], seeing the creatures drawn to the light for example kind of sparked ideas for me. So I do think it enhanced the prompt [the idea] that was generated. “
• P5: No I think, for me, the images did not go with what I was thinking. That could also be because of the lack of information given [in the plot ideas form]. It did not always connect to what the setting and the premise was. It filled in blanks by itself so the image that it generated and what was in my mind were two different things. Q: So it’s too generic, not specific enough to the actual story. A: Yeah, it would not pick up exact keywords from the premise. I think if it could pick up the right keywords, it might have generated a more apt image. But the image generated in the plot ideas was more generic. Q: Do you think it would be better to not include the images? A: No, I think images are great. I don’t think just writing [ideas] would be any help. Images are good even if it’s a generic image, it gives a good visual look and it helps the thought process. Also because you’ve got separate image generation and you can save that image and then come back here [to the storyboard] and insert that – that’s much better. The image generated there [in the image generation page] is much better. There’s an alternative way. So there’s no need to get rid of the images on the storyboard altogether. I think that’s not an issue at all.
• P7: Honestly, no, they didn't help much. Thought they were too vague, didn’t invoke any particular emotion and the text does more work here.
• P8: Ooh yes. Again, again, since I was looking forward to a Sapphic relationship, I'm really glad they came with this at the very end, but my favorite I if I'm being very picky - this is my favorite one.I didn't expect that to come out there and I'm really glad they did come up with this because this is chef’s kiss. It's spot on. [Referring to the image of the two women sitting in a park having a picnic and looking at each other.] We love this. We love to hear this and I'm really glad that it came up that the software came up with this and the title too. Yes, yes. And this one to like the fact that they're actually like, I just really love these two like the images, because they really do add up to the prompt and they really do inspire the person on this like me, for instance, to really come up with, like, a full picture of the storyline.

Image generation feature benefits

• P6: “Being able to visualise what the character looks like and you don’t have to put in a bunch of detail – you can just put in the basics and then have the website come up with something that’s pretty detailed and realistic would help with coming up with characters too.”
• P4: Yes, I feel like the form guides you through thinking about different elements of the character process and also the [generating] plot ideas [form] – thinking about genre, themes, [and] setting is also helpful because I don’t know that I’d have the list at the top of my head if I was trying to use AI to help with writer’s block. So I think that guidance was helpful.
• P5 User: I’m having so much fun with this.
• P5: Liked that the previous prompt and image pairs were not deleted and the prompting history was visible. She said “Sometimes with this sort of process, it’s nice if you can have all your edits [previous images].”
• Q: Yes, do the images make it easier for you to describe the story?

P5: Definitely. It also helps you segregate between the main themes and main plot if you’ve got an image for each card. It’ll definitely help you visualise the story and immerse yourself better rather than for when you don’t have any sort of image. Again, it might get overwhelming because when you write something, you have to create a world in your head. So the images definitely help with that.

• Q: Do you think the images you generated were able to capture the emotion or mood inside the scene?

P5: Yeah, it definitely captured the mood and emotions. It did a good job on that.

• P7: Personalities I think are the most important thing. Even if the elements aren't all lined up, that's easy to imagine, but. Actually like visualising. A character and their expressions and what they look like, I think is it tends to be when you read the hard thing. So I know whenever I read anything I only have like vague ideas of what people actually look like. It's the personality that comes across in dialogue and stuff like that, right? So this I think really helps you solidify people's characters in your mind.

Image generation feature positives

• Participants were generally satisfied with the generated character description and the generated image prompt.
  • They commented that the description was “cool” and “interesting.”
• Participants added at least one of the generated images to the storyboard.
• Participants reacted positively to generated images.
  • P4: “Woah this is cool. I actually think this captures it pretty well.”
  • P4: “Perfect. The cat is there.”
• P7 stated that the personality is most important aspect of image – “I forgot to make him run away. Oh, well, whatever. There we go. That looks better. Now. He looks less nice. Anyway, no, I mean, I think I'm, I'm fine with this anyway. It depicts the personality better and that I think is the most important part.”
• Participant liked the details included in the image - P7: It's pretty good. It looks like the donkey has been shot, but I really like that. That is a tiny cart that is perfectly fitting for a librarian. Yeah, and he looks tired. Perfect. If the thing at the bottom is supposed to be the magic book which has just dropped it on the ground, which also fits him perfectly.
• P8 - User stated that the images were becoming closer to her vision with each round of image generation.
• P8 - Helped to have the flexibility to take out what stood out about the character and use the images as a reference while outlining. Also the atmosphere was helpful. “I [can] use the bits and pieces of the characters in terms of the outlining for me.”

Image generation feature negatives

• P5: [looking at image] Yes, more. Why is the envelope just there? Yeah, it could use some work. It’s captured the character right but probably not the scene. I think it’ll be fine with just one more [regeneration].
  • P5 pointed out that sometimes objects would be positioned incorrectly in the image and this would affect her ability to visualize the scene. Also, when the user failed to specify the character’s actions (i.e. reading vs writing), they tended to behave in ways that didn’t fit the writer’s vision. Understanding the necessity of prompt specificity was something that users understood over the course of the testing session.
• P8 - User was struggling to generate images of queer characters because she wanted to depict the character’s interactions while also conveying some romantic subtext and sometimes the AI would assume that one of the characters was a man or it would fail to explicitly capture the love between them in the way she was looking for.
• P8 - User was pleased with how the images captured some of the details about the character’s clothing or appearance but other times the wrong character would be wearing the clothing or the sapphic representation would be missing. Unfortunately, user didn’t always explicitly specify the character’s gender so the model generated an image of a man and a woman.

Q: Do you think the images were helpful for plotting your story?

• P6 “The image generation helped more with visualising the characters and what they would look like and setting the location and everything. Yeah, I guess maybe not so much in terms of the actual plot itself, but just with the characters. But I think knowing the characters and being able to actually visualize them and see them, it does help a lot with developing them and personality and all that and what you want them to look like, which I think is important.” [Image generation feature more helpful for character development than story development.]
• P4: I felt like they were pretty helpful. I tend to be a fairly visual person so I think it kind of sparks some inspiration for shaping character personality or establishing the setting. Yeah I think it was helpful to have the [generated] images. I felt like the prompts worked pretty well in terms of what I was envisioning on the spot for a character. So I feel like it integrated pretty well for the sections I had so far here [on the storyboard].
• P4: I feel like they [the images] mostly fit with what I had initially. But I think if I started with the characters before going to the plot itself then that [the image generation] might shape it a little more. With the first image that came up – this did change how I might show the magic appearing in the world. I think initially I said there would be a glow to the character. But maybe having some glowing spirit also works. So just kind of working with the magic system – that’s something I would change from that [the image]. But generally it tended to follow what I had initially.
• P5: A: I wouldn’t say it’s useful for brainstorming. If you’re lucky enough, the [idea] generation on the storyboard would do that for you than the image generation tab. I think this one [image generation page] is much more specific. Unless you’re being very general in the form and the prompt that you give, it’s going to give you more or less what you ask for. But I guess if you’re going very generic, it could help with giving you ideas. Mostly I think it just goes with whatever you’re going for. I think you need to have a certain idea in mind. It’s more useful when you have an image in mind and then you’re working towards getting that particular result rather than wait for the AI to give you ideas.
• Q: Based on what you were saying, you would say that the images didn’t really inspire any changes or additions to your existing story outline?

P5: No it did not. It could be a personal thing. Maybe some people are less visual so they would go with something generic and work from there. Someone who is more detail oriented and a visual thinker might already have an idea in mind as they’re writing a story. So they would go for trying to achieve that result. For me, it was the latter – so it did not inspire anything because I wasn’t looking for it. I was trying to build the exact image or a likeness to what I had in mind so I added a lot of details.

• P7: They didn’t affect the plot but he said he’d keep looking back at the images to retain the atmosphere and mood of the story as well as the personality of the old man Wilhelm. They gave him something to work off and calibrated the character’s personality so it doesn’t drift away over time and the images also set the tone for the story.

Q: Did you find the main character recognizable across multiple images? If there were slight differences, did it affect your storytelling process?

• P4:I think they’re identifiably consistent – if I had any issue with that, I might go back and just change one or two [images] in a longer story. With the one that I ended up not using for the forest scene, it was pretty easy to make some adjustments in terms of the prompt I put in – as in how I described the villain and what expression I wanted the character to have. I think you can create the images fairly quickly. I think changing the expressions is the only thing I’d say I changed. But otherwise, I felt character was identifiable.
• P5: I think he was pretty recognizable. I think it could depend on how many characters you’ve put in. I’ve just generated a scene where only the main character is on the screen. Also I think the photo angle affects the image generation. For example, wide angle versus close up. If you’re in a wide angle, it gives you more of the environment the character is in and the background image as well. So that could inspire changes to the plotline. But if it’s a closeup, it’s more focused on the main character and their face and physical appearance.
• P7: A: Well, I I think that, yeah, there are definitely differences. They're significant. They're not the worst. Basically, the way I approached it was. I like having initially having a couple different looks for him. But then I I would probably settle on this last one. Last one. And have this be the mental picture of of him in my head. So then then the other ones stand out a little bit more as being a little bit more jarring and maybe not fitting the image I have in my head, right, like this guy looks fairly calm and tired and normal. And if I'm imagining this as, as Wilhelm, our main character, it's a little bit jarring. So I think I mean if it was possible, it would be useful. To ensure more consistency.

Justification for decision not to add AI generated image(s) to storyboard

• P4: That one (the image she didn’t include) was missing the cat which I forgot to add in. Also they made the elvish priest kind of look like some other kind of creature. I think I specified the age on that too much. So this one [the image she included] seemed more accurate to like a somewhat fearful pursuit. Wheras the other one [image] it looked like it would come later in the story. The villain didn’t look quite right and there were some pieces missing like the book bag and the cat.

Image generation helping to address writer’s block

• P6: “I think so. I think coming up with a character and knowing what they like look like and stuff can be difficult, especially when you're just starting to write a story and you don't really know what exactly is going to happen and what the character, what their personality is like and stuff. So I feel like by being able to see them it helps a lot. When you can actually like physically or visually see them, you're like, oh, I can see what kind of personality they have. Well, it's just sort of easier to come up with their voice and what actions they would do, like how they would respond to certain situations when you can actually see them. And also I think the prompts as well kind of force you to think about things like when the character prompt was like oh what’s their personality like - like what would they wear and stuff – then it forces you to actually think about that kind of stuff like what they would wear in their personality. Whereas sometimes if you're just writing you may not necessarily be thinking about small things like oh what accessories are they wearing and things like that. So I think it's helpful in that sense.”
• P4:I think so. In terms of coming up with characters or the scenery, it would definitely personally help me either be inspired in a certain direction or just have a clearer vision of what I was trying to put out into the world.
• P5: It could help. Like I said, if you’ve got an image in your mind – it helps when you’re able to visualize. It could help writer’s block because you can get all your ideas down – especially because it has this history feature where it doesn’t just delete what you’ve done before. That way it could help you make a mood board of sorts and help you pick the exact field that you’re going for. But at the same time, I don’t think it’s one of the main things that would help. It could be a bonus at best but whether it helps or not is up to chance because again it depends on whether you’re going for a particular description or if you’ve gone for a generic one.
• P7: Yes, maybe a little. I don't think much. I think as a writer. I for for writer's block specifically. I think it's a matter of ideas. But the images. Convey sort of mood and atmosphere very well. I think that's what they really add to the text. So even if perhaps the text is not quite what you're looking for, it's not really grabbing you as a writer. Maybe just a sort of vibes you're getting from the image. Well, then let you spin off into some new idea.
• P8: Ohh yeah, absolutely for sure. This last image here I'm just biased that last because that's just so cool. I could use this elsewhere. I could like try new things and like that's why it's always handy. Like I recommend like I keep like a paper next to me, so I'm like, OK, put that idea down. So I don't lose it because it's like, Oh my God, we need like to keep testing to keep experimenting and giving ourselves, giving out of that freedom to just make and all because of an image can make us feel like that is really cool. So yeah, I'd definitely say yes it does.

Comparing PlotPalette to other AI Writing Tools

• P4: But I feel like this [tool] seemed pretty targeted to the fantasy genre, in this case. I felt like the image generation was helpful with the types of fantasy and cartoon design. I think it matched pretty well to the vibe of the story. In terms of the textual AI, it also seemed pretty seamlessly integrated into both the genre and age group of the story. Initially it felt like a middle grade series that I would lean into and the AI picked up on that for both the images that were generated and the way the story progressed. I felt that it matched up with my original input pretty well.
• P5: A: No reason except that ChatGPT is the main one. They’re constantly getting in new things. Recently, they added a feature where you explore GPTs and you can try other GPTs for writing and productivity. It’s also using plugins so chatGPT gives you a lot of variety. So that’s why it’s the best. You can get several things under one platform. If you’re talking about creative writing – there are a lot of requirements – and you can cater to a lot of those requirements under just one platform instead of having several tabs open. I think it’s just based on convenience right now. Main point of comparison I can see is that this prototype is very niche so it has a very niche audience and consumer base. If we go for ChatGPT versus this, this [website] is very niche. The pro is that because it’s niche, it focuses completely on that [writing] so there’s no unnecessary focus on anything else. The con could also be that it’s got a restricted consumer base and audience whereas chatGPT can be used by anyone from any walk of life.
• P7: A: Yes, I like dabbled with one website that was. Basically like a a chat bot that also did description. There were a couple back and forth messages. It was not too imaginative, didn't do enough description, that kind of thing. I don't know this is a while ago. They were kind of for different purposes because it was. It was definitely more of a a back and forth like you would type in a sentence or two of dialogue and description and then it would feed you something back to whoever you were talking to and their responses. Right. And I just found it to not be very imaginative. It quite tightly followed whatever tone you were setting, whatever actions you were setting, it didn't really do much independently. I think this [protoype] is better in that sense. It definitely springs out completely new ideas, which is, you know, the point. So it might just be a matter of of purpose but. I mean this to me just also in general I, I I like the higher creativity more.

Integrating PlotPalette into Writing Process

• P6: “Yeah. I feel like if I were ever stuck, I would use it just to see what ideas it came up with and then I'd probably take that idea and make it my own somehow. I think I also, yeah, I like it as well for the character generation part. Cause I think it can be helpful to like, yeah, just be able to like visualize your character and then it just helps them seem more real to you.”
• P4: I think so. I feel like it would be helpful in terms of planning and also for further character development.
• P5: A: If I was a mainstream writer, it would be beneficial. It would need constant reviewing to make sure you’re up to date with requirements of writers. That would mean you’d have to constantly take reviews and work on that. But it would be a great idea because creative folks like having a moodboard or vision board to visualize where this project is going. At the end of the day, writing a novel is a project. In that sense, it’s a project management tool for writers. So I think it would be helpful if you could add in better features but I think the prototype is a good base.
• P7: So I would say for original works it might be helpful. 'Cause I mostly write fan fiction or, well, I say write. Fought out ideas for fan fiction stories. Because they're so. So very specific and like you have. Specific world building and characters and and things you're working with that. The AI tools just won't know to use. They will. They will constantly try to just. Go into a sort of more generic route, so I think it would be less useful for that kind of thing where you're working in a specific IP. But for original stuff. It helps plot things out for sure. But I think that's the main draw of this tool is plot ideas. So if you're trying to come up with new characters, for example. Or or I don't know. Drama twist or something? Maybe it's not the best for that. But that's also not really the purpose. If it's meant to get you around wires blocked, then plot is what you want, I think.
• P8: Umm, I do see the potential benefits like I just get really excited because it's like a way of making fan art.

For your, for your own original characters and seeing how you know like you could like seeing your characters come to life through an AI generated image or through writing of its own, and like knowing that the way I put my thoughts can be clearly seen by the AI.

And the image actually comes through when you're like, even if I didn't anticipate this, it's like, OK, it feels sort of like. Wow, these are my characters out there.

They're thriving, they're vibing and we love to see these characters out there.

They actually have a face, a shape, and it's not just in my head anymore. It's actually in the screen which I can keep coming back to for reference when it comes to developing them and developing their characters throughout the story or something like that. Another benefit that I can quickly think of at the moment is like it also makes you think that there is no box. Why keep a box in an art form in a in creative writing at itself, which is really cool, and also you get to see that, yeah, we do tend to say that AI takes out on originality, but sometimes when you check at it, you're like there is some imperfectness, especially when it's made by a human being another human soul, because you're like, OK, we are both imperfect on the situation.

Usability Challenges: Any difficulties in using the application.

• P6: “It was pretty intuitive. Yeah, once you explained it to me, then it became pretty easy to use. Yeah, I guess I don't think I have any sort of notes.”
• P4: In terms of the written content, I felt that it was nice that there was a feature to go in and edit the AI prompt or output. I felt that it had just enough inspiration for what I needed and then I was able to keep some of it [the plot idea] and put my own words in there as well. In terms of the images, the only thing I would say was figuring out how specific to be and the level of detail [for the prompt]. I think that’s probably something to play around with but – I feel like for the most part the expressions I wanted were generated pretty well. But I feel like I’d have to try curiosity vs wonder vs excitement vs fear – kind of identifying those emotions that might be closely connected in an expression. I’d have to do it more times [generate the image] to see what the differences were on that. But I think it was pretty easy to go back on the one [the image] that I did change to match the initial image I had in my head.

Positive Feedback: What users liked about the tool.

• P7: Not hard to translate ideas into prompts.
• I felt really engaged with both the storyboarding and image generation throughout the whole process. As someone who struggles with writer's block frequently, the prompts were often just enough to spark the next idea I needed to move the story along.
• P7: I have to say it turned out a lot more interesting than I was expecting so. I was expecting the the generated story beats to be a little bit, you know, flat and predictable or something like that. But I think it it probably helped that. I had a decent idea for a main character. If I'd made a generic main character, I wouldn't have had as much fun with it. I did kind of set out to subvert the trope and the prompt right. So I feel like if I had just gone straight for fantasy prompt #1, I would've just had less fun with the process because I was less invested in. The story and the characters. So I think it's important that you don't just leave everything up to the AI, because if you don't make some decisions yourself. You just won't feel as invested and they won't feel like your characters and stuff, so I. Well, OK, again theorising, but. I think if we did it again and I just followed the prompt entirely, I wouldn't feel as attached. I wouldn't care what was happening. (comment about the importance of providing creative autonomy to user and to encourage them not to rely heavily on the AI tool to ensure they still feel invested in the outcome of their creative work.)
• P6: “This is a really cool tool to use if you need to visualize a character or come up with ideas. The images are really cool and they are pretty similar to what I was imagining.”
• Two participants expressed the desire to continue using the website to work on their current or future writing project and generate more images/artwork for their writing project. One participant returned to the website after the testing session to generate more images of characters from the story she started writing during the testing session in order to develop a few more characters.

Negative Feedback: What users disliked.

• Image generation inaccuracies - There were discrepancies in the character’s appearance compared to the prompt that was entered – P8 pointed out that the skirt is so short it would be shorts and the tattoos were in the wrong place on the character’s body.

Improvements Suggested: Specific suggestions for improving the tool.

• If possible, the one feature I think would be helpful to add in the storyboard is some sort of drop-down feature to add notes or subsections (to help with organization on longer stories).
• streamlining the image generation prompt inputs would allow for quicker brainstorming but otherwise great
• P5: Maybe you could think of adding an option to delete the whole thing [all the scene cards] as well or deleting a selection of cards if you want to delete more than one card. That would be good instead of doing delete scene separately [multiple times] just to make it more convenient.
• P4: I think for a short story, the boxes as they were were well organised and I was able to structure my thoughts pretty efficiently. But I think it could be helpful to have subpoints within the box. Like if you had a dropdown box within one of these [referring to the scene card] to kind of put some notes. I think in the first segment, it might be nice to have a space to put some ideas about the function of the library or a random thought that might not have a place somewhere else in the storyboard yet without necessarily interfering with the main plot. Q: Like annotating your ideas? A: Yeah a little bit. You know how sometimes there’s the main text box and it has a dropdown feature. I feel like that might be helpful within this to organise subpoints within that box. But I think for a shorter story, as it is [the page] is fine for organisation. Q: You mean like if you’re writing a novel, you can leave little notes to yourself? A: Yes.
• P4: The only thing would be making some kind of conclusion prompt [for generating plot ideas]. I guess I could specify that in the premise that the story is finishing or this is the main point of the story but I can’t think of anything that was glaringly frustrating or didn’t work super well. Q: So you’re saying it would be helpful to have writing ideas specifically for the ending versus the rest of the story. A: Yeah like also in terms of specifying the rising action or the introduction [stages of writing process]. If there was some way to indicate that [in plot ideas form], it would be helpful for placing the new prompts [ideas] into the story.
• P7: A: So I mean, you saw me when I I wanted to just generate the same scene again and see what it came up with. It didn't save all the data, so it'd be nice if there was a regenerate button that'd be quite nice.
• Streamlining the image generation forms a little (maybe hiding some things under a drop down so users could add more or less information depending on what they want without getting overwhelmed by all the fields).
• P7: I think there's too much overlap between expression and demeanour. I wasn't really sure. I don't have that much experience with with image generation, but I feel like they would both kind of serve the same thing in terms of what the image generation is trying to do. So I'm I'm wondering if maybe just combine these two or something would make more sense. Mood I also found - I was thinking this may be a little extraneous depending on what you're trying to do. I think if you wanna use it, that's good. For things like I don't know framing or something like that, a scene just giving it some some overall atmosphere. But when you're just trying to do something a little bit more generic, maybe like I was for my my first. First image where Wilhelm was was meant to drop a book down a well. I think the fact that I put the mood is humorous, maybe contributed to this being a little bit outside of the the vibe in personality. I was looking for. So honestly, maybe I should have just left that blank or left that out entirely, and it'd be a fairly neutral scene.
• P7: Also, maybe saving the character information so you don't have to retype it to just adjust one thing [would be helpful].

Usability Issues

• Issues with buttons: the ‘next’ button wasn’t working; also unclear to the user when some buttons were clicked on the storyboard page so they would click multiple times (which leads to multiple API calls and more $$ wasted).
• Issue with the scene card – when the user deletes all the text and the page saves, the HTML element for entering the text then disappears. So the user is unable to edit the scene description for that scene card and they have to delete it and start over.
• Add a loading bar so the user knows that they don’t need to click the button again to generate ideas or images, etc.
• Size of form fields is too small – hard to see all the text that is written. Make it expand as the word count increases.
• Could include more tips or guidance for the user within the interface – i.e. telling the user that they should craft a prompt for generating one image for ONE scene at a time rather than multiple scenes; telling user to specify character action within the prompt if they are using the consistent character generation feature. [Interface re-design suggestion]

Areas for future work

• Area for future work: preventing plot holes created by adding AI generated ideas to storyboard.
  • User #5 noted that one of the ideas that she liked which featured the setting she had specified in the idea generation form would create a plot hole in the storyline if she added it to the storyboard. Therefore, it would be useful to include additional information to accompany each idea that makes it easier to integrate suggested ideas into the existing plotline (i.e. a list of 1-3 scenes that should precede the idea in order to justify the events that take place in it) and/or some way of flagging aspects of the plot idea to users so that they are aware that certain information within it will lead to a plot hole. For example, if the user is warned that a certain line within the idea will create a plot hole, they can address that immediately.
• Make it possible for user to edit the generated character description directly.
• Using an image2image to improve character consistency in images.

Observations for image generation and prompting

• Users (#5, etc.) were unclear about how to phrase the prompt at first but were advised to describe exactly what was happening in the image. Users were confused whether to use natural language or special keywords and expressed a lack of experience with using image generators. They also weren’t sure how to describe multiple characters in the prompt (aside from the main character) without confusing the AI. Users also became more confident after generating 1-2 images and were able to express their ideas in the prompt field more quickly.
• Participant also confused about whether to describe all the scenes from the storyboard in the prompt field for generating the image.

Potential for improvement

• User unclear about which user-generated content was used to guide the AI – Participant 7 asked if the scenes in the storyboard would be used for the AI image generation and I explained that they would only be used for generating plot ideas. Would be useful to provide future users more clarity on this because Participant 7 then commented that they wouldn’t have to worry about how they phrase their ideas if it won’t affect how the images are created for the storyboard.