Closed ninavizz closed 11 months ago
ninavizz
commented
3 years ago P.S.: Unlike the other "general" survey, this one will only be live for 14 days. FYI. If folks have feedback about the survey itself, pls share it here. We won't be blasting it on Twitter or the forums, until tomorrow, so that minor changes can be added to it that folks might flag, here. TY all for being a wonderful community to work with!
andrewdavidwong
commented
3 years ago Feedback:
QubesOS username here on GitHub and on Twitter).ninavizz
commented
3 years ago @andrewdavidwong TY for taking the time to comb through my text, and for the helpful suggestions! I have implemented them all, as well as the updated image links. I also smoothed-out some of the other bits of language that may have sounded too conceptual (Marta cited a few things as strange to her not-English-first brain, and our few user answers so far indicated the same language could be simplified).
So far Jackie and Bessie are neck-and-neck, and my favorite feature between both of the prototypes is getting the most negative response. Why research is so important, lol!
I'll post to the forums tonight, and will email you with the Twitter version of the blast, too. Thx Andrew! :)
deeplow
commented
3 years ago Feedback: It's kind of hard to map the text explanations to the visual elements in the menu. I wish before the big table there were two pictures (one for each) where each question would be numbered and the picture would have numbered labels and arrows to the corresponding question number (1).
But making this would probably be a lot of work and LimeSurvey may not support dropping in images like that.
ninavizz
commented
3 years ago @deeplow Lime makes it regrettably difficult. I'll certainly give that a second look tho, as you being a not-English-first human only reitterates the urgency of that. @marmarta initially flagged a concern to me that my written delineation of features felt a bit vaguesauce to her... and her grasp of language is pretty phenomenal. But, she also comes to English as a Slavic-native speaker.
I'll ping here when there's an update!
ninavizz
commented
3 years ago Hey @deeplow! Thanks again for the comment and suggestion. I revisited it... and after some thought, decided not to do pictures. Honestly, if something did not make a strong impression with folks, I'm ok with not getting their feedback on a detail. The objective with that question, is purely to get a measure on things folks had extreme feels around—one way or another.
No design solution will ever please everyone, but my primary goal with this survey is to get the best measure I can from a survey, on community sentiment. We also cannot measure true usability, by asking folks to watch videos. Much more feedback can happen, later, when I can do in-person user testing from a laptop with a beta from Marta as its functioning app menu.
TY, again! :)
jpouellet
commented
3 years ago I have no doubt that surveys are very useful, and that collecting private feedback is also useful, but I also know from direct experience that it is very useful to be able to answer questions of the form "what information was considered as the basis for X decision with Y outcome?" by consulting public discussion archives. Answers of the form "because private information you don't have access to indicated we should" are not welcoming or encouraging to future contributors, and makes open source projects feel less open and inclusive.
This prompted my desire to leave feedback in public on github instead of in the survey or via mail to @.***
Perhaps future surveys could come with some statement of or perhaps user-selectable choice about how the results may/may not be published after?
Additional feedback which was more regarding the survey itself than the app menu designs, which I figured more appropriate to leave here rather than in #5677:
There were a couple questions regarding aspects which I truthfully did not notice in the videos, but that does not mean I would not appreciate / do not have thoughts on such a feature, causing me to rewatch the videos to try to notice. There did not appear to be a way to indicate both simultaneously, and it wasn't indicated which should take precedence when answering (e.g. unclear whether you more interested in whether something was conveyed through the video, or whether the given feature/attribute was apparent/intuitive/noticeable from the mockup, or what users would think of such a feature/attribute in the abstract when divorced from the particular implementation). I suspect I might not be the only survey responder with this dilemma. It may be ambiguous from the quantitative results of the survey as written how many answers of "didn't notice" might be more a reflection on the video format, as opposed to conflating it with a response of "neutral" / don't care about the feature/attribute in general.
This was the case for me for at least the "Seeing the template for each qube in its sub-menu", "User created (custom) groups for qubes", and "User created (custom) groups for apps" questions.
jpouellet
commented
3 years ago I forgot that github treats --- as a signature separator (and hides it) and instead of
Additional feedback which was more regarding the survey itself than the app menu designs, which I figured more appropriate to leave here rather than in #5677:
There were a couple questions regarding aspects which I truthfully did not notice in the videos, but that does not mean I would not appreciate / do not have thoughts on such a feature, causing me to rewatch the videos to try to notice. There did not appear to be a way to indicate both simultaneously, and it wasn't indicated which should take precedence when answering (e.g. unclear whether you more interested in whether something was conveyed through the video, or whether the given feature/attribute was apparent/intuitive/noticeable from the mockup, or what users would think of such a feature/attribute in the abstract when divorced from the particular implementation). I suspect I might not be the only survey responder with this dilemma. It may be ambiguous from the quantitative results of the survey as written how many answers of "didn't notice" might be more a reflection on the video format, as opposed to conflating it with a response of "neutral" / don't care about the feature/attribute in general.
This was the case for me for at least the "Seeing the template for each qube in its sub-menu", "User created (custom) groups for qubes", and "User created (custom) groups for apps" questions.
deeplow
commented
3 years ago Perhaps future surveys could come with some statement of or perhaps user-selectable choice about how the results may/may not be published after?
I do not agree with your assertion. Nothing is preventing people like you from giving feedback publicly. Privacy has a big impact on usability surveys and what matters is the aggregate results, and the conclusions taken. Most are only genuine under the condition of anonymity. Anything else is adding complexity and unnecessary scrutiny to what is already an established practice. User-selectable choices for publishing would open the possibility for privacy mistakes and increase the work of @ninavizz to have to publish those.
With Qubes OS you have the guaranteed scrutiny of the core developers. So it's guaranteed none of these changes will be degrade security. Quite the contrary. If your concern is that because the discussion / feedback is not public, it creates possibility for abuse in justification of features, just be aware that if all feedback were to be public, then you'd miss half it (but I don't think this was your suggestion).
To conclude, I don't get your point. If you want to make your feedback public, just post it here. I don't think anyone has questions over their ability to post things publicly here. So what are you trying to address?
jpouellet
commented
3 years ago deeplow wrote:
Nothing is preventing people like you from giving feedback publicly.
Sure, but I certainly wouldn't expect any statistically meaningful sample to do so, given that it isn't suggested or recommended or in any meaningful way visible. Such is the nature of the survey.
Privacy has a big impact on usability surveys and what matters is the aggregate results, and the conclusions taken. Most are only genuine under the condition of anonymity.
I'm certainly not advocating for the removal of a way to anonymously provide feedback, or even for that to be the default.
I certainly would not want to exclude users with the most sensitive needs from being understood. They are perhaps the most important to better understand in order to appropriately serve! I'm only noting that it would be useful to the broader community of Qubes contributors and secure operating system development community in general if the UX preferences of users were better understood in general, that this is probably not the last survey which will take place, and that some users are surely comfortable sharing more or less than others. If effort is being expended to attempt to better understand workflows and preferences now, it seems IMO a shame to isolate all the resulting knowledge and insight to a select few forever, and worse yet to not isolate especially the potentially more-revealing free-form responses without the informed consent of those submitting.
All I'm advocating for is that it be clear up-front to survey responders what will happen with responses (for example, will these results be published in aggregate as the first survey's was? If so, I don't see that stated anywhere), and suggesting that allowing users to opt-in to sharing beyond the survey operator(s) has utility to other contributors, and may be worth considering in the design of future surveys, depending on their nature and generality. (Better understanding general workflows, for example, being more worth sharing more broadly (if acceptable) than "do you prefer A or B between these specific design aspects".)
Anything else is adding complexity and unnecessary scrutiny to what is already an established practice.
The previous Qubes survey provided clear expectations to survey-takers about what would end up being shared and how, and the results were insightful! (Especially regarding the relative popularity of distro choice for templates, for instance, which has been a bikeshed debate for as long as I can remember and can finally be put to rest, at least for now.)
So what are you trying to address?
The observation that in the past, some UX decisions have been made which in hindsight have turned out to be terrible decisions (including by me! not trying to throw blame here) and eventually reversed course, but could have been avoided entirely if the contributors had known more about how real users use things / what real users actually want / what workflows to test.
I'm referring to times like when qubes-manager was deprecated entirely, in favor of tray items as (rather incomplete) replacements, and this was deemed a Good Thing, until it was realized that very many users really did want something like qubes-manager, and it was revived.
Or that time when I chose a most regrettable keyboard shortcut for pausing VMs 1 which, to users who didn't know otherwise, appeared to cause their whole machine to simply freeze up. This pathological behavior went undiagnosed for an embarrassingly long time, in large part because I was aggressively using DispVMs to compartmentalize my browsing whereas other people were using Firefox's new-window-in-private-mode shortcut, and the primary insight I had into how other people use qubes was from the limited set of IRL friends I had convinced to use Qubes at the time (whose own workflows were to varying degrees biased by demonstrations of how I used my own system, in an effort to convince them that Qubes is awesome and worth a try -- so, definitely not representative of the whole diverse user base).
h01ger
commented
3 years ago hi,
as a data point: I wanted to participate in the survey, even though I'm a "weird" i3 user which almost exclusivly starts apps by pressing shift-enter and then typing the vm name to launch a terminal there (and another combination to start browsers), so I opened the survey, saw that I need to watch videos to particite and closed the survey.
I'm the type of user who given the choice between reading 4-8kb text or watching 1m video will prefer to read the text. or skim the text.
I wouldn't have minded if the survey had some questions which require one to watch a video, but a whole survey which doesnt work without watching some video?
it also sends a very unfortunate message to blind qubes users...
-- cheers, Holger
⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄
unman
commented
3 years ago On Mon, May 03, 2021 at 01:45:18AM -0700, Holger Levsen wrote:
hi,
as a data point: I wanted to participate in the survey, even though I'm a "weird" i3 user which almost exclusivly starts apps by pressing shift-enter and then typing the vm name to launch a terminal there (and another combination to start browsers), so I opened the survey, saw that I need to watch videos to particite and closed the survey.
I'm the type of user who given the choice between reading 4-8kb text or watching 1m video will prefer to read the text. or skim the text.
I wouldn't have minded if the survey had some questions which require one to watch a video, but a whole survey which doesnt work without watching some video?
it also sends a very unfortunate message to blind qubes users...
Thank you Holger. I second this, although, of course, visually impaired users will be unlikely to have opinions either way, unless one or other menus has been designed with them in mind.
I have received many reports from Tor users that they are unable to participate. Again, I think this sends the wrong message.
ninavizz
commented
3 years ago (...also: I am now thinking to do an FAQ page to publish somewhere, to publicly answer some of the above. I will work on composing something tonight tomorrow (plans tonite!), and sending it to Andrew to post on the website, unless there is strong pushback in comments, below. No, unfortunately I don't have the time to do an extensively iterative piece with lots of community feedback, as I also need to do billable work. Yes, I also hate Capitalism and do wish a trust fund could fall from the sky, fwiw.)
ninavizz
commented
3 years ago @unman What are desirable video formats, for security? I just uploaded .mov files to my GDrive for @andrewdavidwong to put on the Qubes website server, but then realized that format might not work in an HTML5 embed.
Or, perhaps more broadly—what is the best way to get Tor users videos?
Yes, for user research best-practices reasons, I need folks to answer a bunch of questions to the best of their ability, no right or wrong answers after watching the two complete videos. I could also spell that out a little better, in the page text on the survey, too.
ninavizz
commented
3 years ago @h01ger @unman The "Meet Bessie & Jackie" page on the live survey has been updated, to speak to both our Tor and our visually impaired users. Thank you both, again, for those flags. I also, honestly assumed that Tor users could "just go use clearnet if we say it's gonna be safe, yeesh;" which in hindsight was naive and presumptuous. I also hadn't realized the first question in the first section, required JavaScript to input an answer against. That's pretty dumb. I'll update that to a radio button option, when I publish the updates with a WebM video alternative.
FYI, I'll be sure to give the community a full week to respond to surveys with feedback, before I broadly publish them to the community, in the future. This survey needed to be done rather quickly because of a funder deadline, but I appreciate the wealth of feedback from everyone in this post, a lot.
DemiMarie
commented
3 years ago @unman What are desirable video formats, for security? I just uploaded
.movfiles to my GDrive for @andrewdavidwong to put on the Qubes website server, but then realized that format might not work in an HTML5 embed.
WebM is probably the best option, as it is royalty-free and so is supported by browsers that don’t include H264 decoders.
Or, perhaps more broadly—what is the best way to get Tor users videos?
The most important part is probably testing the site in Tor Browser. I can think of three major problems that Tor users are likely to face:
<video> tag (that works without JavaScript) is the safest option.Edit: Tor Browser in “Safer” mode disables the SpiderMonkey JIT compiler, as it is a large amount of attack surface. This means that JavaScript performance will be quite poor. The good news is that sites that work well on Tor Browser are likely to also work well on mobile, provided that they can handle the smaller screens.
ninavizz
commented
3 years ago @DemiMarie The speed thing is a problem... as I can't make the videos "small," as users need to recognize legible text elements in them. Would MPEG4s be ok? I've never heard of WebM.
@jpouellet Actually, I replicated the privacy disclosure from the last survey. I am a UX practitioner—a designer and a researcher—this is all I do. Prior UX decisions were made by developers. I trust developers to do their trade well, and need the community to trust me to do my trade well. I don't think that is asking too much. I am on the team page on the Qubes website, and my own portfolio of work is viewable on my website—which you can get to from my bio, here.
The last survey was written-up about by Marta, whom I think did a fantastic job presenting meaningful findings in aggregate. It is also a volunteer to-do, to create a User Research page on the website, that will more broadly speak to these things eventually... but that's a volunteer to-do. Yes, I would LOVE to eventually have lots of info on Qubes users for developer contributors to look at, made available... but time/money. Research is regrettably not cheap, and we all want to see Qubes get more funding, to fund things like a robust QA lab... litterboxes that clean themselves for all of our cats, and full-time UX resources. :)
ninavizz
commented
3 years ago @DemiMarie n/m the above, I found a good conversion tool and am converting them right now! :)
h01ger
commented
3 years ago it still starts with a video...that's where I stop participating, sorry.
https://survey.qubes-os.org/index.php?r=survey/index&sid=434783&lang=en
ninavizz
commented
3 years ago @h01ger
andrewdavidwong
commented
3 years ago as I need Andrew to push the videos to the qubes server for me.
I'm not sure if GitHub will allow us to commit such large files, but I'm willing to try (once given video files with which to try).
FWIW, I just tested, and Tor Browser on the "Standard" security level allows me to watch the Vimeo survey videos, as well as arbitrary YouTube videos (both without CAPTCHAs, though that might just be by luck of the exit node).
ninavizz
commented
3 years ago omg! the video sizes are like... massively slashed when converted to WebM whee! They're only 17mb and 20mb apiece, now, @andrewdavidwong. Uploading them to my GDrive on my slow-poke satellite internet, will take another few minutes.
andrewdavidwong
commented
3 years ago omg! the video sizes are like... massively slashed when converted to
WebMwhee! They're only 17mb and 20mb apiece, now, @andrewdavidwong.
Should be fine, then. The GitHub commit file size limit seems to be 100 MB.
Uploading them to my GDrive on my slow-poke satellite internet, will take another few minutes.
No problem. Let me know.
andrewdavidwong
commented
3 years ago To be clear, the place I'd be storing these files is here: https://github.com/QubesOS/qubes-attachment/tree/master/survey
This results in public URLs under the qubes-os.org domain, which is presumably what you want. However, this is not the same server where LimeSurvey is actually hosted. IIRC, the latter is a server under @marmarek's control. I'm not sure if that matters to you or @marmarek; just thought I'd mention it.
ninavizz
commented
3 years ago @andrewdavidwong They're both up on my GDrive, now. I only care about users taking the survey trusting where the videos "live." My very un-educated guess is that anything mapping to a GitHub server could be bad, and that properly storing them on the same server as where the survey itself, lives, would be best?
Making things extra special... Lime does allow me to "upload" things via its admin UI, but then it doesn't give me a URL path (and none of the paths I've guessed, have worked).
@marmarek Might you mind uploading the 17mb and 20mb file from my GDrive, to the Lime server—and then send me the links? Because you only have about fourteen-thousand other things to do, I know. (or @marmarta? or the cats?)
https://drive.google.com/drive/folders/1t2bILfqz_GkNEUYrESEpxDG6HY2wQ1e2?usp=sharing
andrewdavidwong
commented
3 years ago I only care about users taking the survey trusting where the videos "live." My very un-educated guess is that anything mapping to a GitHub server could be bad, and that properly storing them on the same server as where the survey itself, lives, would be best?
FWIW, the whole qubes-os.org website is hosted this way (i.e., from GitHub servers), which is why we have this FAQ entry: "Should I trust this website?"
@marmarek Might you mind uploading the 17mb and 20mb file from my GDrive, to the Lime server—and then send me the links? Because you only have about fourteen-thousand other things to do, I know. (or @marmarta? or the cats?)
Here are the two images used in the survey, if you also want to move those to the LimeSurvey server:
https://raw.githubusercontent.com/QubesOS/qubes-attachment/master/survey/bessie.png https://raw.githubusercontent.com/QubesOS/qubes-attachment/master/survey/jackie.png
jpouellet
commented
3 years ago @ninavizz: I think we may have gotten off on the wrong foot due to missing context.
At the time of posting, I was not aware that this is funded work (implying a fixed timeline and defined scope that are totally sensible to want to stick to). The word "fund" only appeared once between the two issues (here and #5677) up to the point of my comments, in a comment of yours back in March. It is clear why, as someone working on said funded project, you would be naturally aware of its existence and constraints, but please excuse me for not having the same context. This discussion appeared otherwise largely indistinguishable from the vast majority of other qubes issues which are purely technical and open-ended in nature, void of resource constraints besides "someone"'s eventual free time (and where I have ended up being that future "someone" quite a number of times with a great delay between thought-recording and implementation, and found my prior notes and thoughts (or, and especially, others' notes and thoughts) helpful, regardless of how raw a form they may be).
By all means, don't let the perfect be the enemy of the good (and by this I do not presume that my suggestions are perfect nor even good ;). You're the UX person here, and I defer appropriately.
ninavizz
commented
3 years ago @jpouellet It's actually helpful for me to hear that "Funded" and "Funded within constraints" are helpful information to hear for delivering feedback in GH issues—as, yes, much else here is done on pure volunteer terms. My apologies for not having more made that clearer as a boundary around timing and scope, earlier on. Do know, however, that raising the bar on community involvement with UX stuff is a keen interest of mine... and aspirational encouragement towards that end, I always enjoy and appreciate. 🙏🏻
@andrewdavidwong I did not know that's how the Qubes website's content worked—I fully retract my earlier point of finger-wagging at hosting the videos on GitHub. By all means, if you don't mind pushing them, I would love those links!
I'm almost done with a newer version of the survey, with skip-logic to accommodate accessibility needs and videos-vs-pictures. Looking forward to getting newer video links to include in those!
andrewdavidwong
commented
3 years ago By all means, if you don't mind pushing them, I would love those links!
I just requested access to your GDrive files, @ninavizz.
andrewdavidwong
commented
3 years ago Wait, there's two tour-jackie.webm files. Shouldn't one of them be Bessie?
ninavizz
commented
3 years ago @andrewdavidwong Sorry 'bout that! Access granted.
andrewdavidwong
commented
3 years ago Wait, there's two
tour-jackie.webmfiles. Shouldn't one of them be Bessie?
I'm able to see which is which from playing them. Will correct filename before committing.
ninavizz
commented
3 years ago @andrewdavidwong Apologies for that, you are awesome—thank you!!
andrewdavidwong
commented
3 years ago No problem. They're now available at these URLs, @ninavizz:
https://www.qubes-os.org/attachment/survey/tour-bessie.webm https://www.qubes-os.org/attachment/survey/tour-jackie.webm
ninavizz
commented
3 years ago Ok. I've made a copy of the original survey, and made a number of changes—namely:
Please give this a spin and let me know what you all think? @marmarta @deeplow @h01ger @andrewdavidwong https://survey.qubes-os.org/index.php?r=survey/index&sid=129726&lang=en
— EDIT — I can stop the survey and add the updated functionality to it, whenever; but it would need to be offline for an hour, and its data will be saved separately when it re-runs.
So far we have ~300 complete responses, with ~700 total attempted. The extremely high number of attempted-but-abandoned responses, is troublesome—and I suspect because of the aforementioned video issues.
I am VERY pleased by the WebM format's performance—the videos look far better on their own, without the distracting UI elements of Vimeo surrounding them. Thank you @DemiMarie for that marvelous suggestion!
I have updated all of the video links on the other survey. If you all are ok with the updated version of the survey (above) then I can propose some Twitter and Forum/Email text to blast to alert the community that video issues have been addressed, as well as a section added for assistive-device users.
ninavizz
commented
3 years ago ...having not heard back from anybody, and the first question's Javascript requirement being a blocker for Tor "Safest" settings users, I went ahead and stopped the survey, archived its responses to date, and implemented the changes in the "Deux" version, above. Pls to ping with further suggestions or updates. I also added additional answers per @jpouellet flagging the accidental of former users from the app menu and "how long?" questions.
Thx again for all the feedback, folks! Will email @andrewdavidwong with Tweet follow-up texts to speak to users unable to watch videos, Tor/Whonix users, and users dependent upon assistive devices.
ninavizz
commented
3 years ago FYI, we've thus far heard from two folks that use assistive devices, about their non-visual navigation needs. TY again @h01ger for the nudge on my initial oversight in offering those folks a path for solid feedback, last week.
ninavizz
commented
3 years ago Lots was commented on in the survey, about things in Qubes unrelated to the appmenu. Those findings I've compiled in a hack of GitHub's "Project" feature, using KANBAN columns as categories: https://github.com/QubesOS/qubes-issues/projects/9
andrewdavidwong
commented
3 years ago Lots was commented on in the survey, about things in Qubes unrelated to the appmenu. Those findings I've compiled in a hack of GitHub's "Project" feature, using KANBAN columns as categories: https://github.com/QubesOS/qubes-issues/projects/9
I really don't think that this is a good way to try to store and share this information. "Projects" are meant to be "meta" issues or "epics" -- things that contain multiple issues (in the technical sense of the objects tracked in an issue tracking system). So, this sort of survey data is not what "projects" are for. Abusing the feature in this way is likely to lead to unintended side effects, not to mention that it clutters up the projects list, getting in the way when trying to find and manage actual projects. It's also not clear what we're supposed to do with this "project." Is it just going to sit around in the project list forever? I think this sort of data should just go into a public spreadsheet that can be linked in the appropriate places. Or an actual public Kanban board on some other website, which could also be linked in the appropriate places.
ninavizz
commented
3 years ago @andrewdavidwong All of the Tor reasons that were brought up in the AppMenu survey, had me wanting to try a tool available in GitHub. I'm currently creating a "board" in a visual tool I really like, Miro, and am crossing my fingers that one works nicely. Thx for the feedback—never wanting to throw the PM game off!
ninavizz
commented
3 years ago ninavizz
commented
3 years ago This issue will be closed once I publish the findings on the Qubes website (which I will work with Andrew and Marta to achieve).
andrewdavidwong
commented
11 months ago @marmarta, can this be closed?
marmarta
commented
11 months ago yes, you're right, this is generally done / turned into actionable tickets
Hello!
As part of #5677 we are conducting user research to learn more about user needs and community desires, for an all-new QubesOS-specific app menu, that @marmarta will begin development work on very soon.
https://survey.qubes-os.org/index.php?r=survey/index&sid=434783&lang=en
I began this project with the first initial survey of QubesOS users, to more broadly learn about user needs... and now that I've generated these two different design directions (shown in videos in the survey), I'd love folks' feedback. No thoughts are "right" or "wrong," and anticipating what might be best for other Qubes users—while helpful on GitHub—is discouraged. What do you think, and how might these directions suit your needs?
The survey is the best place for the feedback I need... as I cannot do meaningful researcher analysis of GitHub comments. Much as I otherwise enjoy them. :)
We will also be cross-posting this to the forums, and the community wranglers @andrewdavidwong and @mfc to the email lists and to Twitter.