search

dask / community

For general discussion and community planning. Discussion issues welcome.
20 stars 3 forks source link

Dask User Survey #148

Closed mrocklin closed 3 years ago

mrocklin commented 3 years ago

Historically we've timed the annual Dask user survey to be around the SciPy Conference. Do we want to shift that to be a bit earlier this year and instead orient it around the Dask User Summit?

cc @TomAugspurger

TomAugspurger commented 3 years ago

Makes sense! Maybe we target May 5th or 12th to have the form up, so that we can promote it during the conference.

On Thu, Apr 15, 2021 at 12:55 PM Matthew Rocklin @.***> wrote:

Historically we've timed the annual Dask user survey to be around the SciPy Conference. Do we want to shift that to be a bit earlier this year and instead orient it around the Dask User Summit?

cc @TomAugspurger https://github.com/TomAugspurger

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/community/issues/148, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKAOIXMQYD3MD63RTASDNTTI4SBVANCNFSM4277CLBA .

TomAugspurger commented 3 years ago

@mrocklin do you have access to the Google form we used for the previous rounds? I think my access was under my anaconda.com gmail account.

To anyone, let me know if you're interested in tweaking / adding / removing questions from the previous year's survey. We just need to keep in mind that

  1. Changing a question's wording makes it harder to compare to previous years
  2. The survey shouldn't grow too long.
mrocklin commented 3 years ago

Done

On Thu, May 6, 2021 at 8:03 AM Tom Augspurger @.***> wrote:

@mrocklin https://github.com/mrocklin do you have access to the Google form we used for the previous rounds? I think my access was under my anaconda.com gmail account.

To anyone, let me know if you're interested in tweaking / adding / removing questions from the previous year's survey. We just need to keep in mind that

  1. Changing a question's wording makes it harder to compare to previous years
  2. The survey shouldn't grow too long.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/community/issues/148#issuecomment-833504430, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTGXSNL45IFBGT2XJB3TMKHRZANCNFSM4277CLBA .

jacobtomlinson commented 3 years ago

During the Dask Community meeting we had an idea generation session for potential questions. Here's a list from the meeting notes. I encourage folks to add more ideas here if they have them.

TomAugspurger commented 3 years ago

I updated the survey to include some of the suggestions.

Please don't publicize the link yet, but you can view the 2021 survey at https://docs.google.com/forms/d/e/1FAIpQLSfio2RIQGIQsX1QTJh4JXmTFK8s-7BbsR0VnfmsWXu1Ccb2Yw/viewform?usp=sf_link. I'd appreciate if a few people could go through and let me know if they find any of the wording confusing and how long it took you to complete the survey.

A rough changelog from 2020, that's hopefully comprehensive

If I didn't add a question to cover a bullet from https://github.com/dask/community/issues/148#issuecomment-833629623 its because a.) I missed it, b.) I couldn't figure out how to word it in a question, or c.) I felt like it was too specific. If you feel strongly about any I left out, now's the time to speak up (preferably in the form of a question)!

Some questions:

  1. Under "What Dask resources have you used for support" I added the Dask slack. Do we want to add the Slacks / other venues from companies (e.g. Prefect) here?
  2. Are there any questions you'd like to see removed?
jakirkham commented 3 years ago

Thanks Tom and Jacob 🙂

It looks like these two may be duplicates (or perhaps the latter one is just emphasizing the former one is free-form)?

  • Added "How else can Dask improve" ...
  • Added a free-form "How can Dask improve" question.
GenevieveBuckley commented 3 years ago

@jakirkham - I clicked through the survey and there don't appear to be any duplicate questions. I think it just appears twice in Tom's list above.

How often do you use Dask?

a) Just looking for now b) Occasionally c) Every day d) I use Dask all the time, even when I sleep

It seems like there's a big gap between "occasionally" and "every day". In addition, "occasionally" is ambiguous & will mean different things to different people. Finer grained options for answers in this range would help.

d) is funny, but doesn't yield a lot of valuable information compared with c). People will click on it if they think it's funny, possibly regardless of their actual use.

GenevieveBuckley commented 3 years ago

Also, there are two questions about what people want more of, that include answer options like:

This feedback seems difficult to act on, unless you also know which field they're in.

Can we attach that info in the question answer? For the first question "Which would help you most right now?", we could probably modify the "Other: " answer option to something like "More examples in my field (write your field here): __". It might be harder to do for the second question "What common feature requests do you care about most?" though, not sure about that.

I know there's a free text question at the end for "What do you work on? How does Dask help?", but people might skip that answer entirely, or they might not write their answer in a way that makes it easy to figure out what field it is.

TomAugspurger commented 3 years ago

Thanks for the feedback!

It seems like there's a big gap between "occasionally" and "every day". In addition, "occasionally" is ambiguous & will mean different things to different people. Finer grained options for answers in this range would help.

I agreed on both all your points. However, this is a question that's been present since 2019, so I'm tempted to leave it in place so that we can compare results with earlier years. Do you have thoughts on what's more valuable: clarifying the question vs. preserving comparability to previous years? (I'm struck by the similarity to API design here. Changing a question's wording is like an API breaking change 😄)

This feedback seems difficult to act on, unless you also know which field they're in.

Yes, it'd be good to capture that somehow. I'll look to see if the Python or StackOverflow surveys have a question about "industry / field" so that we have a list of industries to choose from.

we could probably modify the "Other: " answer option to something like "More examples in my field (write your field here): __"

One challenge here: that question is currently a multiple-choice (select one) rather than a checkbox. So we can't easily have an "other" field where they fill in that information. But if we capture their industry in a separate question, we'll hopefully get what we need.

GenevieveBuckley commented 3 years ago

I agreed on both all your points. However, this is a question that's been present since 2019, so I'm tempted to leave it in place so that we can compare results with earlier years. Do you have thoughts on what's more valuable: clarifying the question vs. preserving comparability to previous years? (I'm struck by the similarity to API design here. Changing a question's wording is like an API breaking change smile)

Oh ok, your comment above made it sound as though this was a new question added in 2021.

Yeah, that does make things trickier. I guess it depends on what'll be more important in future years: being able to compare backwards to 2019 & 2020, or having a finer grained view.

TomAugspurger commented 3 years ago

To help with the issue Genevieve identified about "This feedback seems difficult to act on, unless you also know which field they're in", I added a question: "What industry do you work in?" Currently the options are

Let me know if you want anything added / removed / split apart in that list.

I think this is essentially ready to go. I recommend that we promote the survey at the start of the social sessions, and have all the workshop organizers promote it too. We could maybe slip it in at the start of the keynotes if there are other things to share then, but I don't want to take too much time.

martindurant commented 3 years ago

Would add marketing

TomAugspurger commented 3 years ago

@fjetter I added a question "Is Dask resilient enough for you? (e.g. computations complete)." that hopes to track the issue you cared about. Do you think that wording captures what you wanted? Any better way to phrase it? The "computations complete" wording isn't perfect, since things can fail for issues in the user code, but we're trying to capture issues in Dask's "runtime". I also added "Improve Resiliency" to the options in "What common feature requests do you care about most?"

GenevieveBuckley commented 3 years ago

Now that the Dask User Survey is being promoted to attendees at the Dask Summit, should we also have a post on the dask blog to promote it too?

I'm happy to organise a draft for that if you'd like - I'm about to spend a bit of time over on blog stuff anyway.

jrbourbeau commented 3 years ago

Have we tweeted from the Dask account about the survey yet? That's also a good way to promote the survey to users

GenevieveBuckley commented 3 years ago

Have we tweeted from the Dask account about the survey yet? That's also a good way to promote the survey to users

...I was just about to leave a comment with this too! I don't think so, I'll schedule something in tweetdeck to go out on Monday morning US time (uh, if I don't get too confused calculating the time difference, that is)

GenevieveBuckley commented 3 years ago

Tweet scheduled. It should go out around 9am Monday PDT. The text says:

The Dask User Survey is out again! Tell us how you use Dask, and help us make it better for everyone: https://forms.gle/F7QSGpSHwBWu8NCg8

GenevieveBuckley commented 3 years ago

Blogpost draft made, see here: https://github.com/dask/dask-blog/pull/97

TomAugspurger commented 3 years ago

Thanks for handling the promotion @GenevieveBuckley. If you want, last year we added it to a couple more places:

  1. The root of the dask docs: https://github.com/dask/dask/pull/6328/files
  2. dask.org homepage: https://github.com/dask/dask.github.io/pull/27/files
GenevieveBuckley commented 3 years ago
  1. The root of the dask docs - PR made here https://github.com/dask/dask/pull/7694
  2. dask.org homepage - PR made here https://github.com/dask/dask.github.io/pull/48

I don't have merge access to either of these repositories, but I'm sure someone who does will be along soon.

GenevieveBuckley commented 3 years ago

The 2021 Dask user survey has closed, and the survey results are available here: https://blog.dask.org/2021/09/15/user-survey