Kubeflow User Research & Survey Strategy

[StefanoFioravanzo]

So far, the Kubeflow community has published a yearly user survey composed of high level questions to gauge how users interact with Kubeflow, what are the most used components, what are the biggest gaps, and how Kubeflow is deployed in production. The survey provides a good snapshot of users demographics and, with the help of freeform questions, it provides a pretty detailed report of specific areas users would like to see improved.

This information has proven very useful to understand what are the most used components; missing capabilities; usability concerns; distributions adoption; etc.

There are two aspects to consider now that the community is maturing and building ever more complex capabilities:

• Interviews: the current process still lacks a second critical phase in user research: the interview. User interviews help create a detailed picture of the user personas, understand their needs, validate the survey findings, and drill down into specific usability concerns.
• Component-specific research: Working Groups need detailed feedback on existing/new components and features. A (yearly) high-level generic user survey doesn’t help working groups getting specific feedback on what they are building and what users would like to see in the future.

Kubeflow is a very complicated platform, the AI/ML landscape is changing rapidly. We need to adopt a product-led mindset and develop a platform strategy to holistically look at Kubeflow as whole, understand what it is and where it is going, based on consumer’s feedback and understanding. We cannot base our decisions on assumptions, we need a user research approach to validate our intuition and maximize our community effort towards building something that pays off.

I don't want to start in this issue a never ending debate on how we should adopt product thinking (that discussion can happen in a separate issue) but rather have a tactical conversation on how we can improve our current user research initiatves to drive better and more informative outcomes.

In this document https://docs.google.com/document/d/1PNBe_OUIIbw2avOUK-aBwL8HmRIRbYRv_6FMx4bvvpQ/edit?usp=sharing I list some bullet points on surveys and interview and make some proposal on what we can do in the next months.

cc @kubeflow/wg-notebooks-leads @kubeflow/wg-manifests-leads @kubeflow/wg-training-leads @kubeflow/wg-pipeline-leads @kubeflow/wg-automl-leads @kubeflow/release-team @jbottum @andreyvelich @johnugeorge @james-jwu @akgraner

[StefanoFioravanzo]

I also opened a CNCF Service Desk ticket https://cncfservicedesk.atlassian.net/servicedesk/customer/portal/1/CNCFSD-2175 to better understand how other CNCF project are doing this. Apparently the TAG App Delivery is conducting a very interesting initiative:

• https://github.com/cncf/tag-app-delivery/issues/546
• https://github.com/cncf/tag-app-delivery/issues/457

That discussion goes way beyond the scope of this issue (purely focused on our mid-term survey+interview strategy) but is good food for thought on how to adopt a product-thinking mindset. There's a lot of material that can be helpful to us. They also also investigating tools and practices to run user interviews with a repeatable framework and in compliance to GDPR policies. So I'll definitely reach out, maybe they can help us

[jbottum]

@StefanoFioravanzo thanks for kicking this off. As next steps, we might create the 1st set of survey questions. As we discussed, we have reused the questions from previous years, but we don't have to this year. As the same time, we need to identify users and ask if they will participate in some interviews. I think those requests need to be very defined in terms of privacy, time commitment and benefits (for the user). To your point, some of the interview candidates will come from the survey. For the survey (which is probably step 1), I would suggest defining 10 questions. Note - some of those questions will be multiple choice and so we will need answers for each question. Would this survey be created and delivered via google forms (as we have done previously)?

[juliusvonkohout]

Amazing, we also have to get this into the hands of internal company employees this time. E.g. DHL, some hospitals I know of and other end user companies. Just announcing it on the internet is not enough. We also have to do so in companies intranets and private channels.

[StefanoFioravanzo]

@juliusvonkohout yeah that's a whole discussion we should have - what is the "hidden" user base, how do we reach out, how do we keep track of gatekeepers, etc. I am sure these would provide a level of insight into real world production scenarios we, as a community, have not heard yet.