Run a user survey

[adrn]

At the NumFOCUS summit, we discussed the merits of running a short user survey to gather general information about user experience. For example, both dask and pandas have recently run such surveys.

We have run surveys (either via google form or by mailing list post) in the past, but recently these have been for very specific goals (assessing performance, python 2 to 3, etc.). Others have also run more general surveys of the astronomical community about, e.g., software use.

This issue is meant to start a discussion about whether we think it would be useful to run a similar survey. However, I think we should mainly discuss this face-to-face at the coordination meeting, and this issue should just serve as a place to remind us and collect broader feedback.

This is all open for discussion, but I was imagining a short (~5 min), information-gathering survey to assess how users use and perceive the astropy core package. For example, what operating systems they use, how they learn to use astropy, etc. To clarify, I'm not suggesting that we solicit general feedback about the package, but rather to gather more general statistics.

Goals of a survey

I've included here a few goals for running such a survey, but please suggest modifications or additions.

• Gather statistics about the systems, computers, and environments in which people are using astropy
• Gather some very basic demographic statistics, e.g., at least country of residence/work, position (academic, industry, etc.),
• Gather statistics about what subpackages our users use the most, and what other packages are being used with astropy most frequently
• Gather information about points of entry to astropy
• Gather information about what educational materials people use, attendance to conferences (e.g., PyAstro, AstroHackWeek, AAS hack day, etc.)
• Gather information about perceptions of the developer community
• Understand whether our users know how to (if they want/need to): report a bug, get help/support, contribute code

Open threads

• We should discuss or coordinate with NumFOCUS on how to review the questions. For example, is there a social scientist we can pay to consult on this?
• If we want to publish results (in a journal, e.g., BAAS), do we need IRB approval?
• Do we want to borrow questions from the Python developers survey to compare / benchmark against the broader Python community? Is that useful?

Discuss!

[kelle]

• I strongly think we need to have a clear idea about what we want this survey to do and what we would change about what we're doing based on the results.
• I am more in favor of designing a survey which we could use to track longer term trends.

[adrn]

Agreed! I was imagining this being a yearly survey. I will write down some things I was imagining that the survey responses could help use take action on.

[pllim]

As for motivation, maybe people who fill out the survey can be entered for a lucky draw to win something. :gift:

[adrn]

Returning to this, I'm still in favor of this idea! I think one sufficient motivation is just to gain a better understanding of the community of users that interface with the Astropy project -- I don't think we need to have a more specific goal. For example, Numpy last year ran a community survey with this goal: https://numpy.org/user-survey-2020/

[hamogu]

After reading over the numpy survey, I suggest to ask for specific feedback at the same time. I believe (but I might be wrong - the survey will show), that a sizable fraction of astropy users never provides feedback to us in the channels that we usually monitor (github issues, mailing list); are only rarely in more informal channels (facebook, twitter). Any opportunity to gather free-from feedback should be taken. We can take on the task of converting any specific feedback into github issues if warranted (based on experience with previous surveys, only a very low fraction of people will give such detailed feedback, but it's always worth trying).

[hamogu]

Just to note that down somewhere: Specific questions I suggest:

• installation and pacakging: Do you use pip, conda, debian pacakges, ...?

[adrn]

Yes good idea! A quick update for all: I've been in touch with the Numpy leads who ran the survey and have some more information, so I will come up with a plan soon.

[adrn]

Just to give another update here: I am working on a draft survey, basically taking the Numpy survey questions and removing any numpy-specific text (replacing some with Astropy, but some don't make sense for astropy). I'll share this soon with the dev telecon (and astropy-dev mailing list) and open it up for discussion about adding new astropy-specific questions.

[astrofrog]

Before I forget, we should include a question as to whether users rely on the existence of the LTS releases.

[adrn]

Note: the draft was never completed, but it is here: https://docs.google.com/document/d/1s1wbOmFjMQqaQC6yYCBBP1LNho0roZxU-GUtxqweq_A/edit?usp=sharing The questions are based on the Numpy community survey. I was working to update the questions to be astropy-specific, and I was flagging questions that I thought could be removed to shorten the survey.

I still think this is an important thing to do, but at the time there wasn't much internal support for this so I lost momentum. If anyone wants to pick this up, I'd be happy to contribute or help work on this again.

[pllim]

Good talk about how to run survey from Scipy 2023: https://cfp.scipy.org/2023/talk/H9FDBV/