josephdsumner
commented
2 years ago This is a non-blocking issue for releasing the survey, but it does need to be resolved when we publish the results. In the mean time, we do need to add 'We will be publishing the full methodology alongside the results' ASAP. I will add this in a separate issue now.
MelSumner
In the interest of transparency, the publication of survey results should have a concomitant addition to the methodology page with the details of the statistical analysis performed against the responses. This should include a "Statistical Analysis Procedures" section from which an individual could reproduce our results with the raw data, at minimum.
We aim to provide context for published results -- primarily, to ensure transparency re:
methods used for analysis
question format: single-select, multiple-select, etc.
how 'blank' responses are handled (e.g. '50% of respondents use ember-test-helpers' => '90% of repondents who elected to answer this question use ember-test-helpers')
statistical analysis
justification / rationale for all instances where
raw !== reported:exclusion of data points (e.g. clear intent to submit a destructive response for the sake of doing so, clear indication that the prompt was misunderstood)
inclusion of valid data points collected by other means (paper form, in-person events 'survey party!')
aggregation of datasets (e.g. 'vim' = 'vi' + 'nvim' + 'vim' + ... )
normalizations or adjustments for the purpose of estimating historical data trends
grouping Ember versions in a range near their LTS or back-calculating from 'Latest'
reporting only the 'Top 5' for the 'IDEs used', because formerly popular tools usually get deprecated from a the list of named options if they haven't had responses for two years (e.g. 'Notepad' to 'Other')
Essentially, we need to publish the fine print section for any * that appear in the published results.