Final! Fall 2015 Rollout

[JasonJWilliamsNY]

1. The links for the pre/post survey are
   a. Pre-Workshop: https://www.surveymonkey.com/r/swc_pre_workshop_v1
   b. Post-workshop: https://www.surveymonkey.com/r/swc_post_workshop_v1
2. We need to associate pre/post with particular workshops. Options are
   a. Dump a text list of the workshops into the surveys @maneesha
   b. Confirm with @pbanaszkiewicz that we can modify the survey URL to pre-populate with this info
   i. If B is true, we will need to modify the survey question
   ii. If B will take longer than a day, we can and should go with A.
3. A pull request to add these surveys to the template for future workshops is needed
   a. I need to know where to make the request (https://github.com/swcarpentry/workshop-template/blob/gh-pages/index.html?)
   b. I need @maneesha to simultaneously update the Eventbright template to include the following questions that will no longer be captured by the pre-survey (taken from https://docs.google.com/forms/d/1th3cMQRh2jo-v1HdL5vddSVOw7y5xHN7zyWbukbCkMg/edit# with modifications as noted)
   i. In three sentences or less, please describe your current field of work or your research question.
   ii. In three sentences or less, please describe your current field of work or your research question.
   iii. What OS will you use on the laptop you bring to the workshop?
   iv. New Question: How would you rate your programming/scripting skills (Multiple Choice: Novice, Beginner, Intermediate, Advanced)
   v. What best describes how often you currently program?
4. Modify question and responses to include ‘scripting’: “What best describes how often you currently program or write scripts?
   vi. What best describes the complexity of your programming? (Choose all that apply.)
   vii. In three sentences or fewer, describe a common scenario in your daily work in which stronger computing skills would improve your workflow and make you more productive.

P.S. Sorry, copied this from email styling is shabby

[jduckles]

3.b.i and 3.b.ii are the same, omit one?

[pbanaszkiewicz]

Hello everyone,

As for 2b: it's possible to use custom variables stored within URL as long as we have platinum plan. Currently I'm unable to verify if we have that plan (no-one shared the Survey Monkey credentials on our passpack account). Link to the documentation: http://help.surveymonkey.com/articles/en_US/kb/What-are-custom-variables-and-how-do-I-use-them

I would recommend:

1. look for the "custom variable" somewhere in the survey design in SurveyMonkey (Design Survey → Logic → Custom Variables)
2. add one (maybe called "slug" or "workshop", labelled "Workshop")
3. submit a test response
4. verify that it's collected correctly

As I said I can't do this myself because I don't have access.

As for 3a: if 2b is possible we'll generate the new variable's content with {{ site.github.project_title }} (see https://help.github.com/articles/repository-metadata-on-github-pages/#usage)

I'll submit this answer via email too.

[JasonJWilliamsNY]

@pbanaszkiewicz @maneesha I have added the custom variable "workshop_id" to all surveys. The links for the pre and post are now https://www.surveymonkey.com/r/swc_pre_workshop_v1?workshop_id=[workshop_id_value] and https://www.surveymonkey.com/r/swc_post_workshop_v1?workshop_id=[workshop_id_value] respectively. Where [workshop_id_value] is replaced with our ID variable, this information will appear in the final data collection spreadsheet.

If we can guarantee that in all cases we will automatically generate this variable, then we could eliminate question 2 and also not have to worry about updating the survey with the latest workshop info. If there is ANY chance that the URL will be used incorrectly by someone, we should do both - generate the URL and also still have the survey taker indicate what workshop they are attending.

[JasonJWilliamsNY]

@maneesha, sure you also noticed the erroneous duplication mentioned by @jduckles. This should be replaced with 'Will you be attending the workshop with a group of colleagues, friends, or classmates?'

[JasonJWilliamsNY]

@pbanaszkiewicz created pull request https://github.com/swcarpentry/workshop-template/pull/264

[JasonJWilliamsNY]

@pbanaszkiewicz the workshop website template has been updated https://github.com/swcarpentry/workshop-template/commit/5c314b5caa5889fc5e8802360642e0c96172c4fd @maneesha can you populate the 3 surveys with a list of upcoming workshops?

[pbanaszkiewicz]

@JasonJWilliamsNY

If we can guarantee that in all cases we will automatically generate this variable, then we could eliminate question 2 and also not have to worry about updating the survey with the latest workshop info. If there is ANY chance that the URL will be used incorrectly by someone, we should do both - generate the URL and also still have the survey taker indicate what workshop they are attending.

Possible scenarios:

A) user intentionally changes the URL. I can feel that this is just slightly less possible than someone intentionally selecting a wrong workshop in the question "which workshop are you attending?". I feel I'd vote for using workshop_id only…

B) workshop organizers don't use GitHub to host the event website. They won't have access to site.github.project_title. Maybe we can alias that variable somewhere in index.html?

[JasonJWilliamsNY]

@jduckles @maneesha any ideas on this. We have to decide soon. How difficult would it be to update the question on the survey once a month?

[jduckles]

I think adding it to the template is the preferred way to go. Lets push the link itself to the _config.yml. I really don't like the "select from a dropdown" option as it will be full of erroneous selections.

So the _config.yml lines are:

swc_pre_survey: "https://www.surveymonkey.com/r/swc_pre_workshop_v1?workshop_id="
swc_post_survey: : "https://www.surveymonkey.com/r/swc_post_workshop_v1?workshop_id="

Leaving the links in the template as:

<a href="{{ site.swc_pre_survey }}{{ site.github.project_title }}">
<a href="{{ site.swc_post_survey }}{{ site.github.project_title }}">

Also, we are going to have to undertake a concerted campaign to notify instructors that this is the way things work now.

Are we OK with a learner filling out the pre bootcamp survey days in advance based on seeing that link?

[JasonJWilliamsNY]

@jduckles I think its OK if people do take the survey before, although hopefully putting it in the schedule as something to do when they arrive I guess we will get few people who do it that way. I will remove the 'Which workshop' question, and we will see if things go smoothly. I've made the pull request, edited the surveys, and yes, we need to send a note to all instructors. @maneesha perhaps can do this?

[maneesha]

So to be clear, the workshop's survey links will be generated when the instructors create the workshop's github page - and there's nothing else @pbanaszkiewicz or I or instructors need to do... correct @jduckles & @JasonJWilliamsNY? In that case, as far as communicating with instructors - it's just matter of telling them that they don't need to wait for me to generate the google docs surveys, but the links will automatically be generated when they push their github page, right? I can communicate that to instructors.

As far as learners completing the survey in advance I think instructors prefer it that way because then they get to know their learners' backgrounds in advance of teaching the workshop. But it would be good to include a few minutes at the start of the workshop for anyone who didn't get to complete it.

[pbanaszkiewicz]

@maneesha @jduckles @JasonJWilliamsNY I just created a test answer with workshop_id of test-workshop. The only way to see that is by using Export All responses data. That's a disadvantage to me, because it's not possible to filter by that value in the SurveyMonkey GUI. What do we do about this fact?

[maneesha]

I didn't realize this would happen if we removed the question where respondents explicity note their workshop. This is a big problem - we need the ability to share results from individual workshops.

[JasonJWilliamsNY]

@maneesha this is a good point. I don't need to change anything on the URLs/workshop templates, but I'll add the question back and notify you so you can repopulate.

[jduckles]

@pbanaszkiewicz Can we get the individual workshop data via an API query?

[pbanaszkiewicz]

@jduckles get_respondent_list returns a list of respondents (we need their IDs), and get_responses returns their responses (based on survey ID and respondents' IDs).

While neither of them allows for filtering, and doesn't say anything about custom variables, I found out that:

• get_survey_details returns question_id for each custom variable
• get_survey_details shows details for each question & its answers

…so I think we can filter (post-process) survey responses by "what workshop did you attend" question and maybe even by the custom variable (they may return custom variables in the get_responses hidden under question_id).

[JasonJWilliamsNY]

@jduckles @pbanaszkiewicz , that's exactly how I imagined it. I bet that even fetching and putting together a few of these for AMY would allow us automated real-time tracking of the most important metrics. Inside Survey Monkey we can generate links ad hoc (such as for the mentoring follow ups) so that instructors can know how they were rated.

[pbanaszkiewicz]

@JasonJWilliamsNY @maneesha @jduckles

can each of you please write down here (https://github.com/swcarpentry/amy/issues/578) what features exactly you want from SurveyMonkey↔AMY integration?

Thanks!