Improve the checkin process

[rgardner]

Problem

The current checkin process is unacceptable and is doomed to fail. After seeing the checkin process first hand tonight at our meeting, I'm worried the current checkin process is too cumbersome for primetime.

• Ethan and I have no idea what our FB passwords are.
• People may not trust us with their FB information (it certainly leaves me uneasy).
• How much time will it take for one person to sign in? 30 seconds? How does this scale to Demo Days?

  Proposal

• Our members enter their netid or their name and email address.
• If this is their first time checking in, we email them asking them to complete their profile
  • they can connect with Facebook or fill out a form manually. It is important to give them the option because some people don't use their real information on Facebook.
  • in this profile completion step, we explicitly say that we are using this information to improve our events (we are an equal opportunity club and want to create events for people of all genders, sexual orientations, races, ethnicities, etc.)

edit: sorry for the harsh words initially, I really appreciate the time you all have put into this and want to make sure that people actually use this for our events.

[ethanresnick]

I was likewise worried about the checkin process after seeing it first hand yesterday, and I've been thinking overnight about how to fix it.

I agree with the first two issues you pointed out (i.e. that some people don't know their facebook password and that others may be uncomfortable signing in this way). I also agree that entering a netid or name/email would be faster. All those considerations make netid/email ideal from a user experience standpoint, as we discussed.

However, switching to netid (or name/email) brings some security challenges. API user accounts created from netids or emails wouldn't have any authentication built in, unlike those made through facebook. So we'd either have to: make it possible for anyone to edit any user's profile information (though of course the ui wouldn't make it clear how to do this); come up with some system of issuing one time or personalized links for people to edit their information (which brings its own set of ux challenges if people want to revisit their profiles later); or, have these users create tech@nyu-specific passwords (which I'm loathe for us to have to handle).

I've also been considering the fact that the checkin process we saw at yesterday's meeting was really not representative. First of all, there was the whole business of merging twitter and facebook api accounts, which of course our attendees won't have to deal with. But, even within the checkin experience, two key things were broken:

1. People couldn't checkin on their own devices by simply going to checkin.techatnyu.org. That ability would have alleviated much of the bottleneck and the problems of people having autocomplete systems for their passwords. This feature already works in production; it just didn't work at our eboard meeting yesterday because that was an "internal" event, and such events don't show up on the checkin.techatnyu.org homepage (for obvious reasons).
2. The auto "signout of facebook and redirect back to the checkin screen" after checkin wasn't working yesterday. Had it been, that probably would've cut checkin time for each person in half. It works now.

Considering all of the above, here's what I'm thinking. A netId/email solution would be an ideal ux solution, but it would require some not-insubstantial design and coding work to solve all the authentication-related problems and reconfigure the checkin system. Meanwhile, the problems with the facebook solution are way less bad than they looked at yesterday's meeting. Also, we've built a lot of things in the past only to discover that we didn't really need to spend the time making them (e.g. twitter auth). So, I'm really strongly inclined to try out the current solution and see how well it works in practice before we spend a lot of time coding new stuff for netid. If the current system really doesn't work, we can always switch to netId.

Now, about the problems with the current system, I think there are two simple solutions. For people who don't know their password: simply tell them to checkin on their own device. For people uncomfortable signing in with facebook: simply explain to them why we're asking them to. That explanation, could go something like this: "In the future, you'll be able to set up a profile with tech@nyu, e.g. to only get notified about events you're interested in. And, by signing in with facebook now, you won't have to make a custom password later."

Thoughts.

cc @maxdumas @AbhiAgarwal

[thebyrd]

What about making techatnyu.org/checkin nothing but a big checkin button. If they click it, you store their geo data and store a unique Id in their session data. You also on the next page put a 'subscribe to newsletter' form and if they sign up, you know their email too ;)

[rgardner]

@ethanresnick: Thanks for your thorough response. I'm still not a fan of FB only checkin, but I agree we should see how the numbers play out before making any substantial changes (people can just sign in manually). The checkin.techatnyu.org on mobile should alleviate most of my concerns (primarily FB sign in on a public computer).

The 'editing their information' part could be done by an email that sends them a unique link to their profile; I've implemented it before for a Rails app in about a day, but I don't know if it's as easy for this project. The assumption is that people with email credentials can be trusted; if someone tries to edit their information with an email they don't have access to, then the person with email access can just ignore the email.

I agree we shouldn't under any circumstances make people create a password just for us. It's unnecessary and bad for ux.

@thebyrd: How does this map users to unique ids? If they sign up for the newsletter, then you can match their unique id and email; if they don't sign up, then all we have is geo and a unique id. This lets us count the number of people in a specific area / for a specific event, but doesn't gain us anything more than just a headcount.

[irisyuan]

Is there a way to leverage a pre-generated list of RSVPed FB users, so that for a larger event like Demo Days, they simply have to find their name (search/ dropdown/ alphabetize/ etc. to make that easier) -- let's say on an iPad and tap on their name to check in? Similar to the Square experience. And then that takes them to a Step 2) Complete Profile screen with a progress bar at the top. They could do this on their own device or at a table. If they didn't RSVP on Facebook, only then do they have to create a profile from scratch?

I'm not sure how this data would be stored or how to best implement this, but it seems like unnecessary overhead making people sign in and out when we aren't making use of the RSVP list. Also, this could force people to fill out the profile as part of the checkin process, because I think people expect to check into larger events like Microsoft, Tumblr, etc. where they don't necessarily know where to go. When they complete their profile, they could get a message like "Thanks! Enjoy the event!"

Maybe this has been discussed before - what are the limitations of this approach? Here's one - anyone could try to fill out anyone's profile. That seems unlikely to me, but how big of an issue is this in terms of gathering diversity insights from attendee data (which from what I understand is the main motivation for the checkin)? Maybe part of the profile is email and then there's email confirmation?

[thebyrd]

It just uses sessions to track users. Lots of ways to get their emails later like putting their email as a query string in email links.

[ethanresnick]

@thebyrd I think the concern is that using sessions wouldn't give us an accurate enough way to track repeat attendees (as the same person will often create multiple sessions). Also, we might end up asking the same person for their email address over and over, if they gave it to us when checking in with one session, but not yet with their current session. Finally, using sessions would make it impossible to do sign in on a shared computer (e.g. one tech@nyu sets up at the front of the room/near the door), which we've treated so far as a valid use case, under the assumption that a shared signin computer would have a higher conversion rate than just asking people to sign in on their personal device (worth discussing).

@rgardner Glad you're up for giving the current system a try first. Re emailing one-time links to handle authentication: I agree that it works from a security perspective (i.e. if a person gets to the link, they can be trusted). The potential problem with it is that the user has to request a new link every time he/she wants to do something that requires authentication. Right now, the only thing that requires authentication is editing your profile, and that's something people probably won't do more than every 6 months, so maybe having to request a link each time is ok. But, this boxes us into a corner if down the road we want to add features that people would use more often, but that still depend on the user being authenticated. And a hybrid approach (i.e. one in which some accounts have been connected to facebook but others are still getting auth links by email) likely becomes a mess from a design point of view. In addition to making all the signin forms more complicated, it means that the user has to remember whether they've connected their account to facebook or not.

@irisyuan I think that's a great idea! And there's a way to implement it that doesn't involve anyone being able to fill out anyone's profile (which, I agree, wouldn't be a big deal in practice, but still doesn't seem smart). Basically, your approach means that we can get people's facebookId (because they're choosing their name from the set of facebook attendees), which is what we need to create a unique, facebook-tied account for them, without them having to sign in (which is the cumbersome part). Then, we can send them an email that asks them to fill out their profile on their own device, so check-in doesn't get bogged down. So they'll only have to sign in to facebook when they actually want to edit their profile (or, if they didn't RSVP, when they want to checkin). Here's how it could work as a flowchart:

[thebyrd]

@ethanresnick they could just as easily use multiple emails each time they sign-in, but this way they wouldn't have to type anything. Your data will certainly be imperfect... but since you're having trouble getting people to use your system seamlessly, start with the most seamless thing and then add fields/friction as you get engagement.

[thebyrd]

My hunch is that people won't even use the big red button b/c checking in sucks. You have to give them some value on top (like a name tag or swag).

[freialobo]

We tried the current system tonight at designdays and it was alright. Not amazing not awful. Nobody objected to using it but I don't think everybody did it, because we didn't make it compulsory. Like @rgardner said, there's going to be a scale issue at Demodays/Startup Week. One solution is to have 3-4 machines on our end to have more people sign up at a time, but that may not always be possible.

Agree 100% with @thebyrd, as of now there is absolutely no incentive for them to check in. "Check in to collect this free swag" will get old after the first couple events when we run out of unique swag.

[rgardner]

We should start thinking of incentives.

• Free Swag
• Attendee Leaderboard (like Foursquare King / Queen)
• Formal membership benefits
• raffle / give away

--  Bob

On February 23, 2015 at 9:56:54 PM, Freia Lobo (notifications@github.com) wrote:

We tried the current system tonight at designdays and it was alright. Not amazing not awful. Nobody objected to using it but I don't think everybody did it, because we didn't make it compulsory. Like @rgardner said, there's going to be a scale issue at Demodays/Startup Week. One solution is to have 3-4 machines on our end to have more people sign up at a time, but that may not always be possible.

Agree 100% with @thebyrd, as of now there is absolutely no incentive for them to check in. "Check in to collect this free swag" will get old after the first couple events when we run out of unique swag.

— Reply to this email directly or view it on GitHub.

[freialobo]

@rgardner

I like free swag and give away, will be pretty easy to implement.

Attendee leaderboard and formal benefits will be harder to flesh out but could work.

[AbhiAgarwal]

@rgardner I enjoy the idea of the formal membership benefits. What is our average member trying to get out of our club? They are interested in either:

• Learning a new skill
• Meeting new people
• Putting something on their resume
• Getting to talk to important people
• Getting an internship or a future job

If work on something that will benefit them from checking in to events and getting these reward then we could probably give them an incentive. Lets not try do something like free swag for each event.

Things like: if we're sponsored by Squarespace or whatever the sponsor is then maybe we can send them your resume or your profile for you? I'm not too sure, but if we try and see what people are looking for the most then we could probably give them some incentive. Love the idea for free give aways and swags as well though.

[rgardner]

@abhiagar I like this a lot, especially offering to send their resumes to the specific company (not necessarily applicable for After Hours, but sometimes Hack Days and always Demo Days).

re swags: this quickly becomes a mess. Am I responsible for the swag for After Hours? Who pays for it? How much? These are questions I don't want to think about...

We could do member only events / learning sessions, but that is a conversation worthy of a proposal. We could gamify it; 5 events gets you X, 10 events gets you Y, full membership at 15, etc.

[AbhiAgarwal]

Earlier we were trying to setup a system to match people up at After Hours events. For After Hours we are set. People will checkin because it will give them some value to checkin here. They will be able to find people with similar skills, and people to work on projects with them.

Freshman Circuit is next in line. People will checkin (If the lead wants this system for them) because it allows him/her to see who came to the events. This is not required as this is a fixed group, but going forward this can be implemented for them if the lead wants this implemented.

For other events it becomes a little less obvious. They are not social events, but Demo Days is probably the most social out of every other event (Excluding Freshman Circuit).

The resume drop will work for sponsored events, but not for everything else. These are the cases where we need to offer an external benefit in my opinion.

@rgardner - I love the idea of the learning sessions. That's a really good benefit to give people as it's a direct reward and something they would want.

We can offer perks as well. If you come to most events then we can give you early access to the CEO talks, or some of the more popular events - such as the keynote. But I'm doubting how successful this one would be.

[freialobo]

I don't really need the system for FC, but it could add value. Mostly because I have a very clear picture of who is going to what through strictly enforced RSVPs. I would be open to trying it anyway!

[AbhiAgarwal]

@freialobo. I agree - not required

[freialobo]

@AbhiAgarwal but you know they will be more than happy to beta test if required :)

[ethanresnick]

@thebyrd Agree 100% about needing incentives to checkin, and about needing to balance low friction with collecting more/more-reliable info.

@freialobo Re the system's performance tonight: were there any major usability stumbling blocks for people (i.e. things they didn't understand/couldn't get past), or was the problem just that the system was slower/more cumbersome than it needed to be and/or that people didn't see the incentive to check in?

For reference, we got 16 check ins at tonight's Design Days. I'm guessing that's about a 50% conversion rate, which is not great. Still, as a floor, 50% is a decent starting point. Even if we kept the system exactly as is, we'd probably see that number rise by a decent amount at subsequent events, as people get used to the process and as more people are returning attendees (who get a simpler checkin flow). With the usability improvements @irisyuan proposed, which should really lower friction, we could probably get the rate up even higher. This would all be while sticking with facebook, which gives the highest quality data.

Still, I doubt all of the above will get checkin up to 90%+, which is where it should be, so that means incentives. My thoughts on the ones proposed:

• Name tags: I love this idea. Name tags are something that would help people get more out of our events anyway. The challenge is to tie them into the checkin process in a plausible way; if all the name tags are pre-printed anyway, people would just want to take the name tag, right?, without necessarily going through the checkin process.
• Find people at the event with similar skills: this goes along with name tags and could actually apply beyond after hours as well. E.g. What if the name tags included the list of the person's skills and/or things they're looking to learn from their profile?
• Reminding people that their checking in helps us get budget + keep events free: This is basically a guilt trip/playing on feelings of the importance of reciprocity. It costs us nothing and I suspect it'd work pretty well.
• Free swag/give aways: I agree with what @rgardner said about it being a mess to coordinate and budget, and what @freialobo said about it possibly getting old.
• Some sort of gamification: I'm open to it, but we'd need to think carefully about what the rewards are. Check out this study, my interpretation of which has always been that "reasons compete". That means, by asking people to check in for rewards, we might actually dampen the effectiveness of appeals to help us with our budget, etc.
• Opt-in resumé sending: I really like offering this service to our members (and our sponsors)! But, for the same reason I'm a bit worried about gamification, I'm not sure we should advertise it upfront as the reason to checkin. Instead, people who do check in could just get a (nice) surprise email saying "Hey! We hope you enjoyed the event. Want us to send your resumé to company x?"
• Other ideas: I really like @AbhiAgarwal list of what members are hoping to get. Let's think of some other possible perks based on that!

Re freshman circuit and eboard meeting checkin: the checkin at those events would be more about accountability than about measuring attendance growth/diversity/etc, so I think we should use a different system. I'm imagining one page listing each possible attendee (i.e. all board or circuit members) with a checkbox next to each potential attendee's name. Then, the lead of the event just loads that page, checks the name of everyone who showed up, and clicks submit. This is less friction and it means that people who didn't actually show up can't check themselves in remotely (which there could be a reason to do, if real consequences ever got tied, e.g., to missing eboard meetings).

[freialobo]

@ethanresnick Yesterday at Design Days we had 30-35 total. We introduced the system before the speaker showed up, and a good number of people trickled in after, which partially explains the difference.

Re: stumbling blocks, I didn't see any but @claremcarroll @jhishan can elaborate on that. I do think the incentive issue was there, but we didn't particularly get any resistance either.

[ethanresnick]

@freialobo Did we promote the checkin at the end of the event for people to do on their phones on their way out? I think that's an important moment too if this is going to work, since some people are always going to come in late and, unless we have someone standing at the public computer to intercept them (or good signage?), they'll probably ignore the checkin box, no matter how easy the process.

[ethanresnick]

Btw, I recognize that, in the above discussion (and in my approach to event marketing/signups more generally), I'm making a big bet on facebook. Long-term, that's probably a bad idea, especially as facebook is already seeing adoption fall among teens. But, the good thing about this approach is that it still leaves us with the email of everyone who checks in (because we get it from facebook or, in the few cases we don't, we ask for it directly). So, when the time comes that switching off facebook makes sense, we can go to an email-based approach, like the one originally discussed in TechAtNYU/api#3. Until then, we get the UX benefits of attendees only having to find their name from the list of rsvps, and we don't have to worry about merging multiple emails into the same api account.

[maxdumas]

This is app is now at TechtAtNYU/event-checkin, with this discussion continuing at TechAtNYU/event-checkin#4. Closing this for here as it is no longer applicable.