Idea for different on-site flow; looking for problem ideas

[bobmazanec]

Framework

Hoping to encourage (force?) more interaction with and get more information about on-site candidates, I offer this "flow" for discussion and ask for suitable problems / challenges:

1. Some days (3? 5? Up to 7?) before their scheduled on-site
   • give the candidate the description/requirements
   • encourage them to ask questions
   • allow them to use any tools, languages, etc.
   • advise them to be prepared to present their solution — including design choices/trade-offs, commit/development history, etc.
2. During their on-site
   1. the candidate makes their repo/history available to the team
   2. 90 min.
      1. the candidate presents their solution
         • how much time they worked on it and their time constraints (e.g., classes, current job, family, etc)
         • demonstration, design, code walk-through, etc.
      2. Q & A (limited to the project)
      3. team/leader presents new and/or changed requirements
   3. 10 - 15 min: solo thinking if the candidate has a strong need/preference for it
   4. 30 min: candidate and team discuss design options/ideas to meet the new requirements
   5. 2 hr.: candidate works on the changes, engaging team members as needed/desired
   6. 45 min
      • candidate demonstrates and presents the changes
      • Q & A — not limited to the project and changes

Information I hope / believe this will provide above and beyond the current on-site:

• what they can accomplish with more time
• how they handle new/changing requirements
• how they interact with the team in a design discussion

Full-stack-overflow developers can simply create solutions from search results even more easily than under the current on-site process, but I believe the walk-through and design discussions would reveal their incomplete understanding.

Need new challenges?

For this to work we need problem/challenge ideas…

• that are "interesting" enough to "deserve" advance time and effort from the candidate
• whose requirements can be changed/extended to require design consideration and 2(?) hours of development time

  Maybe the existing challenges could work?

• Chess

  require only "basic" move checking during the initial? add "advanced" rules — e.g., castling, "King cannot move into nor remain in check," etc?

• TinyURL

  do not mention "bonus" items during setup; select 1 - 3 as the "new" requirements (perhaps choosing one(s) that might be more difficult based on the candidate's initial design & implementation?)

  Other games?

e.g.,

• Battleship — one web implementation — even if simply single-user 'shooting' at computer's ships?
• Mastermind — one web implementation

(both could be implemented as non-web projects, too, of course)

Other ideas

Fetch/collect, process, and display some data?

I think we'd want to avoid 'classic' tutorial examples like microblog posts-and-comments.

[Ivanknmk]

Thanks for putting this together @bobmazanec

Few questions:

1.- Are we gonna tell candidates that they will get 2 more hours after the first presentation?

2.- I LOVE item number 4! This is a good suggestion to achieve the "what is like to work with him/her" question (we talked about it today). Is that the motivation? Normally, in those meetings we discuss actual solutions and trade offs, are you thinking in something like that? I think this one is particularly interesting but a potential failure if the candidate literally follows instructions instead of using his/her problem solving skills.

[legtzp]

It would be cool to try this approach and see how it works. What if the candidate is really passionate about the challenge and he actually covers all of the bonus points without even knowing we would ask for those on the on-site? Would we ask him to refactor his code?, do performance improvements? we should have a big list of extras in case this happens.

[eduardoromero]

I love the idea! Please read this post from the Auth0 team about their hiring process. They use a very similar approach to what Bob suggests here.

We can fine-tune it to each candidate and to our process. Just keep in mind that it requires commitment from both sides.

[pablasso]

I really like the idea. I do think we should let the candidates choose whatever format they like since there's people who just don't have extra-time after work/family.

Just a nitpick though, I don't think we should mention that we're going to evaluate for his "commit/development history" or similar traits since that is expected from a developer, we can get a good read by just letting them do whatever they're used to.

[edregalado]

Awesome Bob! This is exactly what can be extracted for having these experiences @bobmazanec Regarding @pablasso 's comment, some colleagues did not know version control a priori their entrance to Wizeline; however, I agree to him on this is expected from a developer.

We should have a pool of more 'nice to have' features in order to have a better experience in this kind of challenge.

I love no. 4 too, a quick debrief on what the candidate has done and get to an agreement on what would be great for the candidate to challenge their skills.

[zxul767]

How about a wiki? I think it fits very well the idea of an "expandable" problem with many potential features for the user to implement after an initial version.

A couple of years ago, I was asked to implement such a project at the end of an introductory web development course and found it both interesting and useful.

Features for such a project might include:

• All users can see public wiki pages (but they can also register to gain access to more functionality)
• All users (registered or not) should be able to see a wiki page (the latest version by default)
• All users should be able to search for content and get back those pages where there's a match (except for "private pages" which would only be visible/searchable by authorized registered users)
• Registered users should be able to:
  • Create and edit pages
  • Create "private" pages (which only users authorized by the page creator can see)
  • See the history of a page (showing the creator and contributors)
  • See past revisions of a page
  • Restore a previous revision of a page
  • Create the newest version of a page based on a past revision
  • See the differences between two revisions of a page
  • See changes done to a page by a specific user
  • See all pages created or contributed to by them

You can take a look here to get an idea of what a simple implementation (with a small subset of the features above) might look like. In that implementation, there is no explicit button to create a new page, you just simply type on the browser's bar the URL that you'd like to assign to the new page (e.g. http://wiki-767.appspot.com/my-new-wiki-page) and if the page doesn't exist yet you get redirected to an editing form where you can create the page.

I didn't give much thought to ranking the features above by their design/implementation difficulty. We could discuss that if you think it'd be a good project to include.

[bobmazanec]

Thanks for the feedback, ideas, and questions so far (and please keep them coming)!

@Ivanknmk:

Are we gonna tell candidates that they will get 2 more hours after the first presentation?

Sounds right to me.

re: the post-presentation discussion

"what is like to work with him/her" question […] Is that the motivation? Normally, in those meetings we discuss actual solutions and trade offs, are you thinking in something like that?

Yup!

@pablasso:

…I don't think we should mention that we're going to evaluate for his "commit/development history"…

Good points from both @pablasso and @edregalado. I would also like to see when the commits are made in case we want to ask about time management. Maybe "if you already know and choose to use an SCCS, please share it with us."

@zxul767:

How about a wiki? […]

Neat idea — definitely deserves more thought & discussion IM(NS?)HO!

To make initial 'experiments' with this as easy / simple / 'inexpensive' as possible, I'm feeling more positive about using the existing Chess & TinyURL problems — e.g., we don't have to build new requirements from scratch.

That said, we need more problems no matter how we run the interviews — the wiki (and any other ideas!) could be developed and used in any "flow."

I'll start working on a PR to propose and discuss specific details.

[bobmazanec]

Possible change of plans: we might try this on Tue., 1 Nov.

The idea for this 'flow' came up while several of us were discussing how to proceed with a particular candidate; that candidate's on-site is now scheduled for Tue., 1 Nov.

Valeria and I will get the 'basic' requirements for this flow for the TinyURL problem to the candidate and offer him a choice:

• implement this and bring it on Tuesday; or
• do a non-web challenge on Tuesday

[bobmazanec]

What I gave to Valeria to give to the candidate:

How would you like to do your on-site on Tuesday, 1 Nov.?

• Bring your implementation of a web project (described below); or
• Implement a non-web project over approx. 4 hours during your on-site

If you choose, the non-web challenge, please select one of these languages

• C
• C#
• Go
• Java
• JavaScript
• Python
• Ruby

No matter which challenge you choose, we will ask you to present your design and implementation; if you choose the web project, we will ask you to improve / extend your implementation during your on-site interview (see below).

The Web Project

Produce a "TinyURL" service:

• like bitly or Google url shortener

• when I enter a URL I get a "short" URL

  e.g., https://wizeservices.com/careers ⇒ https://localhost:3000/corto

• when I browse to a "short" URL, it redirects to the original URL

  e.g., https://localhost:3000/corto ⇒ https://wizeservices.com/careers

• when I visit some specific URL, I see a list of the short and long URL pairs in the system

Work at any pace and any schedule you like between now and then.

Your implementation must…

• include at least some "minimal" front-end user interface for creating and listing the URLs — i.e., it cannot be an API-only implementation
• be able to handle at least 1,000,000 unique "short" URLs
• be in some kind of Source Code Control System you can share with us (e.g., a repo on Github, Jira, etc.;, a copy of your local repo; access to an SVN server, etc.)

Your implementation may not call any external URL-shortening service. Beyond that, feel free to use any language, tools, frameworks, online resources, etc. to implement your project.

During your presentation, please be prepared to...

• demonstrate your project
• discuss your choices (e.g., language, tools, framework, design, etc.) and trade-offs
• discuss how it should be able to handle concurrent requests

Following your presentation:

• we will present new or changed requirements
• you will lead a discussion of how to meet them
• you will have some time to make the changes

Finally, you will demonstrate and present your revised project.

Thanks!

[bobmazanec]

Re: using puzzles/games, hopefully not many candidates know about https://github.com/leereilly/games?

[oryws]

So, we used this experiment in a candidate already and I liked it, it gave us the time/opportunity to figure out that the candidate was more suited for a frontend-exclusive position. Now, for any experiment to be valid, we need to run it more than one time, the candidate we used for this was not a strong candidate (in fact, he had two no's in his interviews). I would suggest to run this at least two more times, once with a very strong candidate, and once with one that's in the midpoint

[pablasso]

Cool! is there any place to consult the candidate's work?

[bobmazanec]

@pablasso https://gitlab.com/ricardohs35/short-url (we should probably ask him to either delete or make it private soon…)

[Ivanknmk]

@bobmazanec I just asked him to do that in a gitlab issue.

[eduardoromero]

BTW here's the public URL for his project: http://tiny.onizuka.cloud9.mx/

[bobmazanec]

Updates

• still no PR yet 😳
• interesting sequence today:
  • candidate received project instructions last Friday (11 Nov.)
  • on the day scheduled for the on-site, he e-mailed essentially, "I didn't have time to do the project; now what?"

[guscordova]

Just tried this onsite flow today, it's a great opportunity to get to see more about the candidates. However I had some trouble being able to assess the candidate's work since I had no idea how much effort had been put into it.

To me, this is the hard part:

• If you share the challenge a week early with all candidates, each candidate will put a different amount of effort to it (probably depending on their job, free time, etc), thus assessing the candidate's experience vs ability to do work vs effort put into the challenge becomes really hard when there's no exact understanding of time.

Recommendation: Github or any other type of version control is an essential tool assess this, but asking the candidates to keep a simple log of how much time they spend designing, researching, etc could potentially help understanding how much effort was put into the challenge.

[bobmazanec]

Challenge idea: "ages" ago, there was a "Reddit-like, posts and replies" challenge for a 2-hour pre-on-site @isa-bel and/or @kinduff?

[guscordova]

@bobmazanec are you referring to this one?

[bobmazanec]

Yup 😳 I had convinced myself it was 'hidden' in Google Docs — confirmation bias for the lose

[bobmazanec]

Some random ideas — maybe they'll spark more ideas, discussion, and possibly PRs

• include integration with some other service(s) in some web problem(s) to see what the candidate already knows and/or can learn

  e.g. (based on TinyURL; sorry!)

  • record 'clicks' and other 'analytics' data in a Google Sheet
  • generate a Google Sheet(/Doc/Presentation?) with analytic data (and maybe even graphs?)

• implement Chess (and/or other games?) as a web server project

[bobmazanec]

FWIW, @salvadorbfm wrote this up (the words, anyway — I may not be representing his formatting well here) and Ceci sent it to a candidate yesterday (20 Feb 2017):

The Chess

Implement a human vs human Chess game:

• It must be based on the code we provide which is a way to draw on a board.
• It must be based on an input string instructions (i.e., a1a2 which means from a1 to a2).

For more information about how to use the code base that we provide, please see the README.md.

Your implementation must…

• Include a general design about the architecture that will be used.
• Include the code implementation for at least four pieces. The implementation must contain a way to validate the movement of each piece.

We will ask you to present your design and implementation; also we will ask you to improve/extend your implementation during your on-site interview (see below).

During your presentation, please be prepared to…

• Demonstrate your project.
• Talk about your architecture.
• Be prepared to answer questions about your design and implementation.

Following your presentation:

• We will present new or changed requirements
• You will lead a discussion of how to meet them
• You will have some time to make the changes

Finally, you will demonstrate and present your revised project.

It’s very important that you include a journal where you explain how much time you spent planning and coding to solve the problem. Be prepared to present that information to the team.

[eduardoromero]

I like it! Asking them to have a "journal" is great so they know we want them to track their work and thoughts.

It also makes clear the format and what to expect before/during/after.

[bobmazanec]

Closing: technical onsites will not be project-based any more