Replace the elevator question with something similar

[ctro]

The Elevator question isn’t great, we should find a new one. We want to watch people think through a problem. Can we do that if we publish the problem ahead of time? This question does not seem fair, e.g. for front-end folks.

[ryanhofdotgov]

Any chance you could elaborate on what makes the question not so fair for front-end folks? And any other characteristics that make it not so great? Thanks!

[ctro]

During the retro we heard feedback from a couple folks that the Elevator question seemed easier-for/aimed-at people with more back-end experience. We heard that people with primarily front-end experience didn't have a lot to grab onto with this question.

The "fairness" of the question for people was almost a secondary point. In general people seem tired of the elevator question, and don't feel like they've been getting much out of it.

I think we should replace the e.question with a similar question. E.g. large, open-ended problem.

[ctro]

@amymok anything to add here? Or any replacement questions to suggest?

[amymok]

We should still do questions along the line of system design, because we would like to have engineers that have experience with large-scale design experience.

@ryanhofdotgov I was thinking the question may not be easily translated for people with more non-traditional background. It may imply an embedded system while most of the work we do focus on web and software applications. It seems to be hard for people to wrap their heads around the concepts, esp, it may not seem very apparent on the front-end component.

We can ask questions that are more closely aligned with the type of software we may do. Here are some good ones we can try: https://hackernoon.com/top-10-system-design-interview-questions-for-software-engineers-8561290f0444

[ryanhofdotgov]

@amymok Thanks for clarifying. What I like about the elevator question is that it isn't really technology specific at all – it's about general analysis, problem solving, scoping, abstraction, etc. "We can solve any problem by introducing an extra level of indirection." 🙂

As consulting engineers we are often thrown into domains (e.g., smart cities, air tasking orders, nitrogen dioxide emissions) we know little about – so as we look for a replacement question, my bias is definitely for one in this realm of higher-order thinking.

[amymok]

I understand your point @ryanhofdotgov. We should definitely get some frontend folks here to weigh in on what questions seem more fitting for them. Maybe @hbillings? Feel free to add anyone else.

[hbillings]

👋 The thing that I think makes this particular problem difficult for front-end folks is that... well, there's no front end. It relies heavily on back-end systems logic and data structures -- the best answers I've heard to it involve building request queues and logically optimizing the order in which floors are serviced. If the candidate isn't that sort of person, the interviewer probably won't get a great indication of their critical thinking skills. We're probably not going to exclusively staff a front-end dev to a problem that requires understanding gnarly data and systems problems, anyway!

One alternative that I've heard of is something along the lines of "Imagine you're building a computer system for an airline." This could be any kind of computer system: the user-facing system for booking flights, the software the gate agent uses, the airline's central flight scheduling software, anything. You can introduce various kinds of follow-ups depending on the direction they go: how might the airline optimize moving personnel around? what happens if there's a delay, a gate change, an overbooked flight? With something like this, you're still giving candidates a large, vague systems problem, but you're letting them choose which part of it they feel like they can talk about with authority, and you're framing it in terms of the kind of thing that they'd likely be building at 18F. Making this question a bit more flexible would let us get to see where someone is most comfortable problem-solving on the fly.

I also want to say that I felt like the front-end issue was just part of what was raised in retro: I feel like the more overwhelming theme was that folks just don't really feel like they get great, insightful answers to the elevator question across the board.

[mgwalker]

My problem with the elevator question is that the wording is very broad and candidates seem to struggle with what exactly we want to know. Designing big complex systems is definitely something we have to be able to do, but it's not something we can reasonably ask someone to do in an interview. I feel like the context/constraints of the interview format make it hard to even know where to start - you've only got a few minutes, versus if you have a few hours or days.

I wonder if we can get at the system-thinking aspect by asking the candidate to talk us through changing the behavior of an existing system. Leave the system description broad so they can talk about what they see as potential components of the system and the interactions, but narrow down the specific task.

And I really like the "computer system for an airline" idea from @hbillings. I think it also speaks to different systems in a clearer way than an elevator does. E.g., who are the stakeholders in an elevator, versus who are the stakeholders in an airline ticking system? I think the latter is easier to quickly grasp, but maybe not.

[kfoley-18F]

One question I've used in phone screens prior to 18F is "Let's say you and your team are tasked to build a feature or an app that would support photo sharing. How would you approach the design and build for this?"

I've intentionally left it open-ended, and it's been interesting to see how people ask questions (or not) to get started. Some folks dive into an architecture and start talking about the parts, some folks start with some user research or feature questions, some folks talk about workflows and integrations with other existing tools. It has interesting directions to probe for UX, storage & data (formats), scalability, testing, architecture, integrations, and APIs.

Nearly everyone these days has used a photo-sharing thing (social media, mobile, website, etc) so it can go in lots of directions.

[cantsin]

I'm all for retiring the elevator question since it was one of the least effective questions I've asked during interviews. I get almost no information out of it, and when people are prepared to answer this question, I feel like I don't get any insight in how they would approach any other problem.

Open ended questions are better, imho. but @kfoley-18F beat me to it :grin: A discussion on how to change existing systems would work too. Just ... anything other than this elevator question, please.

[echappen]

Adding @lauraGgit's prompts here for discussion:

Design a messaging app

• How would you design one-on-one conversations between users?
• How would you extend your design to support group chats?
• What to do when the user is not connected to the internet?
• When to send push notifications?
• Can you provide end-to-end encryption. How?

or

Design a search type-ahead

• How to store previous search queries?
• How to keep the data fresh?
• How to find the best matches to the already typed string?
• How to handle updates and the user is typing too fast?

[alexbielen]

Just a thought:

Similar to the technical pre-work, we could offer a variety of system design questions and have the candidate choose one that best suits their interests, background, and experience. This might be easier than finding a single question that has something for everyone.

[mark-meyer]

I thought the same thing @alexbielen. If the goal is to make this question a better fit for a wider variety of engineers, it might difficult to settle on a single question. For example, the type-ahead question seems like a natural fit for a front-end developer who could really shine with a good answer. The messaging app doesn't really seem as useful for finding great front-end talent but has a lot to chew on for someone interested in protocols, data, encryption, etc.

[hbillings]

I'm going to leave this question open, but wanted to record here that for the next round of interviews, we're going to try out the photo sharing app question in the interest of time. I like the idea of having a bank of questions that the candidate can pick from, though!