As someone who, not too long ago had to slog through stuff like this I had to stop by and offer my thoughts.

This is not a good look and really stupid from all sides.

This is a LOT of work just for, from my understanding of things, the "chance" for an interview. I mean come on, this is only a few steps away from being a complete product.

If someone is looking for work chances are pretty good that a large chunk of their time is already spent... well just trying to get by.

Who the fuck has time for this much extra work all without pay?

Look, I get there's no easy way to judge someone's technical abilities but this really is going too far and asking too much of an applicant; heck, I still think it's too much to ask even during the interview process.

Some other ideas

Since you're looking for a lead from what I can see why not... look at their existing work? Chances are pretty good they have something to look at, at least in code form if not something on a website.
Why not just have them implement a small portion of the required functionality, say just a list of the first page of testimonials without the extra stuff like pagination, track listings, back-end filtering(is this not a front-end role?) etc.
Why not ask them to make something fun? For example, I've been asked to make little mini games in the past.
Another idea - assuming you like them after talking to them and they appear to have the general qualifications, why not just ... maybe hire them on a trial basis? Treat them like a contractor for an agreed upon period of time and once that is done, assuming no major screw-ups or extremely poor performance, if all parties want to continue the relationship, hire them as an actual employee? Seems like a win-win to me, at least for a little while. The potential employee gets a paycheck for a little bit and possibly something to show or at least talk about and you folks get the help you're looking for.
Of course, depending on their situation this may not be something every applicant is able to do but it is, at least in my opinion and based on past experience, a far more equitable way to go about looking for a potential hire.

I will give you folks credit though for not asking a candidate to run through a just as idiotic Leetcode / HackerRank style interview but In any case, this is not the way to go to try and hire someone.

PS - just axe the video pre-interview introduction, that's pretty dumb as well.

Hi. Thanks for your thoughts. I'd love for you to read my overarching response (see https://github.com/exercism/job-application-process#background) and then let me know your thoughts here. Many of your ideas require either time that we don't have to apply to hundreds of candidates, or money, which we don't have. So how we can solve this with those constraints is something I'd love to work out. Thanks.

@iHiD Again and again, all of your responses boil down to "my time is too valuable to waste on talking to people we might want to hire". In which case, I would suggest that perhaps you shouldn't be hiring if you're not willing to commit the appropriate amount of time to it. Give the job to somebody who has the time and energy (and dare I say, the prior experience) for it.

Personally, if I were looking for a job, and the first interaction with a company was like this, I'd not even bother responding to them. If you can't spend the time to talk to me as an interview candidate, then I doubt it's going to be any better as an employee. Do remember that the interview stage is one where the company should also be impressing the candidate. Not just the other way round.

@asleepysamurai If you have a suggestion as to how this application process could be improved, I'd love to hear from you. Right now, all I'm seeing is you suggest is that I should interview 200 candidates per day, which I doubt is what you're intending to imply. What exactly does your ideal hiring process look like?

I don't think anyone is actually suggesting you interview 200 candidates a day. That's a bit fallacious.

What does the hiring process look like for your three other job roles? This sounds far more reasonable for first contact.

Firstly, if you're about to apply, thank you! We're grateful and excited that you're taking the time to do so 🙂

To apply send an email to jobs@exercism.org with:

Subject of "Educational Content Factotum" A couple of paragraphs introducing yourself and providing some brief background. A link to a short video introducing yourself in more detail

I don't think you need to open this for input really. Just do the same thing for this role as you're doing for the others.

initial contact > narrow down the pool > short interviews > narrow down the pool more > then when you have a small pool of potential candidates give out a small challenge, look at existing work or hire them as a freelancer on a small project.

I think Cassie basically sums up my feelings more or less of what I think should be done but if you want my more long-form opinion ...

First off, kudos to being open to feedback. This is definitely a pleasant change compared to some other past experiences I myself have experienced and seen through other stories. Though it’s still not the best solution in my opinion I have to give credit for at least recognizing how much of a time suck the original project was and splitting things into separate parts.

I also have to give credit for offering feedback; I often times have asked for feedback only to never receive a response or told they couldn’t disclose that information simply for nebulous “legal reasons”

That all being said, I've looked over your response and I don’t think my opinion has changed very much if at all.

First off

To be clear I am not advocating for getting rid of some kind of test; again I recognize that it can be difficult to judge someone’s technical capabilities; that being said what you’re putting forth is still way over the top and a bit much to ask of an applicant.

I will try to address the bigger points that I take issue with both in your post and your initial response to my comment. I have spent far more time thinking about this than I want to or should have to so I may a little direct and short but please feel free to ask for clarification if something I mention isn't clear.

Being unethical and exploitative

I can agree that if you were to base the usage of the words on their direct meaning, the argument may not make sense. However I think you’re taking the meaning and use of the words too literally.

As you’ve described, the work already exists in your folk’s product in one form or another. I have no reason to doubt you on this.

However, regardless of whether or not the code already exists, asking someone to reproduce something to a level where it's only steps away from being able to be inserted into a product(or be a product itself) is the issue I think everyone has with this.

While asking someone to go through this “technically” isn’t being unethical or exploitative, how is the situation any different if you folks actually were trying to exploit the applicant and gain free work?

Again I applaud you for recognizing that the overall task is quite long for a lot of people and splitting things up but that in my mind may not still be enough depending on how you proceed with the application logic phase.

(side note - I also don't see the point of splitting it all up if "...the CSS is all easily extractable from Figma." Also I'm curious about this line "All filtering of the testimonials should be done at the server-side level via API calls.")

Why not ask someone to do a much simpler task? You claim it will be cost you more time and money but ... how?

Lack of time

This is a common refrain I hear from a wide variety of people doing hiring and this, for me at least, is not a valid reason.

If you’re getting flooded with applications, then the logical step would be to, well, stop taking applications for a bit no? Alternatively, and I have seen this done before, you could specify an application period and cut things off for a bit while you review.

If the issue is cost in terms of advertising, then that’s just the cost of doing business, not to mention there are other avenues to looking for hires beyond LinkedIn and recruiters. Aside from some obvious avenues like forums or other public places, did you try any of the smaller job boards? I know you're in the UK but for the sake of example, Tokyodev is one where it looks like you could negotiate a bit with the site owner and possibly work out a personalized deal.

About “wanting” to work at Exercism

I also wanted to call out one more concerning aspect of your post - the idea of “wanting” to work for a company.

I do understand wanting someone who has a genuine desire to work at Exercism, it’s only natural to think that way.

However I find it concerning that that mindset is being factored into the application process because sometimes, people just need the work.

Now this doesn’t directly apply in this instance of course but just to give an example of one of the reasons of why I bring this up; here in the US, as you may be aware of, we have an utterly idiodic system for getting healthcare.

The most affordable and usable coverage is really only available if you’re either incredibly poor, have the good fortune to hold a job with benefits or are simply old.

For everyone else in-between, which has included myself at various points in life, you pay some pretty hefty prices on your own.

As you can imagine, people here tend to take any job they can, even if it happens to be something that they don’t have 100% interest in. It shouldn’t matter whether or not someone “wants” to work at a particular company as long as they have the necessary qualifications.

Looking at existing work / CVs

Visually, we don't know what their work was supposed to look like. Does what they're showing us look like what the designer created? (have they worked with a designer before?) Is the design they implemented ugly to our eyes, but their implementation is perfect. I don't know how we could validly filter based on this.”

I’m having trouble understanding why visual fidelity and design should even factor significantly in here unless you were hoping to hire some kind of designer/developer hybrid(in which case you should make that clearly stated). It shouldn’t matter if what is being presented looks terrible, you’re hiring a developer and not a designer. For all you know the designer could simply be terrible. I personally have worked many freelance jobs where the end result was definitely not aesthetically pleasing but, it was what was spec’d.

At the end of the day you ought to be looking at code and evaluating based more heavily on that. it shouldn’t matter what the end result was, what should matter is how the person went about a particular problem and seeing if there’s anything in their work that doesn’t quite make sense or could have possibly been done better.

Does it function the way an applicant says it should function?
Can they explain how they got to the final result they’re presenting you?
Have they done a project postmortem? Do they have any regrets or afterthoughts as to how to go about something after the fact?

These are just some of the things you should actually be focusing on instead of the visuals.

It biases everything towards people's past experience. If they've worked in jobs where they don't have something to show (e.g. in a walled or closed-source system, or as part of a frontend team) they're going to immediately get rejected on that basis. In which case, we'd be asking someone to have contributed to OSS or had hobby projects, before they can apply, which is something also problematic

I have to admit, considering the job description specifically mentions "lead", I don't understand this line of reasoning.

If they truly have "lead" experience, they're bound to have SOMETHING to show, even if it's tiny. While I don't expect someone to be furiously coding even during off-hours, given how quickly the web has changed over the past couple of years I would expect an applicant to have something to show that demonstrates they've been keeping up with some of the changes.
Even if they're under NDA and can't expose the actual code underneath, they might still be able to share the final product. Could they lie? Sure but there are obvious ways around that, for example asking for references perhaps?
I can kind of understand OSS contributions but why on earth is it problematic to ask for hobby projects? Again if they really do have lead level experience, while I don't expect anyone to be constantly coding it's incredibly likely they have SOMETHING to share, on or off the web.

We could rely on people sending resumes, but that's hugely problematic too. It biases us towards people who have had the opportunity to have good jobs previously. It doesn't allow for applicants who have built great skills but not yet had the privilege of great jobs.

Why are you focusing on applicants that have held "good" jobs. Sure the big names stick out but you're making it sound like it's some huge insurmountable task to even look at the people who haven't been as lucky to land in a big company.

This leads me into my final point...

Paid Apprenticeships/Contracting/Freelance work.

While it is certainly not a one-size fits all solution, I still believe this is the most equitable way to evaluate and possibly hire someone. While it is still a “chance” as it were, at the very least, the applicant is getting paid for their time and effort. On your side it offers you folks the opportunity to take some of the load off at least for a little bit and see how that person actually functions on-the-job vs some heavy pressure filled video chat

To go back to your initial response in regards to time and money, and to be frank, I didn’t think I would have to actually directly specify this, but I’m not saying you should do this for every applicant. That’s obviously not a realistic solution.

Interview people like a normal company first. Look over their CVs, check out any work they might have and have a chat with the people that stick out. Figure out which one you like the most, have everyone come together and agree upon some terms and offer that person the opportunity.

Alternatively, if you're worried about taking too much time, like Cassie also mentioned, figure out something in your pipeline you think the candidate can do and offer them a project. (while paying them of course)

To end

All in all my thoughts remain largely the same even after reading your post. I mean come on, while often times even their processes are rife with problems themselves, not even the big names ask you to do this much work.

@cassieevans

Whilst I'm sure Jeremy will respond fully, the only thing I'd like to point out is that there are actually comments that say "you should just make that time or pay someone to hire it, or not hire people", when discussing the number 200. The number itself was based on a prior job offer for a different company, where the initial step was "send in your CV" (like suggested by others) and we're literally talking hundreds of e-mails a day. This is no exaggeration, and we expect it's because it's a fully remote job, with shit-for-us-and-london-standards-but-otherwise-decent-pay, which this one is also. Doesn't change your point, but it's a number that has some footing in reality.

That said, I do resonate with what you're saying. I think that we could discourage people who we know won't be a good fit (purely from that first e-mail or first chat or second chat) before venturing in "here do this 1-5 hour project".

@sortofsleepy

Also I'm curious about this line "All filtering of the testimonials should be done at the server-side level via API calls."

I know this is not your main point, but I can answer this for you. It's explained here:

Access the testimonials from: https://exercism.org/api/v2/hiring/testimonials

You can use the following params to modify which records are returned:

page: Specify to return a different page (e.g. 2) of the results (note: page 1 is the first page).

track: Pass a complete track slug (e.g. python) to only return the Python track's testimonials.

exercise: Pass a partial exercise name (e.g. ming) to only return the exercises that contain the word "ming".

order: Specify either newest_first or oldest_first (default) to switch the order.

In other words: pass the correct parameters to the API call and display the results.

I won't respond on the other points, but I am certain Jeremy will and appreciates you taking the time to write them out. The only two thoughts I had when reading this were:

The final sentence which is, from my own experience, just wrong. I worked in Silicon Valley for various years and the big names ask much more than this, and have even more idiotic processes. I personally don't want Exercism to be like that, and in general, I don't think it's wise to compare yourself to the big names in the first place (as they have horrible practices all around, such as lacking Accessibility, concern for individualism, and hate for privacy).
The designer is actually not shiiiit, and hiring a developer whomstve is unable to implement that design would not be a good fit for the role. Visual fidelity is important for a front-end developer role, in my opinion, especially for this one.

Side note: I am absolutely aware of the practices of some other companies which literally steal applicants work and use it in production, which I don't have words for. It makes me very sad. I think there has been naivety on Exercism's part to not recognize that the link like that was easily made, despite this not being the case for this job offer. Good thing we can still learn, and now we (all) know for the next time.

@SleeplessByte

I know this is not your main point, but I can answer this for you. It's explained here: ...

If I'm understanding what you're getting at, you're basically saying that the line I quoted is saying the user shouldn't have to worry about handling filtering, etc correct? If the sole task really is simply to have the applicant call out an API from the front end then the line I quoted really should be re-written or removed as it, still now, reads to me as if you folks are expecting the applicant to write server side code in order to process API calls and filter responses

The final sentence which is, from my own experience, just wrong. I worked in Silicon Valley for various years and the big names ask much more than this, and have even more idiotic processes. I personally don't want Exercism to be like that, and in general, I don't think it's wise to compare yourself to the big names in the first place (as they have horrible practices all around, such as lacking Accessibility, concern for individualism, and hate for privacy).

I have worked for some of the big names as well and as I'm sure you're well aware of, teams generally tend to have different practices when it comes to hiring. Personally I've only ever had one actual project-type assignment during an interview with one of the big guns, and even that was FAR less work than what you folks are asking of applicants here.

To add to that, I happened to grow up in Silicon Valley and I've heard the stories so I'm not doubting your experiences at all. I even acknowledged the Valley's notoriously poor practices in my last line. My point was simply that, based on my own experiences and experience I've heard from others, for front-end roles like this one I've never heard of anyone being asked to do this much work during the interview stage. In hindsight I could have made that more clear but the post was already quite long at that point and it didn't occur to me to clarify.

The designer is actually not shiiiit, and hiring a developer whomstve is unable to implement that design would not be a good fit for the role. Visual fidelity is important for a front-end developer role, in my opinion, especially for this one.

I uh... never made fun of the designer? I also never said visual fidelity wasn't important. I was responding to the comment in the post about the weird inability to judge an applicant's past work. For example

Is the design they implemented ugly to our eyes, but their implementation is perfect....

...... Is it not enough the applicant did what they were asked to do?

Are you folks not looking for a lead developer like job posting says? It stands to reason that if someone has really that level of experience, they can speak to that experience and explain or better yet, show their past work which can offer insight into their ability to style things.

In any case, my point was simply that whether or not an applicant's past work looks visually pleasing should be far lower on the totem pole of needs; maybe that's simply how that past work was intended to be. If the code they show makes sense for the project that they're presenting then that's what should matter more.

Not to mention, and I haven't looked at the file myself but if this is true

As many people pointed out in Reddit, the CSS is all easily extractable from Figma.

What is even the point of placing such a high level of importance on visual fidelity then?

Whilst I'm sure Jeremy will respond fully, the only thing I'd like to point out is that there are actually comments that say "you should just make that time or pay someone to hire it, or not hire people", when discussing the number 200

Yes - I understand the number 200 isn't being pulled out of thin air, but no one is saying interview 200 people a day. That's reducing the argument to an absurdity. That would be absolutely impossible to do.

Luckily the options are not 'Interview 200 people a day' or 'Give people a huge challenge before first contact'

Hope you find a happy middle ground.

@cassieevans Yep! :)

@sortofsleepy

Thank you again for the long answer. It's far more helpful than the majority of other interactions I've had on the socials. Muchos appreciated ❤️

[...] filtering

Well, they do have to worry about it, but it is completely implemented for them on the server side (which Jeremy provides). The applicant does need to implement the logic to "change the API call" with the appropriate options. Would you have a recommendation how to change the wording so it:

seems less daunting
more accurately describes what you need to do

Personally, I found the description in the README to be clear enough to see that you don't need to implement server code, but I understand that this may not be the case for others.

In hindsight I could have made that more clear but the post was already quite long at that point and it didn't occur to me to clarify.

Thank you for making it clear anyway. It makes sense, and as mentioned before, it has come up during our weekly community call. Reducing the amount of work is definitely on the list of changes Jeremy will implement.

I uh... never made fun of the designer? I also never said visual fidelity wasn't important. I was responding to the comment in the post about the weird inability to judge an applicant's past work. For example

I didn't mean to imply that you did. It was a response to the following comment:

I’m having trouble understanding why visual fidelity and design should even factor significantly in here unless you were hoping to hire some kind of designer/developer hybrid(in which case you should make that clearly stated). It shouldn’t matter if what is being presented looks terrible, you’re hiring a developer and not a designer. For all you know the designer could simply be terrible. I personally have worked many freelance jobs where the end result was definitely not aesthetically pleasing but, it was what was spec’d.

What I meant to convey is that looking at past work, Jeremy wouldn't be able to tell if:

the designer there wasn't great (like you mentioned as an option)
the developer wasn't capable of implementing the design

You say that it shouldn't matter, but it does. Being able to make decent decisions/trade-offs when implementing a design is important -- for this type of front-end engineering role. The fact is that there are a lot of developers whomstve may be really good on the engineering side, but don't have a "eye for detail" when it comes to design implementation. This doesn't dismiss them as developers, of course. It would make them less suitable for this role, for the mere fact that the expectations would be high for this role.

(This is not an agreement or disagreement of the process Exercism currently has. There are surely better ways to test this)

What is even the point of placing such a high level of importance on visual fidelity then?

If you copy all the CSS from the figma, the entire page will be absolutely positioned. The applicant would need to understand CSS in order to make that work without absolute positioning (as you would almost never implement a page like that). There are choices to be made from a maintainability standpoint, such as repeating colours and other theming.

Other than that, only Jeremy can answer that question :)

(I have to get ready for work so this may get too succinct but anyways)

Thank you again for the long answer. It's far more helpful than the majority of other interactions I've had on the socials. Muchos appreciated ❤️

To be honest I am still a little ticked off and flabbergasted that some of what I've said even has to be said but I understand flinging mud at each other is not a great way to begin discussion. I also want this process to improve for everyone, not just at Exercism but at other companies as well. I've also been through too much crap that I hope I can help start change for others, even if it's only at a small company in the UK/Europe.

Well, they do have to worry about it, but it is completely implemented for them on the server side (which Jeremy provides). The applicant does need to implement the logic to "change the API call" with the appropriate options. Would you have a recommendation how to change the wording so it: seems less daunting more accurately describes what you need to do Personally, I found the description in the README to be clear enough to see that you don't need to implement server code, but I understand that this may not be the case for others.

(pardon the formatting - seems there's not a great way to quote something with too many line breaks)

This is the line that I think, will be confusing for a lot of people.

"All filtering of the testimonials should be done at the server-side level via API calls."

My reading of this is that the applicant is expected to write some server side code calling out to the api, do the filtering then pass it to the front-end. If all you're expecting of the user is to just to write front-end HTTP requests to an already existing api like the rest of the readme describes, then I don't understand why this line is in there. The simplest thing to do would be to just remove that bullet-point.

I didn't mean to imply that you did. It was a response to the following comment:...

That's not exactly how it reads, especially if you leave out the last sentence but ok, I'll give you the benefit of the doubt

You say that it shouldn't matter, but it does. Being able to make decent decisions/trade-offs when implementing a design is important -- for this type of front-end engineering role. The fact is that there are a lot of developers whomstve may be really good on the engineering side, but don't have a "eye for detail" when it comes to design implementation

I never said it "shouldn't" matter but that compared whether or not the applicant understands how a web page is put together , final visual fidelity should be lower on the list of things to look out for. Again, if Jeremy is looking for some kind of design/developer hybrid person, then that should be clearly stated.

If he's solely looking for a developer then that person shouldn't have to make any kind of minuscule design decisions. They're there to build something to spec and that's it. Of course, there are going to be situations where some kind of decision has to be made in terms of design and flow but, again, if a lead level person is who he's looking for, it should be clear from evaluating the source of past work or hearing someone talk about their past work to get a sense of whether or not the applicant can recognize those situations. Heck, asking what people did in situations like that has been pretty common in my experience.

To be honest I am still a little ticked off and flabbergasted that some of what I've said even has to be said...

I find the majority of non-constructive commentary on the various social platforms very US-centric and also shows that those people seem to never have tried to hire someone, nor do they realise work culture in the rest of the world may be different. This comment is not directed at you (or anyone in this issue for that matter), but whilst not an excuse, and not a good reason for the things that have happened, I do think is contributor to the disconnect (e.g. You see very similar "wtf is X doing Y" around subjects of salary/wages (and its height), tipping culture, and PTO.).

Non of that dismisses any of the problematic things which is why we (truly) are grateful that, despite perhaps the obvious nature of the comments, you (and others) do chose to write out the thoughts.

Which brings me to:

[...] I've also been through too much crap that I hope I can help start change for others [...]

...to which I can only say: I am sorry you had to, but thank you for contributing now ❤️ .

[...] but I understand flinging mud at each other is not a great way to begin discussion.

It just isn't going to help anyone.

The simplest thing to do would be to just remove that bullet-point.

Yeah, that we can definitely do.

The rest of the comment I leave for Jeremy, as I can't and don't want to speak for him.

(p.s. I am on the spectrum and subtleties between the lines as well as sarcasm are very hard for me to detect, especially online. I therefore try to take everything at face value and guess how people mean things, but cross-culture communication makes that even harder. Hope this helps understanding the way I respond to your commentaries)

exercism / job-application-process

This is .... a lot #34