search

gitcoinco / skunkworks

experimental laboratory
63 stars 35 forks source link

code coldoutreach.io #38

Open owocki opened 6 years ago

owocki commented 6 years ago

background

third party recruiters add little value to the ecosystem, and are ripe for disintermediation. see my body of writing on the subject at https://owocki.com/tag/recruit-engineers/

i think the future is one wherein individuals recruit for their own projects, with the help of software.

i would like to provide a tool for repo owners to write cold emails that aren't so off-putting. keeping in mind that as repo maintainers, we have to recruit talent for our projects, it is important to provide tools that guide a user into forming meaningful connections with recruitees.

proposal

i would like to build a tool that creates good cold outreach emails.

by syncing the linkedin profile of a user when they first land on the tool, we could get information about the senders place of employment, skills, schooling, living location.

from there, we provide an interface wherein one could paste the linkedin profile of a candidate one wants to reach out to..

and receive a MOSTLY custom recruitment email to send to the other party.

Design is done!

Please see the design referenced on https://github.com/gitcoinco/skunkworks/issues/20#issuecomment-355878936

Bounty: Code this up!

Coded pages

  1. ladning page
  2. 'connect with linkedin' page
  3. paste candidate profile url page
  4. generated email page

Functionality

Technical considerations

tra38 commented 6 years ago

I can definitely work on writing the code to generate emails. I would also be happy to accept help with the CSS, thereby splitting the bounty. :)

gitcoinbot commented 6 years ago

This issue now has a funding of 0.25 ETH (308.0 USD) attached to it.

owocki commented 6 years ago

donno if this is the right amount of funding... let me know community ^^

vs77bb commented 6 years ago

@nlaz Interested?

tra38 commented 6 years ago

donno if this is the right amount of funding... let me know community ^^

I would argue for increasing the funding, since this issue is really composed of two tasks - (a) building the script (client-side or server-side) that will generate a customized cold-call email, and (b) building the web application that allows people to use the script. I'm not sure how much additional funding would be needed though...maybe ~0.25 ETH might be enough to cover the broadened scope.

For comparison purposes, here were the rates charged for two similar ETHWallpaper gigs - 0.27 ETH ($180.36 at time of bounty-close) to write the Python script, and 0.49 ETH ($351.79 at time of bounty-close) to create the web app that allow people to use the Python script).

stojce commented 6 years ago

I am able to finish the whole thing. Just let me know if you guys are not going to do it. I agree with the increasing the bounty.

eswarasai commented 6 years ago

@tra38 -- I might be able to help you with the Frontend if you're planning to split. @owocki -- Let me know how we're planning to go ahead with this. Thanks.

owocki commented 6 years ago

I'm not sure how much additional funding would be needed though...maybe ~0.25 ETH

Please move forward assuming that an additional 0.25 ETH will be provisioned for this. I can't update the bounty itself because Gitcoin doesn't support that (but will soon!).... but you have my word i'll tip on top of the current amount.

Can you all decide amongst yourselfs whos doing what and how much of the bounty that's worth? @eswarasai @stojce @tra38

eswarasai commented 6 years ago

@owocki -- I think it's fair enough to split the total bounty 50-50 for Frontend as well as Backend. @tra38, @stojce Let me know your thoughts on this.

I'd be able to handle the entire Frontend tasks. So I can work with someone who can help me out with the Backend for this task :)

owocki commented 6 years ago

sounds ok to me... up to the group tho

michiang commented 6 years ago

If the front end group is already established, I don't mind working on the data pulls.

tra38 commented 6 years ago

So here's how I plan on organizing this project:

1) Fork the repository and create a skeleton Django app. Provide a link to my fork in this issue.

2) While I work on the backend and the "email generation code", I'll also create issues representing all the different tasks necessary for the success of the project (such as the front-end work). This way, people can step in and out of the project as necessary.

3) People can lay claim to those tasks and submit Pull Requests that I will then review and merge into the repo.

4) We'll keep going until everything is ready to be sent to @owocki for final approval and merging into the main skunkworks repo.

tra38 commented 6 years ago

Additional Notes:

1) I plan on starting work on this project Friday evening, so I can spend all weekend focused on this project. 2) I also want to set up a staging site as well (deployed on Heroku) to make manual testing of the site easier (as we are adding more and more features). We can always migrate it over to a different hosting server when it's ready for production.

eswarasai commented 6 years ago

@tra38 -- The plan seems pretty solid to me except for the part of again creating different tasks in the forked repo and submitting PRs. I'd recommend dividing Frontend and Backend work accordingly between us so that we can deliver them independent to each other's task and can integrate them down the line. Let me know your thoughts on this. Thanks.

owocki commented 6 years ago

Fork the repository and create a skeleton Django app.

Note: I'm not married to this living in 'skunkworks'. If you want, I can create a new repo for it and give you write access to it. Interested?

stojce commented 6 years ago

Anybody started working on this? I don't see anybody that made a claim to this bounty.

tra38 commented 6 years ago

@eswarasai - Sounds like a better idea. Here's how I imagine the work being divided: @tra38 focus on all the "backend" tasks and the text generation (which will probably be done on the client-side using JS rather than server-side). @eswarasai can handle all the Front-End tasks, including incorporating the text generation script. @michiang and @stojce can follow the project as appropriate and step in when necessary.

@owocki - Yeah, that sounds reasonable. I'll be happy with putting the code in a new repo rather than forking it.

@stojce - I can't really claim it because it'd be a collaborative project, so it doesn't really make sense for a single person to claim credit for the project. But I might claim the bounty anyway if that'll make everything easier. Your thoughts, @owocki?

owocki commented 6 years ago

we are working on building in collaborative bounties into the system soon :)

for now... just claim it and i willi figure out how to do the payouts down the line!

tra38 commented 6 years ago

@owocki - Seems like my balance is not sufficient to make a claim. The suggested gas price by Metamask is 0.004 ETH, and I only have ~0.0002 ETH. Any suggestions on how to handle that?

owocki commented 6 years ago

@tra38 send me your ETH address over slack and ill send you some ETH

also... gas prices have come down in the last 24hours.. might be worth trying again?

eswarasai commented 6 years ago

@owocki -- While we wait for @tra38 to claim this issue, in the meanwhile can you create a repo for the coldoutreach.io and invite both of us as collaborators so that we can get started with the initial setup? Thanks.

tra38 commented 6 years ago

Got some ETH from another source and attempted a claim today, but encountered this error (duplicated twice):

This issue is no longer active. Please leave a comment here if you need help.

According to that issue, this should probably occur only when the issue is impossible to be claimed (probably because somebody else has claimed it). But the issue seems like it's unclaimed, at least according to the Issue Explorer...

gitcoinbot commented 6 years ago

The funding of 0.25 ETH (326.13 USD) attached has been claimed by @eswarasai.

@eswarasai, please leave a comment to let the funder (@owocki) and the other parties involved your implementation plan. If you don't leave a comment, the funder may expire your claim at their discretion.

owocki commented 6 years ago

looks like the issue was claimed by @eswarasai but was not processed by the web app (possibly due to https://github.com/gitcoinco/web/issues/128 ? )

@eswarasai lets keep your claim on the issue for now.. when its time to payout... ill pay it out to both of ya

owocki commented 6 years ago

@tra38 @eswarasai https://github.com/gitcoinco/coldoutreach/blob/master/README.md

eswarasai commented 6 years ago

@owocki -- That's weird. I thought my claim didn't go through because of the low gas price as my browser tab was open all along. I might have refreshed the tab though.

owocki commented 6 years ago

okie dokie... well i guess let @tra38 if you want him to take the lead on the bounty or if you want to be involved in coding!

tra38 commented 6 years ago

Thanks, @owocki. I'm trying to push up a basic scaffolding of the Python app that me and @eswarasai can work on, but it seems that I don't have permissions to do so.

remote: Permission to gitcoinco/coldoutreach.git denied to tra38. fatal: unable to access 'https://github.com/gitcoinco/coldoutreach.git/': The requested URL returned error: 403

owocki commented 6 years ago

just invited you both to a team that has write permissions to that repo

owocki commented 6 years ago

hows it going folks?

tra38 commented 6 years ago

We're kinda roadblocked at the moment because, officially, we need access to certain private LinkedIn APIs so that we can gain access to the full profiles of candidates/users. I already filled out an application to LinkedIn requesting for permission to those APIs. If we don't get those permissions, we'll need to think of alternative ways of getting the data we need for coldoutreach.io.

LinkedIn should reply to us within 14 days.

tra38 commented 6 years ago

LinkedIn replied back and has denied our application. Currently exploring alternative routes...

tra38 commented 6 years ago

It turns out that the alternative route that we may take is to request users upload "resumes" instead of providing links to LinkedIn profiles. It's very easy to convert a LinkedIn profile into a resume (if you're logged in, you can visit a person's profile page and press on the three dots to get the chance to convert the profile to a PDF).

screen shot 2018-01-21 at 7 26 16 pm

This isn't an ideal solution because it requires more effort on the part of the user (and the profile PDF only contain the title, summary, experience, and education). However, it does make the system more flexible (we could parse arbitrary resumes, not just profiles from LinkedIn) and maintainable (we don't need to get into a "scraping" arms race with LinkedIn).

However, I'm open to a better alternative route that will allow us to match the original specs where users simply get information by providing the URL of LinkedIn profiles. Below is the technical reason why we're not able to use links to LinkedIn profiles.

As mentioned before, we were denied access to certain private LinkedIn APIs. So this weekend, I and @eswarasai worked on seeing whether we can "scrape" the LinkedIn website on-demand, thereby allowing us to get users' profiles on-demand. However, LinkedIn is very aggressive in blocking scrapers, and will constantly send 999 error messages to any IP it thinks is scraping. Existing open-source scraping libraries are not sufficient, partly due to LinkedIn's IP blocking rendering the scrapers useless but also due to LinkedIn regularly changing its HTML markup (forcing open source maintainers to play catch-up).

The only way that I was able to bypass LinkedIn's defenses is to build a scraper that contains my own "cookie" (specifically, a cookie with an li_at token) when I am logged into LinkedIn. But it's clear that this approach is fragile: not only could LinkedIn modify its codebase to detect and block accounts that are abusing the li_at token, but LinkedIn could easily block the IP address of the central server.

To address the former requires creating multiple accounts that are logged in, so if one li_at token is banned, we can switch it out with another token. Seems simple.

To address the latter requires a client-side solution so that we could use users' IP addresses to scrape the LinkedIn site, so LinkedIn would have a harder time trying to block us (even if they block one computer, other computers could still function)....but we were unable to find a solution that would allow us to embed an "automated-web-browser"-within-a-web-browser. For example, I tried to convert the X-Ray server-side javascript library to a client-side javascript library using Browserify, but it turns out that X-Ray is reliant on "_http_common", a package that only available in server-side Node.js (and thus cannot be exported to client-side JavaScript). And it seems that we do need that automated-web-browser, since the only way I know of to bypass LinkedIn's defenses is by providing them with a cookie with a li_at token.

Ultimately, we are putting in a lot of time and effort to try to do something that might not be possible. Even if it is possible, it's just a workaround that LinkedIn's engineers could probably destroy in 6 months if they really wanted to.

Now, again, we could be wrong. Maybe there's an effective, efficient way of scraping LinkedIn profiles out there. If you can help us out by providing a solution to this scraping issue, we would greatly appreciate it. But in the meantime, in order to get an MVP out there and continually iterate and improve on the idea, we will pivot over to just requesting resumes.

As a side-note, LinkedIn's policies on scraping public profiles is currently being challenged in US court, so it's possible that the legal climate may become more favorable for us in the future.

owocki commented 6 years ago

LinkedIn replied back and has denied our application

did they say 'why'?

It turns out that the alternative route that we may take is to request users upload "resumes" instead of providing links to LinkedIn profiles.

I agree that this is suboptimal from a UX perspective because of the friction it induces..

HTML Scraping

First off... seem like you really did your homework here.. Kudos

Why do we need a full client side scraping solution? Don't we just need to do a GET request (with the li_at cookie for the users main profile page ? a la linkedin.com/in/owocki/

tra38 commented 6 years ago

did they say 'why'?

No. According to their email: "Due to the high volume of requests we receive, we unfortunately cannot provide detailed feedback about your individual application. If you have further questions about the use cases we support through our Open APIs, please review our developer site (https://developer.linkedin.com/) and API Terms of Use (https://developer.linkedin.com/legal/api-terms-of-use)."

My main guess is that our request violated Section 1.4, bullet point 4 of the API Terms of Use. "[Y]our Application WILL NOT store or export any data from LinkedIn other than the LinkedIn Profile Data for the LinkedIn member that requested the data." Since our application need the LinkedIn Profile data of the candidate (who has not specifically requested for us to use their data), that's probably why LinkedIn said no.

Don't we just need to do a GET request (with the li_at cookie for the users main profile page ? a la linkedin.com/in/owocki/

I don't think it is possible to send a simple GET request along with a cookie (my assumption had been that it requires a browser to visit the page directly), but let me see if I can try that out.

I agree that this is suboptimal from a UX perspective because of the friction it induces..

I agree too. Hopefully, in the future, we can find a better approach.

owocki commented 6 years ago

i think it depends on the cross origin settings of the domain https://stackoverflow.com/questions/2870371/why-is-jquerys-ajax-method-not-sending-my-session-cookie

tra38 commented 6 years ago

I think so too. When I tried sending a GET Ajax request with a cookie containing the li_at token, I got a new error message: "preflight-response-is-not-successful". This error message occurs only if the server doesn't send the correct headers authorizing the GET Ajax request with cookies. Which suggests that the only way we can successfully make a GET request with the li_at token is if we get LinkedIn's approval first. Otherwise, they will simply reject any Ajax requests that attempts to send a cookie.

Source: https://stackoverflow.com/questions/8685678/cors-how-do-preflight-an-httprequest

Pretty neat trick though (sending cookies through Ajax requests). Always learning something new every day... :+1:

eswarasai commented 6 years ago

@owocki -- For editing the generated email, do we need a Rich Text Editor similar to the one below?https://leejaen.github.io/react-lz-editor/index.html

I can see that in the Design, the editor doesn't have any options, yet there are some specific styles being applied to the text. Let me know if you'd like me to integrate with the above one.

owocki commented 6 years ago

i think that for the MVP, we don't need a rich text editor

vs77bb commented 6 years ago

@eswarasai @owocki @tra38 Wanted to check in here... is this still being worked?

eswarasai commented 6 years ago

@vs77bb -- I'm at the verge of wrapping these changes. Will make the changes live over the weekend

owocki commented 6 years ago

hi all. i just killed the bounty for this issue for some gitcoin-internal migration reasons but wanted to let you know that, regardless of the issue description on gitcoin.co.. im good to pay out this bounty when the time is right. just @ me back then!

gitcoinbot commented 6 years ago

⚡️ A tip worth 0.125 ETH (67.23 USD @ $537.83/ETH) has been granted to @eswarasai for this issue from Kevin. ⚡️

The sender had the following public comments:

Thx :)

Nice work @eswarasai! To redeem your tip, login to Gitcoin at https://gitcoin.co/explorer and select 'Claim Tip' from dropdown menu in the top right, or check your email for a link to the tip redemption page.

gitcoinbot commented 6 years ago

⚡️ A tip worth 0.125 ETH (67.23 USD @ $537.83/ETH) has been granted to @thelostone-mc for this issue from Kevin. ⚡️

The sender had the following public comments:

Thx :)

Nice work @thelostone-mc! To redeem your tip, login to Gitcoin at https://gitcoin.co/explorer and select 'Claim Tip' from dropdown menu in the top right, or check your email for a link to the tip redemption page.

gitcoinbot commented 6 years ago

⚡️ A tip worth 0.25 ETH (131.85 USD @ $527.42/ETH) has been granted to @tra38 for this issue from Kevin. ⚡️

The sender had the following public comments:

Thx :)

Nice work @tra38! To redeem your tip, login to Gitcoin at https://gitcoin.co/explorer and select 'Claim Tip' from dropdown menu in the top right, or check your email for a link to the tip redemption page.