linkedin batch scraping

RSV3 / redfly

contact intelligence for Redstar

1 stars 0 forks source link

linkedin batch scraping #264

Open justinTNT opened 11 years ago

justinTNT commented 11 years ago

there's a bunch of profile details that the linkedin api doesnt give and attempts to keep track of changes since last call can be confounded by OOB visits to the linkedin webpage if we want to pick up changes, and get that extra data, we might like to do batch scraping of the linkedin profiles already noted for existing contacts (we'll still rely on users selecting the menu option to update with new contacts: or maybe this can also happen periodically during visits to the site?)

pharcosyle commented 11 years ago

Make a batch job? We already have one for the weekly email parsing

kwantopia commented 11 years ago

This is for linkedin parsing. And would be run once every two weeks.

On Wed, Jan 23, 2013 at 12:49 AM, Krzysztof Baranowski < notifications@github.com> wrote:

Make a batch job? We already have one for the weekly email parsing

— Reply to this email directly or view it on GitHubhttps://github.com/RSV3/redfly/issues/264#issuecomment-12582611.

Kwan Hong Lee, Ph.D. Technology Director Redstar Ventures http://www.redstar.com 617-871-0710

justinTNT commented 11 years ago

losing interest, now that:

I realise how much linkedin don't want people to scrape, and
I've found the arcane incacntation that gives us hires photos (yay!)

what we're really missing out on is the list of skills, which would be great for making tags.

pharcosyle commented 11 years ago

This seems time consuming and awful, but if you do do it, use http://zombie.labnotes.org/ if the site markup isn't inscrutable and http://phantomjs.org/ otherwise