27crags import

[sanderpick]

Got at least 5 requests so far for 27crags import.

[eyalcohen]

I have this working although it still needs some testing.

Before it goes live, we should scrape 27crags for their crag list. Otherwise many ticks will get lost.

[sanderpick]

Makes sense. Poked around some. They don't have a master list, but you can get crags around some point location:

https://27crags.com/areas/39.94,-105.01

One idea: we just sweep across a grid in x,y - snagging the list at each point and removing duplicates

0,0 0,0.1 0,0.2 ...

While we're at it, could snag the ascents too... ? or maybe enough to let those come in with the scorecards?

[eyalcohen]

I've got the crag scraping portion done. We can get their full list with crag, latitude and longitude. It works by scraping thier crag list page and parsing the Google Maps URL. It takes about 5 seconds per crag page and there are 400 crag pages so I haven't gotten the full list yet.

One problem I haven't figured out a solution yet: Their crags don't line up well with ours. For example, we have Bishop as a crag, whereas they have Buttermilks as a crag.

We could also scrape for ascents, but the import tool will automatically create ascents it doesn't know about as long as it can establish a crag to connect them with.

[sanderpick]

Oh, nice one!

Could lump together nearby crags? Hmmm, that's tough. I guess better to error on the side of duplicating since we'll need to pretty soon dedicate some muscle to cleaning the DB anyway.

+1 for letting ascents come through w/ import tool

[eyalcohen]

Ok, so I have the full list of crags from 27crags, but I'm not crazy about adding them without a major scrubbing effort. There's a lot of weird ones, with weird names and a lot of duplicates due to slight unicode differences (accents etc)

Scrubbing also makes importing much more difficult, since we won't have an easy way to correlate ascents coming from 27crags (new prop for the crag collection, alternateNames?)

[sanderpick]

yeah, I see what you mean. damn. maybe this is where we start looking for some manual labor :)

On Friday, June 19, 2015 at 12:13 AM, Eyal Cohen wrote:

Ok, so I have the full list of crags from 27crags, but I'm not crazy about adding them without a major scrubbing effort. There's a lot of weird ones, with weird names and a lot of duplicates due to slight unicode differences (accents etc) Scrubbing also makes importing much more difficult, since we won't have an easy way to correlate ascents coming from 27crags (new prop for the crag collection, alternateNames?)

— Reply to this email directly or view it on GitHub (https://github.com/The-Island/island-app/issues/112#issuecomment-113386958).

[eyalcohen]

I have a script that will load all the missing crags from 27crags. Can you push to prod a small tweak to the crag.js resource for me? Its in develop.

[sanderpick]

pushed to prod

[sanderpick]

I'll do a blog post, tweet, etc.

[eyalcohen]

hang tight on the blog post until I get the crags uploaded. Its tricky cuz we're rate limited on geolocating and there are 1000s to do. I'll do it tonight.

On Tue, Jul 7, 2015 at 11:04 AM Sander Pick notifications@github.com wrote:

I'll do a blog post, tweet, etc.

— Reply to this email directly or view it on GitHub https://github.com/The-Island/island-app/issues/112#issuecomment-119286419 .

[eyalcohen]

hmm, can't hit the crags API on island.io for some reason. Any idea why? POST to http://www.island.io/api/crags returns the main html POST to http://island.io/api/crags returns a 405 method not allowed local works fine

[sanderpick]

should work with https

curl -X POST https://www.island.io/api/crags

[eyalcohen]

I get this using request.js on https://www.island.io/api/crags

Posting Lietlahti Park 13 left
[Error: UNABLE_TO_VERIFY_LEAF_SIGNATURE]
error: [Error: UNABLE_TO_VERIFY_LEAF_SIGNATURE]

http://stackoverflow.com/questions/20082893/unable-to-verify-leaf-signature says its a problem with the intermediate cert, although they offer a workaround to disable security

[eyalcohen]

yup, workaround works

[sanderpick]

sweet. yep we'd do that at cartodb -

process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';

frowned upon but works :)

[sanderpick]

Guess we should roll this back cause of their GIVE ME MONEY page eh?

[eyalcohen]

Should be fixed, prod needs to pull the latest lib27crags On Sat, Jul 11, 2015 at 9:28 AM Sander Pick notifications@github.com wrote:

Guess we should roll this back cause of their GIVE ME MONEY page eh?

— Reply to this email directly or view it on GitHub https://github.com/The-Island/island-app/issues/112#issuecomment-120637487 .

[sanderpick]

Kick ass. Got it live now. I find that two name searches throw an error while single names work like a charm... e.g., "nalle hukkataival" vs. "nalle"

[eyalcohen]

Hmm used to work, will look at it when I'm back in SF. Maybe time for a regression suite eh? On Sat, Jul 11, 2015 at 10:26 AM Sander Pick notifications@github.com wrote:

Kick ass. Got it live now. I find that two name searches throw an error while single names work like a charm... e.g., "nalle hukkataival" vs. "nalle"

— Reply to this email directly or view it on GitHub https://github.com/The-Island/island-app/issues/112#issuecomment-120646512 .

[sanderpick]

yep yep, sounds good. tests and travis.yml from the island-app would be good start… i see you have a couple tests on the crags lib already. ill sub an issue

On Saturday, July 11, 2015 at 10:43 AM, Eyal Cohen wrote:

Hmm used to work, will look at it when I'm back in SF. Maybe time for a regression suite eh? On Sat, Jul 11, 2015 at 10:26 AM Sander Pick <notifications@github.com (mailto:notifications@github.com)> wrote:

Kick ass. Got it live now. I find that two name searches throw an error while single names work like a charm... e.g., "nalle hukkataival" vs. "nalle"

— Reply to this email directly or view it on GitHub https://github.com/The-Island/island-app/issues/112#issuecomment-120646512 .

— Reply to this email directly or view it on GitHub (https://github.com/The-Island/island-app/issues/112#issuecomment-120647743).