Document minimal setup for local testing of web apps

[jimallman]

This could be a wiki page with simple tricks for different operating systems (starting with my own setup on OS X).

[jimallman]

Oops, looks like we've already created this page, it was just a little stale. I updated it to reflect my latest setup and helper scripts for secure connections. @kcranston is testing these instructions; we'll close this issue once everything is working as expected.

[jimallman]

Needs more information on:

• symlinking our web apps into the web2py/applications/ directory
• tweaks to web2py routing? (to set default app)

[kcranston]

The symlink instructions are in the readme. We should homogenize the readme and the wiki page (or, better yet, put all of the instructions in one place).

[jimallman]

Other quirks that are handled by the deployment tools (should we be using these somehow?):

• we have a few patched web2py files to install
• we probably want to specify a particular version of web2py

[jar398]

It would be really nice to be able to use bits and pieces of the deployment tools. The scripts ought to be made more modular than they are, with fewer dependencies. Then they could be used locally.

On Wed, Jul 15, 2015 at 11:25 AM, Jim Allman notifications@github.com wrote:

Other quirks that are handled by the deployment tools (should we be using these somehow?):

• we have a few patched web2py files to install
• we probably want to specify a particular version of web2py

— Reply to this email directly or view it on GitHub https://github.com/OpenTreeOfLife/opentree/issues/676#issuecomment-121651363 .

[jar398]

Aren't the host names parameters in some config file somewhere? Better to change them there to be localhost. Modifying /etc/hosts is an awful burden, is very confusing, and shouldn't be needed.

On Wed, Jul 15, 2015 at 12:04 PM, Jonathan A Rees rees@mumble.net wrote:

It would be really nice to be able to use bits and pieces of the deployment tools. The scripts ought to be made more modular than they are, with fewer dependencies. Then they could be used locally.

On Wed, Jul 15, 2015 at 11:25 AM, Jim Allman notifications@github.com wrote:

Other quirks that are handled by the deployment tools (should we be using these somehow?):

• we have a few patched web2py files to install
• we probably want to specify a particular version of web2py

— Reply to this email directly or view it on GitHub https://github.com/OpenTreeOfLife/opentree/issues/676#issuecomment-121651363 .

[jimallman]

Aren't the host names parameters in some config file somewhere?

I don't think so, actually. Because we rely on hostnames for registering apps with GitHub, etc., some features will break if we choose localhost or other names.

[jar398]

I don't get this. How does github even know that we've changed our /etc/hosts file, and why should it care?

On Wed, Jul 15, 2015 at 12:14 PM, Jim Allman notifications@github.com wrote:

Aren't the host names parameters in some config file somewhere?

I don't think so, actually. Because we rely on hostnames for registering apps with GitHub, etc., some features will break if we choose localhost or other names.

— Reply to this email directly or view it on GitHub https://github.com/OpenTreeOfLife/opentree/issues/676#issuecomment-121665631 .

[kcranston]

Let's modify config.example so that there is a single base URL in the [domains] section used to build variable treemachine, taxomachine, oti, opentree_api, CACHED_treemachine and CACHEDtaxomachine. As it stands, too much typing (and the current instructions miss the part about modifying the CACHED*) URLs.

[jimallman]

How does github even know that we've changed our /etc/hosts file, and why should it care?

Sorry if I'm being unclear. Authentication and other API calls are between the browser and remote site, but they rely on domain boundaries (and known-good domain names) for security. GitHub and others will reject calls from localhost if the relationship (for example, a registered app on GitHub) requires the proper domain name.

[jimallman]

Let's modify config.example so that there is a single base URL in the [domains] section used to build variable treemachine, taxomachine, oti, opentree_api, CACHED_treemachine and CACHED_taxomachine.

The current setup allows for distributed services on n servers, as requested. I suppose we could allow for substitution of "upstream" values, as we would in a shell script, but it's not really saving keystrokes IMO:

[domains]
common = //devapi.opentreeoflife.org
treemachine = {common}/treemachine/v1
taxomachine = {common}/taxomachine/v1
oti = {common}/oti/v1
opentree_api = {common}/phylesystem/v1
collections_api_base_url = {common}/v2
favorites_api_base_url = {common}/v2
# Cached versions of some APIs will speed up repeated calls (see below).
# These use a simple web2py cache implemented in phylesystem-api
CACHED_treemachine = {common}/cached/treemachine/v1
CACHED_taxomachine = {common}/cached/taxomachine/v1

[jar398]

How does github know who the call is from? If it can be spoofed it's certainly not secure. Sorry, I'm just completely forgetting how this works. Is it a CORS thing? a Host: header thing? ...

On Wed, Jul 15, 2015 at 12:45 PM, Jim Allman notifications@github.com wrote:

How does github even know that we've changed our /etc/hosts file, and why should it care?

Sorry if I'm being unclear. Authentication and other API calls are between the browser and remote site, but they rely on domain boundaries (and known-good domain names) for security. GitHub and others will reject calls from localhost if the relationship (for example, a registered app on GitHub) requires the proper domain name.

— Reply to this email directly or view it on GitHub https://github.com/OpenTreeOfLife/opentree/issues/676#issuecomment-121675159 .

[jimallman]

I've updated the section 'Create local webapp config files' to include the CACHED_* entries in the [domains] section of config.

[jimallman]

How does github know who the call is from? If it can be spoofed it's certainly not secure.

In the case of OAuth, the domain name is part of a redirect URL back to (in our case) the web app's login page. This is specified in the app config:

github_redirect_uri = http://devtree.opentreeoflife.org/opentree/user/login

This must (exactly) match the URL stored on GitHub for a registered app. So it assumes that the user's browser, at least, will recognize this URL and do the right thing when redirected by the OAuth server.

[jar398]

Got it, thanks. Need to capture this in the wiki page to remind forgetful minds like mine.

On Wed, Jul 15, 2015 at 3:26 PM, Jim Allman notifications@github.com wrote:

How does github know who the call is from? If it can be spoofed it's certainly not secure.

In the case of OAuth, the domain name is part of a redirect URL back to (in our case) the web app's login page. This is specified in the app config:

github_redirect_uri = http://devtree.opentreeoflife.org/opentree/user/login

This must (exactly) match the URL stored on GitHub for a registered app. So it assumes that the user's browser, at least, will recognize this URL and do the right thing when redirected by the OAuth server.

— Reply to this email directly or view it on GitHub https://github.com/OpenTreeOfLife/opentree/issues/676#issuecomment-121720699 .

[kcranston]

I think we can close this issue now. I've been able to create a working local install using these instructions.

[kcranston]

Noting that I've added the explanation about why we need to modify /etc/hosts as requested by @jar398

[jar398]

Excellent!​

[jimallman]

Modifying /etc/hosts is an awful burden, is very confusing, and shouldn't be needed.

At @jar398's suggestion, I've looked into alternatives to fiddling with /etc/hosts. To my surprise, I found a fairly elegant solution in making an Automatic Proxy Configuration (PAC) file file at this URL:

https://gist.githubusercontent.com/jimallman/408dd36f6356440dea5a/raw/5d8c46d51d3dd47f64fef931f5101ed71c67d082/local-devtree-proxy.pac

This file (the URL, actually) can be used to configure domain-specific proxy'ing on all modern operating systems. I've had some success, but very sporadic, so I'm still tinkering with this. Will add more information here once it's working consistently.

Update: I'm having trouble debugging this, and getting consistent results across HTTP and HTTPS. At this point, it doesn't seem any easier than /etc/hosts manipulation, and even more obscure and poorly documented.

[jimallman]

@kcranston, @jar398: OK, I figured out how to debug and get alert messages in Chrome, so this version of the PAC-file (gist) is working consistently for me: https://gist.githubusercontent.com/jimallman/408dd36f6356440dea5a/raw/406f080530e4f73dede552c158067aeb15bc614e/local-devtree-proxy.pac

This script should cause the proxy (test server on localhost) to be used for any URL whose host is devtree.opentreeoflife.org. Simple! There's some diagnostic chatter there in case we get unexpected results, but it's disabled for now.

Applying this PAC file requires different steps, depending on your OS. For OS X, open the Network preferences and choose the active network service (Wi-Fi, Ethernet, whatever), then Advanced... In the advanced settings, choose Proxies, then check Automatic Proxy Configuration and enter the URL above. Click OK to save, then Apply. Your browser (and other apps) should now look to localhost for devtree.opentreeoflife.org, just as if you had modified /etc/hosts.

Is it easier to manage, or less prone to cause confusion? I'm not so sure. Thoughts?

[kcranston]

In the post you referenced earlier: http://tektab.com/2012/09/26/setting-up-automatic-proxy-configuration-pac-file/

It describes setting this on a per-browser basis rather than in general Network settings. Would that not let you test localhost in one browser while allowing other browsers to navigate to devtree?

On Fri, Jul 17, 2015 at 12:06 PM, Jim Allman notifications@github.com wrote:

Reopened #676 https://github.com/OpenTreeOfLife/opentree/issues/676.

— Reply to this email directly or view it on GitHub https://github.com/OpenTreeOfLife/opentree/issues/676#event-358654570.

karen.cranston@gmail.com
@kcranstn

[jimallman]

It describes setting this on a per-browser basis rather than in general Network settings.

I think I see what you're going for: a dedicated "test browser" that doesn't interfere with normal operation elsewhere. Unfortunately, while most browsers offer proxy settings (e.g., Chrome > Preferences > Show advanced settings > Network), these actually defer to (or update) system-wide settings.

I gather Firefox can be more isolated, and it looks like at least one browser extension (FoxyProxy) will use the PAC file above, but just within Firefox.

[jar398]

Well that's unfortunate.

I bet lynx doesn't look at those settings. :) I wonder if you get them in private browsing mode. Probably...

Jonathan

On Fri, Jul 17, 2015 at 1:17 PM, Jim Allman notifications@github.com wrote:

It describes setting this on a per-browser basis rather than in general Network settings.

I think I see what you're going for: a dedicated "test browser" that doesn't interfere with normal operation elsewhere. Unfortunately, while most browsers offer proxy settings (e.g., Chrome > Preferences > Show advanced settings > Network), these actually defer to (or update) system-wide settings.

I gather Firefox can be more isolated, and it looks like at least one browser extension (FoxyProxy http://getfoxyproxy.org/) will use the PAC file above, but just within Firefox http://getfoxyproxy.org/mozilla/faq.html#pac.

— Reply to this email directly or view it on GitHub https://github.com/OpenTreeOfLife/opentree/issues/676#issuecomment-122346602 .

[kcranston]

Can we move the discussion of /etc/hosts to a new issue and close this one? The minimal documentation is now complete.