make GOLR server / URL configurable

[nathandunn]

• [x] bootstrap proxy work
• [x] make golr proxy config work
• [x] create doc for proxy

[nathandunn]

there are many servers and we may want to move to other servers eventually.

[nathandunn]

from Seth, other public URL's that could be used:

http://golr.geneontology.org/

As a useful balanced point to get the data, but they seemed rather lukewarm about it with all of the other stuff that we're trying to do over there. Until it is sorted out, it may be better to pick one of the golr instances and depend on that (stanford should have better downtime numbers than us in this case).

http://a2-proxy1.stanford.edu:8080/solr/ http://a2-proxy2.stanford.edu:8080/solr/

[nathandunn]

tested and working

[cmdcolin]

I haven't been able to get this to work yet. Not sure why?

Also, I guess this is a template string that is filled in at compile/build time? I wouldn't want our configuration variables to proliferate like this too much.

[nathandunn]

Not all of the URL's that are available work. . . the idea is that it should just be easy to swap them out. That is a default set.

[cmdcolin]

Ya, I think I tried them all though, and couldn't get any to work?

[nathandunn]

This one doesn’t work (right now): http://golr.geneontology.org/ http://golr.geneontology.org/

but the other two should:

http://a2-proxy2.stanford.edu:8080/solr/select?defType=edismax&qt=standard&indent=on&wt=json&rows=10&start=0&fl=*,score&facet=true&facet.mincount=1&facet.sort=count&json.nl=arrarr&facet.limit=25&fq=document_category:%22ontology_class%22&fq=source:(biological_process%20OR%20molecular_function%20OR%20cellular_component)&facet.field=annotation_class&facet.field=synonym&facet.field=alternate_id&q=red*&qf=annotation_class%5E5&qf=annotation_class_label_searchable%5E5&qf=synonym_searchable%5E1&qf=alternate_id%5E1&json.wrf=jQuery17107463070501107723_1422917711324&_=1422917761144 http://a2-proxy2.stanford.edu:8080/solr/select?defType=edismax&qt=standard&indent=on&wt=json&rows=10&start=0&fl=*,score&facet=true&facet.mincount=1&facet.sort=count&json.nl=arrarr&facet.limit=25&fq=document_category:%22ontology_class%22&fq=source:(biological_process%20OR%20molecular_function%20OR%20cellular_component)&facet.field=annotation_class&facet.field=synonym&facet.field=alternate_id&q=red*&qf=annotation_class^5&qf=annotation_class_label_searchable^5&qf=synonym_searchable^1&qf=alternate_id^1&json.wrf=jQuery17107463070501107723_1422917711324&_=1422917761144

http://a2-proxy1.stanford.edu:8080/solr/select?defType=edismax&qt=standard&indent=on&wt=json&rows=10&start=0&fl=*,score&facet=true&facet.mincount=1&facet.sort=count&json.nl=arrarr&facet.limit=25&fq=document_category:%22ontology_class%22&fq=source:(biological_process%20OR%20molecular_function%20OR%20cellular_component)&facet.field=annotation_class&facet.field=synonym&facet.field=alternate_id&q=red*&qf=annotation_class%5E5&qf=annotation_class_label_searchable%5E5&qf=synonym_searchable%5E1&qf=alternate_id%5E1&json.wrf=jQuery17107463070501107723_1422917711324&_=1422917761144 http://a2-proxy1.stanford.edu:8080/solr/select?defType=edismax&qt=standard&indent=on&wt=json&rows=10&start=0&fl=*,score&facet=true&facet.mincount=1&facet.sort=count&json.nl=arrarr&facet.limit=25&fq=document_category:%22ontology_class%22&fq=source:(biological_process%20OR%20molecular_function%20OR%20cellular_component)&facet.field=annotation_class&facet.field=synonym&facet.field=alternate_id&q=red*&qf=annotation_class^5&qf=annotation_class_label_searchable^5&qf=synonym_searchable^1&qf=alternate_id^1&json.wrf=jQuery17107463070501107723_1422917711324&_=1422917761144

as well as the original:

http://golr.berkeleybop.org/select?defType=edismax&qt=standard&indent=on&wt=json&rows=10&start=0&fl=*,score&facet=true&facet.mincount=1&facet.sort=count&json.nl=arrarr&facet.limit=25&fq=document_category:%22ontology_class%22&fq=source:(biological_process%20OR%20molecular_function%20OR%20cellular_component)&facet.field=annotation_class&facet.field=synonym&facet.field=alternate_id&q=red*&qf=annotation_class%5E5&qf=annotation_class_label_searchable%5E5&qf=synonym_searchable%5E1&qf=alternate_id%5E1&json.wrf=jQuery17107463070501107723_1422917711324&_=1422917761144

What error are you getting?

Nathan

On Feb 2, 2015, at 2:54 PM, Colin Diesh notifications@github.com wrote:

Ya, I think I tried them all though, and couldn't get any to work?

— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/148#issuecomment-72556465.

[cmdcolin]

Basically I don't get any auto-completion on GO terms in the Information Editor.

Example: normally i click on the "Enter new Gene Ontology ID", then I hit "z", and then I wait for it to return something like "zinc binding", but this doesn't happen.

I tried using origin/master and checking out 1.0.3 and same results for both

[nathandunn]

I think that you need to type at least 3 letters. I can double-check with Seth, but that appears to work and for datasets this large, that is the way I would typically implement them.

Nathan

On Feb 2, 2015, at 3:25 PM, Colin Diesh notifications@github.com wrote:

Basically I don't get any auto-completion on GO terms in the Information Editor.

Example: normally i click on the "Enter new Gene Ontology ID", then I hit "z", and then I wait for it to return something like "zinc binding", but this doesn't happen.

I tried using origin/master and checking out 1.0.3 and same results for both

— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/148#issuecomment-72560915.

[cmdcolin]

Oh for some reason I thought I could just hit 1 letter. Thanks for clarifying, seems to work after 3

[cmdcolin]

@deepakunni3 reports that it is not properly replacing the GOLR_URL in RC1 and I can't get it to do the replacement either

[nathandunn]

@deepakunni3 Can you let me know how you’re replacing them? i.e. step / by step

Nathan

On Feb 16, 2015, at 12:50 PM, Colin Diesh notifications@github.com wrote:

@deepakunni3 https://github.com/deepakunni3 reports that it is not properly replacing the GOLR_URL in RC1 and I can't get it to do the replacement either

— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/148#issuecomment-74570492.

[deepakunni3]

@nathandunn I was testing out the 1.0.4 RC1 and while adding GO terms to a new annotation, there is no autocompletion. Checked further and saw that GET request to GOLR server has a 404 error and the request url has @GOLR_URL@ in it.

[cmdcolin]

I reverted a commit I did previously and it seems fixed now

[nathandunn]

@cmdcolin Ah . . should @deepakunni3 re-test?

[cmdcolin]

@nathandunn Actually no, it still doesn't seem to work. I don't understand what the deal is.

[deepakunni3]

One observation, when I checkout from master branch, the GO autocompletion works fine. But it seems broken on 1.0.4 RC1.

[cmdcolin]

@deepakunni3 Are you sure about this? I am testing this off master and I cannot get it to work.

I propose a fix like this

diff --git a/build.xml b/build.xml
index e257fe8..460a5b5 100644
--- a/build.xml
+++ b/build.xml
@@ -225,10 +225,10 @@
         </copy>
     </target>

-    <target name="copy.apollo.plugin.webapp" unless="jbrowse.precompiled" if="jbrowse.webapp.installed"
+    <target name="copy.apollo.plugin.webapp" unless="jbrowse.precompiled" if="jbrowse.git.present"
             description="Copy client into webapp plugin directory.">
         <echo>copying plugin to webapp ...</echo>
-        <copy todir="${apollo.jbrowse.directory}/plugins/WebApollo" failonerror="true">
+        <copy todir="${jbrowse.download.directory}/plugins/WebApollo" failonerror="true">
             <fileset dir="${apollo.client.directory}"/>
             <filterset>
                 <!--<filter token="GOLR_URL" value="http://golr.berkeleybop.org/"/>-->

Note that we are copying into the jbrowse.download.directory now (ie the git repo instead of the src/main/webapp/jbrowse).

[nathandunn]

@deepakunni3 @cmdcolin

I think that the issue is that it won't recopy unless A) the file isn't there or B) it hasn't been updated.

I am going to test a bit more. But if you have the RC1, @deepakunni3 you should confirm that it works in case B (touch client/apollo/js/View/Track/AnnotTrack.js and apollo deploy) and case A (./apollo clean-all ./apollo deploy). Using deploy or release should work the same, so whatever you'd been using should be fine.

[nathandunn]

Since there are changes between RC1 and 1.0.3 the update should be fine. I will continue testing against master on my end, though. I had reverted back to the 1.0.3 change for now as I think its the most robust depending on how the build is done and it seems to be working.

[nathandunn]

So the problem here is that now it copies into jbrowse-download from the client. However it does not copy from jbrowse-download into src/main/webapp . . . , so if changes are made to the client they don't get propagated properly. . . . ./apollo run and ./apollo deploy should be able to pick up any changes.

I'm going to mess with this a bit.

[deepakunni3]

@nathandunn I tried with 1.0.4 RC1 and the GOLR_URL works for case B but not case A.

[nathandunn]

@deepakunni3 @cmdcolin As awesome an idea as this would be, configuring over the build for a system that is intermittently down just isn't making sense.

It should A) be in the config, not in the code (even the build code).
B) It should be somewhat configurable at runtime.
C) not be all that complicated.

I'm going to leave the other servers in the comments in the AnnotTrack.js so that folks can recompile as necessary.

A proxy (your other idea Colin) is a great idea as it solves A & C, however I'm a bit worried it introducing speed issues. I'm curious if we couldn't do a local GOLR lookup.

A third possibility would be to have a dedicated proxy server that handles backups. Probably not feasible in the short-term.

Anyway, I'm moving this into long-term.

Sorry for the additional work / hassle.

[cmdcolin]

Agree.

[cmdcolin]

Technically it looks like we use the apollo server to proxy to ncbi entrez

NcbiProxyServiceController

This is used in similar contexts to the GOLR (information editor)

[nathandunn]

Right. This could / should be done in an identical fashion.

Nathan

On Oct 16, 2015, at 11:59 AM, Colin Diesh notifications@github.com wrote:

Technically it looks like we use the apollo server to proxy to ncbi entrez

NcbiProxyServiceController

This is used in similar contexts to the GOLR (information editor)

— Reply to this email directly or view it on GitHub.

[nathandunn]

Are you wanting to do this one instead of the faceted track list? I think this will be important for https so might be worth spending a couple of hours fixing.

Nathan

On Oct 16, 2015, at 11:59 AM, Colin Diesh notifications@github.com wrote:

Technically it looks like we use the apollo server to proxy to ncbi entrez

NcbiProxyServiceController

This is used in similar contexts to the GOLR (information editor)

— Reply to this email directly or view it on GitHub.

[nathandunn]

On the branch golr_config

Getting this issue:

Failure:  Test the save action correctly persists an instance(org.bbop.apollo.ProxyControllerSpec)
|  grails.validation.ValidationException: Validation error occurred during call to save():
- Field error in object 'org.bbop.apollo.Proxy' on field 'referenceUrl': rejected value [null]; codes [org.bbop.apollo.Proxy.referenceUrl.nullable.error.org.bbop.apollo.Proxy.referenceUrl,org.bbop.apollo.Proxy.referenceUrl.nullable.error.referenceUrl,org.bbop.apollo.Proxy.referenceUrl.nullable.error.java.lang.String,org.bbop.apollo.Proxy.referenceUrl.nullable.error,proxy.referenceUrl.nullable.error.org.bbop.apollo.Proxy.referenceUrl,proxy.referenceUrl.nullable.error.referenceUrl,proxy.referenceUrl.nullable.error.java.lang.String,proxy.referenceUrl.nullable.error,org.bbop.apollo.Proxy.referenceUrl.nullable.org.bbop.apollo.Proxy.referenceUrl,org.bbop.apollo.Proxy.referenceUrl.nullable.referenceUrl,org.bbop.apollo.Proxy.referenceUrl.nullable.java.lang.String,org.bbop.apollo.Proxy.referenceUrl.nullable,proxy.referenceUrl.nullable.org.bbop.apollo.Proxy.referenceUrl,proxy.referenceUrl.nullable.referenceUrl,proxy.referenceUrl.nullable.java.lang.String,proxy.referenceUrl.nullable,nullable.org.bbop.apollo.Proxy.referenceUrl,nullable.referenceUrl,nullable.java.lang.String,nullable]; arguments [referenceUrl,class org.bbop.apollo.Proxy]; default message [Property [{0}] of class [{1}] cannot be null]
- Field error in object 'org.bbop.apollo.Proxy' on field 'targetUrl': rejected value [null]; codes [org.bbop.apollo.Proxy.targetUrl.nullable.error.org.bbop.apollo.Proxy.targetUrl,org.bbop.apollo.Proxy.targetUrl.nullable.error.targetUrl,org.bbop.apollo.Proxy.targetUrl.nullable.error.java.lang.String,org.bbop.apollo.Proxy.targetUrl.nullable.error,proxy.targetUrl.nullable.error.org.bbop.apollo.Proxy.targetUrl,proxy.targetUrl.nullable.error.targetUrl,proxy.targetUrl.nullable.error.java.lang.String,proxy.targetUrl.nullable.error,org.bbop.apollo.Proxy.targetUrl.nullable.org.bbop.apollo.Proxy.targetUrl,org.bbop.apollo.Proxy.targetUrl.nullable.targetUrl,org.bbop.apollo.Proxy.targetUrl.nullable.java.lang.String,org.bbop.apollo.Proxy.targetUrl.nullable,proxy.targetUrl.nullable.org.bbop.apollo.Proxy.targetUrl,proxy.targetUrl.nullable.targetUrl,proxy.targetUrl.nullable.java.lang.String,proxy.targetUrl.nullable,nullable.org.bbop.apollo.Proxy.targetUrl,nullable.targetUrl,nullable.java.lang.String,nullable]; arguments [targetUrl,class org.bbop.apollo.Proxy]; default message [Property [{0}] of class [{1}] cannot be null]
    at org.grails.datastore.gorm.GormInstanceApi.doSave(GormInstanceApi.groovy:182)
    at org.grails.datastore.gorm.GormInstanceApi.save_closure5(GormInstanceApi.groovy:162)

[cmdcolin]

Note that we do not want to support "arbitrary proxying"

i.e. we don't want something that can say "http://apolloserver.com/apollo/proxy/?proxyurl=http://google.com" to forward a request to google.

That type of setup allows hackers to bounce malicious requests through your server

I was surprised to see code that looks like this here https://github.com/GMOD/Apollo/commit/c39837ae3b0d6bc0bad23d21e636995de7dd8c2b

Instead the server side to allow a limited number of URL options, as the NcbiProxyController was doing originally

[cmdcolin]

Do appreciate the work on this though :) I just think it should come from simple server side config variables than allowing the client to do it

[nathandunn]

You're right . . an unregistered proxy should fail. I think I need to bootstrap the proper ones and go from there.

http://stackoverflow.com/questions/16904414/code-to-get-a-java-servlet-to-act-as-a-proxy https://github.com/mitre/HTTP-Proxy-Servlet

[nathandunn]

Also the seed for #606.

[nathandunn]

To test:

• confirm that GOLR lookup (GO type-ahead evidence) still works as you would expect

[monicacecilia]

I confirm that GOLR lookup is working as expected. However, see what happens when adding new GO terms at #627

[mpoelchau]

Sorry to chime in late on this conversation - but is the http://golr.geneontology.org/solr/ URL deprecated now? It's not working on our WA1 instances, nor does it appear to be on http://icebox.lbl.gov/Apollo2/annotator/index. Looks like it is on your staging site, but that's where the proxy is set up. Any advice would be appreciated!

[nathandunn]

You are correct.

In 2.0.1 it automatically uses a database configurable proxy (with interface!) and has the proper default set.

If you are using 1.0 you need to switch it to this in AnnoTrack.js:

http://golr.berkeleybop.org/solr/

I’ll make an announcement for 2.0.0 and show how to change it.

Nathan

On Nov 19, 2015, at 7:45 AM, mpoelchau notifications@github.com wrote:

Sorry to chime in late on this conversation - but is the http://golr.geneontology.org/solr/ URL deprecated now? It's not working on our WA1 instances, nor does it appear to be on http://icebox.lbl.gov/Apollo2/annotator/index. Looks like it is on your staging site, but that's where the proxy is set up. Any advice would be appreciated!

— Reply to this email directly or view it on GitHub.

[childers]

Is the ssl version supported too? It shows unable to connect: https://golr.berkeleybop.org/solr/

Can we get native ssl support for your GOLR server?

Thanks! Chris

http://golr.berkeleybop.org/solr/

On Thu, Nov 19, 2015 at 11:50 AM, Nathan Dunn notifications@github.com wrote:

You are correct.

In 2.0.1 it automatically uses a database configurable proxy (with interface!) and has the proper default set.

If you are using 1.0 you need to switch it to this in AnnoTrack.js:

http://golr.berkeleybop.org/solr/

I’ll make an announcement for 2.0.0 and show how to change it.

Nathan

On Nov 19, 2015, at 7:45 AM, mpoelchau notifications@github.com wrote:

Sorry to chime in late on this conversation - but is the http://golr.geneontology.org/solr/ URL deprecated now? It's not working on our WA1 instances, nor does it appear to be on http://icebox.lbl.gov/Apollo2/annotator/index. Looks like it is on your staging site, but that's where the proxy is set up. Any advice would be appreciated!

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/148#issuecomment-158115702.

[nathandunn]

So . . part of the reason for the proxy (in 2.0.X) is that the Apollo server can run in https, but the connectivity is in http.

I asked about getting the golr server to use https and the answer was that it will happen at some time, but nothing immediate.

Nathan

On Nov 19, 2015, at 9:06 AM, childers notifications@github.com wrote:

Is the ssl version supported too? It shows unable to connect: https://golr.berkeleybop.org/solr/

Can we get native ssl support for your GOLR server?

Thanks! Chris

http://golr.berkeleybop.org/solr/

On Thu, Nov 19, 2015 at 11:50 AM, Nathan Dunn notifications@github.com wrote:

You are correct.

In 2.0.1 it automatically uses a database configurable proxy (with interface!) and has the proper default set.

If you are using 1.0 you need to switch it to this in AnnoTrack.js:

http://golr.berkeleybop.org/solr/

I’ll make an announcement for 2.0.0 and show how to change it.

Nathan

On Nov 19, 2015, at 7:45 AM, mpoelchau notifications@github.com wrote:

Sorry to chime in late on this conversation - but is the http://golr.geneontology.org/solr/ URL deprecated now? It's not working on our WA1 instances, nor does it appear to be on http://icebox.lbl.gov/Apollo2/annotator/index. Looks like it is on your staging site, but that's where the proxy is set up. Any advice would be appreciated!

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/148#issuecomment-158115702.

— Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/148#issuecomment-158120381.

[mpoelchau]

Great, thanks, the new URL works!

Do you know for how long the previous URL has not been functioning? We may need to go back and inform our annotators that their GO and PubMed information hasn’t been saving, and it would be helpful to have a time frame.

Thanks!

Monica

Monica Poelchau, Ph.D. USDA-ARS National Agricultural Library 10301 Baltimore Ave, Beltsville, MD 20705 monica.poelchau@ars.usda.gov

From: Nathan Dunn notifications@github.com<mailto:notifications@github.com> Reply-To: GMOD/Apollo reply@reply.github.com<mailto:reply@reply.github.com> Date: Thursday, November 19, 2015 at 9:50 AM To: GMOD/Apollo Apollo@noreply.github.com<mailto:Apollo@noreply.github.com> Cc: Monica Poelchau monica.poelchau@ars.usda.gov<mailto:monica.poelchau@ars.usda.gov> Subject: Re: [Apollo] make GOLR server / URL configurable (#148)

You are correct.

In 2.0.1 it automatically uses a database configurable proxy (with interface!) and has the proper default set.

If you are using 1.0 you need to switch it to this in AnnoTrack.js:

http://golr.berkeleybop.org/solr/

I’ll make an announcement for 2.0.0 and show how to change it.

Nathan

On Nov 19, 2015, at 7:45 AM, mpoelchau notifications@github.com<mailto:notifications@github.com> wrote:

Sorry to chime in late on this conversation - but is the http://golr.geneontology.org/solr/ URL deprecated now? It's not working on our WA1 instances, nor does it appear to be on http://icebox.lbl.gov/Apollo2/annotator/index. Looks like it is on your staging site, but that's where the proxy is set up. Any advice would be appreciated!

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/GMOD/Apollo/issues/148#issuecomment-158115702.

[nathandunn]

It should be saving just fine. Just the lookups wouldn’t have worked (so they couldn’t annotate it).

Nathan

On Nov 19, 2015, at 11:11 AM, mpoelchau notifications@github.com wrote:

Great, thanks, the new URL works!

Do you know for how long the previous URL has not been functioning? We may need to go back and inform our annotators that their GO and PubMed information hasn’t been saving, and it would be helpful to have a time frame.

Thanks!

Monica

Monica Poelchau, Ph.D. USDA-ARS National Agricultural Library 10301 Baltimore Ave, Beltsville, MD 20705 monica.poelchau@ars.usda.gov

From: Nathan Dunn notifications@github.com<mailto:notifications@github.com> Reply-To: GMOD/Apollo reply@reply.github.com<mailto:reply@reply.github.com> Date: Thursday, November 19, 2015 at 9:50 AM To: GMOD/Apollo Apollo@noreply.github.com<mailto:Apollo@noreply.github.com> Cc: Monica Poelchau monica.poelchau@ars.usda.gov<mailto:monica.poelchau@ars.usda.gov> Subject: Re: [Apollo] make GOLR server / URL configurable (#148)

You are correct.

In 2.0.1 it automatically uses a database configurable proxy (with interface!) and has the proper default set.

If you are using 1.0 you need to switch it to this in AnnoTrack.js:

http://golr.berkeleybop.org/solr/

I’ll make an announcement for 2.0.0 and show how to change it.

Nathan

On Nov 19, 2015, at 7:45 AM, mpoelchau notifications@github.com<mailto:notifications@github.com> wrote:

Sorry to chime in late on this conversation - but is the http://golr.geneontology.org/solr/ URL deprecated now? It's not working on our WA1 instances, nor does it appear to be on http://icebox.lbl.gov/Apollo2/annotator/index. Looks like it is on your staging site, but that's where the proxy is set up. Any advice would be appreciated!

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/GMOD/Apollo/issues/148#issuecomment-158115702. — Reply to this email directly or view it on GitHub https://github.com/GMOD/Apollo/issues/148#issuecomment-158161172.

[nathandunn]

from @kltm . . .

http://golr.geneontology.org/select?q=: http://golr.berkeleybop.org/select?q=: http://golr.berkeleybop.org/solr/select?q=:

The first two are "forever" URLs, with the first being the "production" one and the second being a special use one (failback, some experiments, etc.). The last one there is functional for the time being, but is on notice--it should be replaced in all code going forward with the first (this has been harmonized with production, so we should be good in the future during outages and the like). At a future date, the third should be purged at version EOL

[nathandunn]

To Test:

• make GO annotations on a feature
• see if you can get to the Admin -> Proxies tab

[monicacecilia]

:dancer: