Swap fetchVariableQuery over to SOLR from NEO4J

[Robbie1977]

We need to move fetchVariableQuery via neo4J (https://github.com/VirtualFlyBrain/geppetto-vfb/blob/solr-queries/model/vfb.xmi#L625) over to SOLR (started here: https://github.com/VirtualFlyBrain/geppetto-vfb/blob/solr-queries/model/vfb.xmi#L917).

This requires a new SOLR datasource started here: https://github.com/openworm/org.geppetto.datasources/tree/addingSOLR

New uk bundle with boiler plate copy of the NEO4J processing step for solr added: https://github.com/VirtualFlyBrain/uk.ac.vfb.geppetto/blob/addSOLRcache/src/main/java/uk/ac/vfb/geppetto/VFBProcessTermInfoCachedJson.java This will need a step to convert the JSON string into something the be passed as per the neo4j results which is also a string but the reponses are formatted differently. This is as far as I got as couldn't get any reponse from debug so assume the datasource is failing to process the SOLR call correctly but little feedback on the server end.

The partially started model is here (new query added with the start of processing): https://github.com/VirtualFlyBrain/geppetto-vfb/tree/solr-queries

Note the branches used here:

ARG geppettoRelease=vfb_20200604_a
ARG geppettoModelRelease=vfb_20200604_a
ARG geppettoCoreRelease=VFBv2.2.0
ARG geppettoSimulationRelease=vfb_20200604_a
ARG geppettoDatasourceRelease=addingSOLR
ARG geppettoModelSwcRelease=v1.0.1
ARG geppettoFrontendRelease=VFBv2.1.0.3
ARG geppettoClientRelease=VFBv2.2.3
ARG ukAcVfbGeppettoRelease=addSOLRcache

https://github.com/VirtualFlyBrain/geppetto-vfb/blob/solr-queries/Dockerfile#L11

[jrmartin]

Hi @Robbie1977 , I have been debugging this issue with the branches you created. I'm debugging using the SOLR query instead of Neo4J, getting 400 responses from the SOLR server. https://solr.VirtualFlyBrain.org/solr/vfb_json/select

This is the query parameters sent to the SOLR server, do they look good to you: { "fl":"term_info","indent":"true","q.op":"AND",q:"VFB_00101567", "defType": "edismax", "qf": "id", "indent": "true", "start": "0", "pf":"true", "rows": "10000", "wt": "json", }

The 400 returned error is this , finds the query invalid with 'fl' as a key:

{
  "responseHeader":{
    "status":400,
    "QTime":1,
    "params":{
      "json":"{\n  \"fl\":\"term_info\",\"indent\":\"true\",\"q.op\":\"AND\",q:\"VFB_00101567\",\n  \"defType\": \"edismax\",\n  \"qf\": \"id\",\n  \"indent\": \"true\",\n  \"start\": \"0\",\n  \"pf\":\"true\",\n  \"rows\": \"10000\",\n  \"wt\": \"json\",\n}"}},
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"Unknown top-level key in JSON request : fl",
    "code":400}}

[Robbie1977]

@jrmartin there isn't anything wrong with the query but you are missing quotes on the q:, you should also only request 1 row, and pf shouldn't be called.

http://solr.virtualflybrain.org/solr/vfb_json/select?defType=edismax&fl=term_info&indent=true&q.op=OR&q=VFB_00002990&qf=id&rows=1

The simplest that will work is: http://solr.virtualflybrain.org/solr/vfb_json/select?fl=term_info&q=id:VFB_00002990&rows=1

[Robbie1977]

@jrmartin the response bugs me as not sure the sub 'json' should be there as I get responces like:

{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"VFB_00002990",
      "defType":"edismax",
      "indent":"true",
      "qf":"id",
      "fl":"term_info",
      "q.op":"OR",
      "rows":"1",
      "_":"1675434400746"}},
  "response":{"numFound":1,"start":0,"numFoundExact":true,"docs":[
      {

with parameters in the top level under params

[jrmartin]

Thanks for checking @Robbie1977 , the query works now. Still working out the JSON processing part.

[jrmartin]

Made progress and it looks most things are loading with SOLR query now. Further testing next two days. @Robbie1977 All changes are in 3 branches you created if you want to test.

ARG geppettoDatasourceRelease=addingSOLR ARG ukAcVfbGeppettoRelease=addSOLRcache ARG geppetto-vfb=solr-queries

[jrmartin]

The swap works fully now. Only branches with changes made are:

ARG geppettoDatasourceRelease=addingSOLR ARG ukAcVfbGeppettoRelease=addSOLRcache ARG geppetto-vfb=solr-queries