Use New REST API

[everbeek]

We will need to use the new API, in particular because I was told they will remove the existing REST services once that API is completed.

I also expect to be able to use the new API to make fewer, more efficient REST calls (fulfilling #255 hopefully). I was told that making special calls for visualizations will still be available, when we need them.

http://stagedata.bioontology.org/ http://stagedata.bioontology.org/documentation https://github.com/ncbo/ncbo_rest_sample_code

[everbeek]

Can keep the top level client.services package, with addition of URL builder that points at new API endpoint, but all the actual services and parsers will need ot be re-written. Will make parallel package of classes to do so, which we can shift around after.

[everbeek]

The ontology overview performance is the big motivator for me at the moment, but the mapping services are not yet available. I want to work on the ontology embed, so I will come back to this later.

[everbeek]

Looking at an end of June or end of July date for this task. I will work on what is already accessible, so that when the time comes there's something to work off of. Not sure if this can work at the same time as what we have in place already, since the data forms returned may be significantly different, and might necessitate broader changes. We'll see.

[everbeek]

Starting with the ontology list service.

[everbeek]

A few ontology things have been moved off the ontologies listing json. Description is one of them. I am checking with Paul if there are plans to be able to request that a given json that is filled in with a resource link can be requested to be expanded before receiving them back (to eliminate an additional request and the associated turn around time).

[everbeek]

Have ontology search working with the new API. Looking to see if I can replace parts one by one and still allow the old system to be in place where necessary.

[everbeek]

Have both the new endpoint working with ontology search, and nothing else, while everything else uses the old URL builder and original parsers.

Due to some data changes in the service, having a fully functioning part-way converted version won't really work. For example, ontology IDs aren't available int he same way it seems, and URLs are now more useful for unique identification. If the new URL ontology ID is pushed into the old rest service calls for other pieces of data, it will fail, because those expect integers. I am looking into this to make sure I am seeing it correctly. If this is the case, we should not merge to master until the beta for the new API is out. If we do, we have to unbind the NEW_API stuff in BioMixerClientModule.

[everbeek]

The next step is to identify the parsers and services that can have their data requirements fulfilled by the new API.

[everbeek]

Having a seriously difficult time with injecting the classes for concept search. I had some difficulties with the ontology search, which is parallel. Also, it seems there aren't enough services in the new API yet to continue. I will get what I am working on clean, commit it, and wait until beta comes out, I think. The term search makes use of several rest services...maybe I can sort something out.

[everbeek]

Sorted out the injection issues, and have the concept search classes in place, but unmodified from the old ones. The search service is feeding me odd data (444 pages for the term "mouse", and nothing if I access it from a browser). The page count is because their implementation doesn't tell me how to request an exact match (as we do in the old one).

[everbeek]

To maintain both sets of classes, I can't seem to use @Named annotations with injection. It is supposed to work, from what I can find out online, but it won't for me. Rather than banging my head against the desk, I will extend the interface being injected upon, with another interface, which allows me to specify the exact injection. I still question if injection is useful to us, but I am not stripping it out right now.

So...I cannot use @Named annotations for injection, because the broader injection clobbers the named one whenever the corresponding binding is not commented out. After commenting it out, the smooth sailing might be an illusion, because compiling again forces it to use the old one again. Hard to articulate, but it's a development and deployment problem.

[everbeek]

I think I changed my mind. I should not try to keep all the old REST calls working as I make this branch. I should indeed replace the old ones fully, as each becomes usable. It's too much of a hassle to do it the way I am trying to.

I will think about it a bit more before changing my tack.

[everbeek]

Keeping the old stuff makes it somewhat easier to keep track of what needs work, so I'll keep on this path for now. Committing so I can do an internal test deployment on master for the ontology view.

[everbeek]

Getting back on this, but I am saving some effort by updating the D3 implementation of the ontology mapping overview to the new API. Will work on that in master since it is self contained anyway.

[everbeek]

See #300 for a particular REST call I am unsure about.

[everbeek]

Removed the mess I made before. The old system is defunct, so can be fully removed. This work is in a state of mess right now, but that's why we have branches, right?

Left off fixing up ontology overview new API, because I am more familiar with how that works out. It is a huge hassle to do this in BioMixer relative to the more direct D3 based script file. See upcoming commit for where I was most active, to regain context.

http://127.0.0.1:8888/?mode=embed&embed_mode=ontology_mapping_neighbourhood&virtual_ontology_id=1033&ontology_acronym=NMR

Need to set up third layer callback for individual ontology details.

[everbeek]

Continue along this path with remaining REST calls, test all visualizations and main search functionality in full version. Mostly excised virtual ontology ids.

[everbeek]

Side note: the new API with the BioMixer based ontology overview loads for me in 17 seconds. It would be significantly faster than the old 22 seconds if the latest_submission calls were not required for the ontology Description data, though I could be wrong. Hmm, I re-ran and am getting 18 second loads on the old one today, so perhaps it is a wash. The new D3 views are getting about 14 seconds and 10 seconds (new API and old, respectively). There might be stuff going on on the REST server side right now.

Onward.

[everbeek]

Working on concept calls.

Seems like for path to root, I can call something like this: http://stagedata.bioontology.org/ontologies/BRO/classes/http%3A%2F%2Fbioontology.org%2Fontologies%2FBiomedicalResourceOntology.owl%23Ontology_Development_and_Management?include=parents,ancestors

And receive the parents of parents, forever. This is odd, since only the combination of parents and ancestors in the URL parameters gives this sort of result.

[everbeek]

When testing path to root, I like this comparison case: http://bioportal.bioontology.org/ontologies/SNOMEDCT?p=classes&conceptid=http%3A%2F%2Fpurl.bioontology.org%2Fontology%2FSNOMEDCT%2F35146001#visualization

[everbeek]

Still down the rabbit hole, and I am working on parsing mappings. This has changed, since mappings calls now give the full data for the mapped concepts:

http://data.bioontology.org/ontologies/SNOMEDCT/classes/http%3A%2F%2Fpurl.bioontology.org%2Fontology%2FSNOMEDCT%2F35146001/mappings/?apikey=6700f7bc-5209-43b6-95da-44336cbc0a3a

This is happening for each concept as it is added to the graph, via the automatic expanders. It even happens for the graphs that do not involve mappings. The reason why is that we need the mapping data in case a node that maps to it is brought into the graph. This can happen via search in the full version, but also via expansion of the node using the node drop down menu in the embed versions. For larger graphs this results in very many REST calls, and now that the REST calls return the concept data (not merely concept ids to store as mapping objects), it means we have a lot of data transmission and parsing to do, just to get mapping data that may or may not be used.

So what do we do? Under what conditions do we strictly need, and strictly not need mapping data? Can we disable the automatic expander? Is there an alternate REST call that only gives us IDs? See #303.

[everbeek]

Have the "term neighborhood" working, except for has_a relations which are vital. http://127.0.0.1:8888/?mode=embed&embed_mode=concept_neighbourhood&ontology_acronym=SNOMEDCT&full_concept_id=http://purl.bioontology.org/ontology/SNOMEDCT/35146001

Also, sometimes I don't get results for some arcs. It is intermittent. I will commit then fix that bug, then get on to path to root and mappings neighbourhood.

[everbeek]

Time to fix up mappings neighbourhood now. First, I will fix #302, with the composition relations.

[everbeek]

The has part and part of relations were being parsed and being added as partial properties to the target resource, but nothing was happening. I dug in further, and realized that the same resource might be getting created multiple times. I am not sure if this was a problem before, but it was certainly capable of happening. I am fairly certain that resources are intended to be truly unique, especially since the comparison method makes use of the "unique" URIs. I decided to add a factory method that checks for an existing resource prior to creating one from scratch.

If this is the wrong solution, or is invalid, it will be a bigger mess to fix up my problem with the partial properties.

[everbeek]

Ok, second problem (possibly the real one). applyProperties clobbers previous properties, which doesn't surprise me per se, but it sure seems like an odd thing to do with properties such as node relational data that won't be changing over the course of a visualization. This is a hack, but I am adding a conditional filter in that method to inspect the property name, and call existing specialized add methods for properties when they come in that method. But...what other properties get applied like this, anyway? Well, it appears that only neighbourhood data gets added this way! I am renaming the method, and getting it to add rather than to replace.

[everbeek]

I have part of and has part relations working now. Unfortunately...layouts are acting up! Making new case for that (#308).

[everbeek]

Term neighbourhood is in good shape: http://127.0.0.1:8888/?mode=embed&embed_mode=concept_neighbourhood&ontology_acronym=SNOMEDCT&full_concept_id=http://purl.bioontology.org/ontology/SNOMEDCT/35146001

Now work on mapping neighbourhood: http://127.0.0.1:8888/?mode=embed&embed_mode=mapping_neighbourhood&ontology_acronym=SNOMEDCT&full_concept_id=http://purl.bioontology.org/ontology/SNOMEDCT/35146001

[everbeek]

I have things in good shape. I need to make sure I didn't miss anything, and I think the easiest way is by stripping out old XML parsers, and looking for any XML parsers that are also now invalid. Made case #312 for that.

[everbeek]

Should I make the REST calls work for the full version? Yes, I think so.

[everbeek]

I think I have all the new REST calls in place. Some views trigger a lot of them, so we might look at removing redundant calls (#311), and at asking for certain types of info to be made available in other calls to reduce the number made. (It took about 24 days to get through the changes from the D3 embed to the dashboard version).

[everbeek]

Merged to master. Related problems can go in new issue Closing this one.