Evaluate and determine ontology/user database system selection for overhauled ORR backend (was: Upgrade BioPortal Core to recent version)

[mmisw]

From caru...@gmail.com on September 29, 2010 17:09:13

What capability do you want added or improved? The ORR system (in particular the Ont service) currently uses tag 1005 of the BioPortal core (Sept/08). This request is to upgrade the the most recent and stable version. What version of the product are you using? - Bioportal Core tag 1005

• MMI Ontology and Term URI Resolver. Version 2.0.9.beta (201009291428). This service interacts with the BioPortal back-end via its REST interface, but also via direct access to the bioportal database. Only the Ont service will be affected by potential incompatibilities upon an upgrade of bioportal core. The plan is to interact with bioportal exclusively via the REST service. A first step is to have a recent version of such system in place.

Original issue: http://code.google.com/p/mmisw/issues/detail?id=283

[mmisw]

From paul.r.a...@gmail.com on September 29, 2010 18:46:49

Unfortunately the upgrade that was done about a year ago to support ontology-based metadata storage really changed a lot of the internals of the system and since Stanford only had to migrate once there's no really simple path to doing it.

The upgrade basically made it so that all of the metadata for ontologies and users (and soon projects and reviews -- which shouldn't affect you I think) are stored as instances in a Protege back-end. In the case of users they are synchronized between a MySQL table and the ontology (at user creation time, there is no ongoing process that checks to make sure they are still in sync).

I advised a person who recently contacted me about upgrading to install a new version of BioPortal alongside their existing one and then write some scripts to translate the data from old to new using the REST services. The only thing you would lose via this method are the version identifiers that are used. Since MMI has basically abstracted those away anyway it might not matter much.

Unfortunately we don't have a great way to compare database structure between tags. You can look in the db folder to see what migrations have happened. For sure the table that stored ontology metadata information, including the urn field, is gone. There is no equivalent field in the metadata ontology, but we do support OMV and some extensions, and the metadata is meant to be extensible (though the code dealing with it in the system may not be as easily extended as adding the appropriate classes in the metadata ontology). More info on the metadata is here: http://www.bioontology.org/wiki/index.php/Architecture

[mmisw]

From caru...@gmail.com on September 30, 2010 14:26:29

Thanks Paul.

Yeah, tag 1030 (the latest at time of writing) does not have the urn field (*). And in our deployment we exploit the "urn" column in tag 1005 to store the ontology URIs. We also access teh view v_ncbo_ontology (which is gone in 1030).

So, the upgrade will definitely not be as a straightforward as I thought.

However, I find the new bioportal metadata architecture very interesting and in fact I think it should align well with what we have been already doing in terms of OMV. However, one of the first things we'd need to solve is the appropriate handling of URIs for the registered ontologies. In our approach, these URIs are first-class citizens (ie., the URI is the primary identifier for the ontology). The "urn" field in tag 1005 allows us to, for example, quickly retrieve an ontology by its URI. BTW, we also capture the URI in the omv:uri metadata property, but I think a more direct mechanism should be put in place for efficient handling of URIs as ontology identifiers.

(*) tag 1050:
Database: bioportal
+-------------------------------------------+
|                  Tables                   |
+-------------------------------------------+
| ncbo_admin_application                    |
| ncbo_app_text                             |
| ncbo_app_text_revision                    |
| ncbo_l_additional_metadata                |
| ncbo_l_app_text_datatype                  |
| ncbo_l_category                           |
| ncbo_l_role                               |
| ncbo_l_status                             |
| ncbo_ontology                             |
| ncbo_ontology_additional_version_metadata |
| ncbo_ontology_category                    |
| ncbo_ontology_file                        |
| ncbo_ontology_load_queue                  |
| ncbo_ontology_version                     |
| ncbo_ontology_version_metadata            |
| ncbo_user                                 |
| ncbo_user_role                            |
| v_ncbo_ontology                           |
+-------------------------------------------+

tag 1030:
Database: bioportal
+--------------------------+
|          Tables          |
+--------------------------+
| ncbo_admin_application   |
| ncbo_app_text            |
| ncbo_app_text_revision   |
| ncbo_l_app_text_datatype |
| ncbo_l_notification_type |
| ncbo_l_role              |
| ncbo_l_status            |
| ncbo_l_usage_event_type  |
| ncbo_ontology_file       |
| ncbo_ontology_load_queue |
| ncbo_usage_log           |
| ncbo_user                |
| ncbo_user_role           |
| ncbo_user_subscriptions  |
+--------------------------+

[mmisw]

From caru...@gmail.com on September 30, 2010 15:56:34

Paul: Where can I find documentation or guidance about the "update" RESTful operations (eg., POST, PUT, DELETE)? http://www.bioontology.org/wiki/index.php/BioPortal_REST_services seems pretty much about GET operations.

I'm thinking that an interesting approach for the upgrade is:

• Assume A is the current deployment of tag 1005 (which is where the Ont service is running as well).
• Assume B is the new deployment (probably in a different machine as it seems is not easy to indicate a different set of names for the databases, especially regarding the sql scripts).
• set up new bioportal from scratch in B
• write a program that accesses all registered ontologies (including all its versions) from A via the Ont service, and then submit them to the new deployment B using the corresponding bioportal REST service.
• along with this one could start doing the modifications in the Ont service, so all interaction with bioportal is via REST.

Cc: paul.r.alexander

[mmisw]

From caru...@gmail.com on September 30, 2010 21:30:32

I just completed the installation of tag 1030 on a linux box (mmi2). In summary, very successful (creation/update/deletion of users OK; creation/update of ontologies OK). Only a couple of issues identified so far:

• REST operation to delete ontology fails with: "The method specified in the request is not allowed for the resource identified by the request URI" (errorCode:405)
• creation of project succeeds, but retrieval fails.

More details in the attached file.

It seems that I'm missing some appropriate configuration related with the protege instance database(?). Suggestions/pointers are welcome.

Attachment: DEPLOYMENT_ON_MMI2.txt

(contents of the attch follows)

Installing/testing BioPortal Core tag 1030
2010-09-30

All the following on the mmi2 machine:
$ uname -a
Linux mmi2 2.6.24-16-generic #1 SMP Thu Apr 10 13:23:42 UTC 2008 i686 GNU/Linux

  cd ~/bioportal_core/tags/
  svn checkout https://bmir-gforge.stanford.edu/svn/bioportal_core/tags/1030
  cd 1030/

Created the databases:
   bioportal
   bioportal_protege
   bioportal_lexgrid

For users, I'm basically using MySQL's root account.

carueda@mmi2:~/bioportal_core/tags/1030$ mysql -u root -p < db/sql/bioportal_db.sql 
carueda@mmi2:~/bioportal_core/tags/1030$ mysql -u root -p < db/sql/bioportal_lookup_data.sql 

# this is new: it's not described in the dployment document, but this solved
# an error related with the bioportal_protege.metadata table not found.
carueda@mmi2:~/bioportal_core/tags/1030$ mysql -u root -p < db/sql/bioportal_metadata_base.sql 

# prepare directory for bioportal resources:
carueda@mmi2:~/bioportal_core/tags/1030$ mkdir -p /ORR/bioportal-1030/resources

# edit build properties
carueda@mmi2:~/bioportal_core/tags/1030$ vi build.properties

# with the following worth-noting properties:
appserver.home=/home/carueda/Software/apache-tomcat-6.0.20
obo.pull.scheduler.enabled=false
ontology.parse.scheduler.enabled=false

bioportal.jdbc.url=jdbc:mysql://localhost:3306/bioportal
protege.jdbc.url=jdbc:mysql://localhost:3306/bioportal_protege
lexgrid.db.url=jdbc:mysql://localhost:3306/bioportal_lexgrid

bioportal.jdbc.username=root
bioportal.jdbc.password=******

# similar account info for other entries: protege.jdbc.*, lexgrid.db.*, test.db.*

# now, build:
carueda@mmi2:~/bioportal_core/tags/1030$ ant clean
...
BUILD SUCCESSFUL

carueda@mmi2:~/bioportal_core/tags/1030$ ant deploywar
...
BUILD SUCCESSFUL

$ tail -f /home/carueda/Software/apache-tomcat-6.0.20/logs/catalina.out
# looks good.

Tests:   
-----

* GET ontologies
OK:  <?xml version="1.0" encoding="UTF-8"?><success><accessedResource>/bioportal/ontologies</accessedResource><accessDate>2010-09-30 17:16:38.18 PDT</accessDate><data><list/></data></success>

* GET users
OK:
  <?xml version="1.0" encoding="UTF-8"?>
  <success>
    <accessedResource>/bioportal/users</accessedResource>
    <accessDate>2010-09-30 17:25:26.14 PDT</accessDate>
    <data>
      <list/>
    </data>
  </success>  

* GET categories
OK: 
  <?xml version="1.0" encoding="UTF-8"?>
  <success>
    <accessedResource>/bioportal/categories</accessedResource>
    <accessDate>2010-09-30 17:28:08.983 PDT</accessDate>
    <data>
      <list/>
    </data>
  </success>

* Create a user (POST): OK. However a lot of exceptions were logged out prior to
the successful message printed by the restlet engine. It seems the stacktrace
was generated by a different thread, perhaps some scheduled task not necessarily
related with the create-user action(?). The first portion of the latest
exception was:

WARNING: Could not process untyped resource from statement: (_:A878, http://www.w3.org/2002/07/owl#onProperty, http://protege.stanford.edu/ontologies/ChAO/changes.rdfs#created)
java.lang.ClassCastException: edu.stanford.smi.protege.model.DefaultCls cannot be cast to edu.stanford.smi.protegex.owl.model.RDFResource
    at edu.stanford.smi.protegex.owl.model.impl.AbstractRDFSClass.createInstance(AbstractRDFSClass.java:103)
    at edu.stanford.smi.protegex.owl.jena.parser.TripleProcessorForUntypedResources.handleUndefinedSubject(TripleProcessorForUntypedResources.java:248)
    at edu.stanford.smi.protegex.owl.jena.parser.TripleProcessorForUntypedResources.createUntypedObject(TripleProcessorForUntypedResources.java:171)
    at edu.stanford.smi.protegex.owl.jena.parser.TripleProcessorForUntypedResources.resolvUndefinedTriple(TripleProcessorForUntypedResources.java:140)
    at edu.stanford.smi.protegex.owl.jena.parser.TripleProcessorForUntypedResources.createUntypedResources(TripleProcessorForUntypedResources.java:108)
    at edu.stanford.smi.protegex.owl.jena.parser.TripleProcessor.createUntypedResources(TripleProcessor.java:109)
    at edu.stanford.smi.protegex.owl.jena.parser.TriplePostProcessor.doPostProcessing(TriplePostProcessor.java:108)
    at edu.stanford.smi.protegex.owl.jena.parser.TripleProcessor.doPostProcessing(TripleProcessor.java:100)
    at edu.stanford.smi.protegex.owl.jena.parser.ProtegeOWLParser.doFinalPostProcessing(ProtegeOWLParser.java:349)
    at edu.stanford.smi.protegex.owl.database.OWLDatabaseKnowledgeBaseFactory.loadKnowledgeBase(OWLDatabaseKnowledgeBaseFactory.java:111)
    at edu.stanford.smi.protegex.owl.database.creator.OwlDatabaseCreator.createFromExistingSources(OwlDatabaseCreator.java:86)
    at edu.stanford.smi.protegex.owl.database.creator.OwlDatabaseCreator.create(OwlDatabaseCreator.java:47)
    at org.ncbo.stanford.manager.AbstractOntologyManagerProtege.createMetadataKnowledgeBaseInstance(AbstractOntologyManagerProtege.java:299)
    at org.ncbo.stanford.manager.AbstractOntologyManagerProtege.getMetadataOWLModel(AbstractOntologyManagerProtege.java:318)
    at org.ncbo.stanford.manager.metadata.impl.UserMetadataManagerImpl.saveUser(UserMetadataManagerImpl.java:19)
    at org.ncbo.stanford.service.user.impl.UserServiceImpl.createUser(UserServiceImpl.java:119)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at $Proxy15.createUser(Unknown Source)
    at org.ncbo.stanford.view.rest.restlet.user.UsersRestlet.createUser(UsersRestlet.java:73)
    at org.ncbo.stanford.view.rest.restlet.user.UsersRestlet.postRequest(UsersRestlet.java:39)
    at org.ncbo.stanford.view.rest.restlet.AbstractBaseRestlet.handle(AbstractBaseRestlet.java:50)
    at org.restlet.Filter.doHandle(Filter.java:150)

* Update a user (POST): OK!

* Validate user credentials: OK!

* Delete user: OK!

* Create user again: OK!

* Create another user: OK!

* Create (POST) ontology: OK!

And I notice that the response includes <urn>...</urn> ! (rather unexpected
faccording to Pauls' comments)

However the regular listing: GET ...ontologies does NOT include <urn>. At this
point I update WebRoot/WEB-INF/resources/xslt/ontology_light.xsl as with the
1005 tag, ie:

    <xsl:template match="ontology">
        <ontology>
            <xsl:copy-of select="id" />
            <xsl:copy-of select="urn" />
            <xsl:copy-of select="displayLabel"/>
            ...
and, after rebuiling/deploying, the listing now includes <urn> as before!

* Update ontology (PUT)   OK!

* Delete ontology FAIL! : method not allowed:
<?xml version="1.0" encoding="UTF-8"?>
  <errorStatus>
    <accessedResource>/bioportal/ontologies/delete/10002</accessedResource>
    <accessDate>2010-09-30 19:52:37.669 PDT</accessDate>
    <shortMessage>Method Not Allowed</shortMessage>
    <longMessage>The method specified in the request is not allowed for the resource identified by the request URI</longMessage>
    <errorCode>405</errorCode>

  </errorStatus>  

Although not of immediate relevance, I quickly tested some of the other operations:

* create project http://mmi2.shore.mbari.org:9090/bioportal/project/new
OK

* However, retrieval fails: "RDFSLiteral with unrecognized type: http://www.w3.org/2001/XMLSchema#dateTime"

Seems like this problem is related with the Protege instance data.

* DELETE project seems to have completed OK.  http://mmi2.shore.mbari.org:9090/bioportal/project/1

* GET projects for a user: 
OK  --> empty list 

* Create again the project--> OK
* List projects --> <errorCode>500</errorCode>
The stacktrace:

java.lang.IllegalArgumentException: RDFSLiteral with unrecognized type: http://www.w3.org/2001/XMLSchema#dateTime
    at org.ncbo.stanford.util.protege.PropertyUtils.convertRDFSLiteralToObject(PropertyUtils.java:190)
    at org.ncbo.stanford.util.protege.PropertyUtils.convertOWLValueToJava(PropertyUtils.java:146)
    at org.ncbo.stanford.util.protege.PropertyUtils.getPropertyValue(PropertyUtils.java:49)
    at org.ncbo.stanford.manager.metakb.protege.DAO.base.PropertyMap.getOWLValue(PropertyMap.java:167)
    at org.ncbo.stanford.manager.metakb.protege.DAO.base.PropertyMap.copyValueToBean(PropertyMap.java:93)
    at org.ncbo.stanford.manager.metakb.protege.DAO.base.AbstractDAO.copyTimestampsToBean(AbstractDAO.java:389)
    at org.ncbo.stanford.manager.metakb.protege.DAO.base.AbstractDAO.copyIndividualPropertiesToBean(AbstractDAO.java:341)
    at org.ncbo.stanford.manager.metakb.protege.DAO.base.AbstractDAO.getInstancesForSPARQLQuery(AbstractDAO.java:230)
    at org.ncbo.stanford.manager.metakb.protege.ProjectMetadataManagerImpl.getProjectsForUser(ProjectMetadataManagerImpl.java:28)
    at org.ncbo.stanford.service.metadata.impl.ProjectMetadataServiceImpl.getProjectsForUser(ProjectMetadataServiceImpl.java:27)
    at org.ncbo.stanford.view.rest.restlet.metadata.ProjectListRestlet.getRequest(ProjectListRestlet.java:49)
    at org.ncbo.stanford.view.rest.restlet.AbstractBaseRestlet.handle(AbstractBaseRestlet.java:44)
    at org.restlet.Filter.doHandle(Filter.java:150)

Again, seems related with Protege. Probably some missing configuration or wrong
preparation of the database?

[carueda]

Renaming entry to "Evaluate and determine ontology/user database system selection for overhauled ORR backend" as a more general task; in particular to consider not only BioPortal but also other potential alternatives.