search

essepuntato / LODE

Live OWL Documentation Environment, to convert OWL ontologies into HTML human-readable pages.
ISC License
106 stars 55 forks source link

http://www.essepuntato.it/lode service down #16

Open lewismc opened 6 years ago

lewismc commented 6 years ago

Hi @essepuntato @seralf @nicholascar I love this service and have been using it to generate a cache visualizations of the rather large SWEET ontology suite, the visualizations can be seen at https://esipfed.github.io/stc/sweet_lode/sweetAll.html

As I began iterating through the generation of visualizations for all of the files in SWEET, after a while I encountered the following responses

<html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.servlet.ServletException: Servlet execution threw an exception
</pre></p><p><b>root cause</b> <pre>java.lang.OutOfMemoryError: Java heap space
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/6.0.35 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.35</h3></body></html>

...

java.lang.OutOfMemoryError: Java heap space

I therefore fear that I may have crashed your service. I wanted to say that I was not executing these requests in parallel or within a short time of each other. Each request was submitted literally minutes apart. Apologies, if I crashed the service.

nicholascar commented 6 years ago

Hi Lewis, We have a v2 up at http://lode2.linked.data.gov.au that incorporates a few updates to the HTML generated. @essepuntato and I have discussed folding those changes back into the main codebase but, for now, there are two services in case one goes down and you are caught off guard.

lewismc commented 6 years ago

Thanks @nicholascar :)

lewismc commented 6 years ago

@nicholascar there are several improvements in the data.gov.au service you pointed me to so thank you for that. I just used the service to render the following file https://github.com/ESIPFed/sweet/blob/master/src/sweetAll.ttl which is essentially just a list of imports. Again, I'm running in to issues when executing the following request

http://lode2.linked.data.gov.au/extract?imported=true&webvowl=true&url=https://raw.githubusercontent.com/ESIPFed/sweet/master/src/sweetAll.ttl

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /extract.

Reason: Error reading from remote server

Apache/2.4.18 (Ubuntu) Server at lode2.linked.data.gov.au Port 80

Any advice on better use of the service? Thanks in advance.

nicholascar commented 6 years ago

Hmmm... neither LODE version can handle your hugh import command! It's clearly just timing out. I ran this Python script to merge all the graphs:

import requests
import rdflib

gg = rdflib.Graph()

with open('imports.txt', 'r') as f:
    for l in f.readlines():
        r = requests.get(l.strip(), headers={'Accept': 'text/turtle'})
        g = rdflib.Graph().parse(data=r.content.decode('utf-8'), format='turtle')
        print(len(g))
        gg += g

print(len(gg))

with open('all.ttl', 'w') as f:
    f.write(gg.serialize(format='turtle').decode('utf-8'))

I ran that and then ran LODE2 on all.ttl but that's timing out too. I think SWEET is just too big!

You might have to template this one out: I have, for large SKOS vocabs, just used a simple Python script with the rdflib + Jinja2 templating modules. Creating a template for Concept, Collection & ConceptSchema for SKOS works well due to the limited set of classes and properties for them. If SWEET uses only a few basic annotation properties, this approach would be reasonable I think (template for Class, Property, Ontology with basic relations.

I may, if you lead, help create such a template-based approach as an alternate to this LODE XSLT approach...

lewismc commented 6 years ago

Thanks for the above @nicholascar This is an issue indeed and yes I agree that both SWEET's large nature and highly connected characteristics seem to be bogging down the service. This is no surprise as ontology editors e.g. Protege, also struggle depending on availability of memory on host machine. I do not have cycles for coding up the LODE templating right now, however we may be able to consider it for a future GSoC or ESIP Labs project... for example ESIP Labs CfP opens in November. Do you envisage the LODE templating approach to eventually replace the XSLT architecture?

On a separate thread, regarding the LODE deployment you have at http://lode2.linked.data.gov.au, are you guys developing that code in an open forum or elsewhere? I've am not seeing any branches in this repository so a bit puzzled as to which code I should be looking at. Thanks

nicholascar commented 6 years ago

OK, I'll see how I go with a templating thing. It would be very different approach to LODE of course but I think there are a couple of reasons to implement something like that. Large ontology size is one reason but another is for allowing people to override parts: we could implement a series of templates that transform certain RDF into HTML but then if someone wanted something done differently - perhaps catering for a particular class in a particular way - then they could either add a template for that or override an existing one. If wedo this, it will be within the Python rdflib world (https://github.com/rdflib) since we mainly use Python for RDF handling and there's quite an ecosystem of tools we're familiar with there.

The code for LODE2 is at https://github.com/CSIRO-enviro-informatics/LODE and we're continuing to develope it as we have a continuing need for standardised ways of documenting many Australian govt. ontologies. We will probably make a fork of LODE2 that has Aust. Govt. logos, style, and so on already embedded. We could go down the templating path here though... No decisions yet! I suspect we will have a hard requirement to choose to either make that Aus Govt fork or make templates by December, based on our project work.

essepuntato commented 6 years ago

Hi all,

Sorry for being late in answering. Great to see that @nicholascar has already addressed all the issues raised. We would need to work, indeed, in integrating the LODE2 version in the main repository, indeed, at certain point. I just need to figure out when I have a bunch of spare time (ah!) for doing so.

np00 commented 5 years ago

LODE (1) is down: "internal server error" :/

http://www.essepuntato.it/lode

nicholascar commented 5 years ago

http://lode2.linked.data.gov.au is working if you need LODE in a hurry, although Silvio's pretty quick at getting his system back up usually!

np00 commented 5 years ago

yeah thanks for the pointer. I found it, but somehow it also does not work for me :( https://github.com/essepuntato/LODE/issues/20

nicholascar commented 5 years ago

@np00 do you mean http://lode2.linked.data.gov.au isn't working for you? Just a bit hard to tell in this thread.

If so, just drop Edmond an email (edmond.chuc@csiro.au) and explain the bug you're seeing and he'll fix it within a day since he actively maintains the tool. We do know about the service maxing out if very large ontologies are loaded.

np00 commented 5 years ago

Correct, somehow lode2 does not work with my ontology. I created an issue for my problem with lode2: https://github.com/essepuntato/LODE/issues/20 I will drop him a mail about it.

lode1 seems to not work at all.