Extend GeppettoJupyter so that we can load a Geppetto Project with a Geppetto model coming from file

[tarelli]

We already have unit tests that dump a snapshot of the model to an XMI. The XMI needs to be a complete one which doesn't require therefore any model interpreter or simulator.

• [ ] Understand how to extend the Python backends to load a Geppetto project
• [ ] Understand how to extend the Python backends to load a Geppetto Model (XMI) when linked from a Geppetto project
• [ ] Test the sample can be loaded when using the Jupyter backend using load_project_from_url and pointing to the sample (this file needs to be updated https://github.com/openworm/org.geppetto.frontend.jupyter/blob/development/src/geppettoJupyter/__init__.)

For both last points refer to the implementation of the Java backend https://github.com/openworm/org.geppetto.frontend/blob/04629f881ade99f3dcfd326bfebcb09d2ed560d5/src/main/java/org/geppetto/frontend/controllers/ConnectionHandler.java#L162

[tarelli]

@LordKrabo Full XMIs for testing https://gist.github.com/tarelli/0d461b3e4a429e25225542e7bd11cff0 Geppetto project sample https://github.com/openworm/org.geppetto.samples/tree/development/UsedInUnitTests/SingleComponentHH Use an XMI from the first link rather than the XMI in the sample. This is how you load a geppetto project for instance: http://127.0.0.1:8080/org.geppetto.frontend/geppetto?load_project_from_url=https://raw.githubusercontent.com/openworm/org.geppetto.samples/development/UsedInUnitTests/SingleComponentHH/GEPPETTO.json.

This which works with the Java backend doesn't work with the Python backend since the code is not there yet to parse the Geppetto project and the Geppetto model. pygeppetto would have to be used to parse the XMI. The JSON can be parsed using standard Python JSON parser.

[LordKrabo]

Right, so at this stage I have been working on trying to import the XMI samples into pygeppetto. @aranega has written a nice guide on how to import these XMI samples and create new ecore modules but there are some things which are still missing so I have just been modifying pygeppetto along the way.

[aranega]

@LordKrabo @tarelli I've finally pushed a new version on the developement branch of PyGeppetto (I'm really really sorry about the delay). This new version automatically register all the required metamodel URIs (master and develop URI) and uses the last PyEcore version (0.1.3).

@LordKrabo Could you tell me what's missing? I will try to enhance/fix the code generator and/or PyEcore if it's required. Thanks for all the tests!

[LordKrabo]

@aranega @tarelli Sorry guys I've been inactive on this one. I kept on chasing a bug around for the metamodel registration of the URIs and I think there might be a missing double type in there somewhere which was creating a NoneType exception. I'm sorry for my ineptitude. Did you want my to commit my code with the fix? Thank you.

[aranega]

@LordKrabo It could be an issue in the code generator, I have to check, could you commit/push your modifications on a new branch from develop (with a problematic XMI so we had it to the test suite)? This way, it will be easier to merge your modification, and for me to spot the issue in PyEcore.

[aranega]

@LordKrabo I'm working on some new features for PyEcore (possibility to delete elements, proxy and stuffs) and I've checked your branch of the pygeppetto for the xmi loading and your modifications about the missing types. These types were missing in the v.0.1.2 but I've added them in the v.0.1.3 version of PyEcore :). I've upgraded the PyEcore version in the pygeppetto setup.py in the development branch so it uses the v.0.1.3 instead of the other one. Perhaps this would this fix all of your issues?

[aranega]

@tarelli @LordKrabo Hi there, things are going pretty fast for PyEcore with the intervention of a very active contributor. We are currently discussing which Python version we should target aranega/pyecore#6. The current PyEcore targets Python >= 3.3 and we would like to target at least Python 3.4 (or higher). The question is, which Python are you using guys? I would prefer to keep a strong compatibility with PyGeppetto so you are not "forced" to migrate to a newer Python version (unless you want it of course).

[LordKrabo]

Hi @aranega I am currently using python 3.4 on my version.

Out of interest I am also trying to implement the websocket functionality from org.geppetto.frontend.jupyter into this version of Python so that we can import XMIs from hyperlinks. At the moment I am doing this by reimplementing the code in geppettoJuypter in init.py model. Is this the correct way of doing things?

[aranega]

@LordKrabo Great, I think we will target Python 3.4 as smaller Python version.

Regarding your XMI import, I never played with websockets in Python so I don't know if there is special constraints or stuffs, but I implemented a special kind of URI, the HttpURI in PyEcore that could fetch XMI from a remote location like this (but at the moment, it cannot write at this remote location, only read):

from pyecore.resources import ResourceSet
from pyecore.resources.resource import HttpURI

rset = ResourceSet()  # We create first an empty resource set
rset.get_resource(HttpURI('http://www.mylocation.org/myproject/myxmifile'))

If it does not help you, you can implement your own URI almost like the HttpURI. In PyEcore, the URI is responsible for the creation of a file-like object as input stream and a file-like object as outpout stream you can check the HttpURI code to see how to create your own.

Could this help you?

[LordKrabo]

@aranega Thank you.

With the HttpURI implementation, would something like this help with writing to a remote location?

http://stackoverflow.com/questions/27893929/python-appending-file-remotely

[aranega]

@LordKrabo Regarding the link you post, it seems that the solution relies on fabric which proposes facilities to handle ssh remote connections/actions. It really depends on your server/protocol and stuffs. With the current state of the lib, the solutions I could think of are:

• finding a library that gives you a kind of abstract FS representation and writing a custom URI with a custom create_inputstream and create_outstream to match your library.
• using the HttpURI to read the XMI from a remote location and, when the model is modified, saving it in a tmp file using my_resource.save(URI('my_tmp_local_location')). Once the model is saved in a local XMI, you can send it to the server and destroy the tmp file.

Once again, it really depend on your architecture, from/to where you want to read/write files... etc

[tarelli]

@LordKrabo I think for what you need to do this should be enough http://stackoverflow.com/questions/15138614/how-can-i-read-the-contents-of-an-url-with-python

[aranega]

@LordKrabo @tarelli If the final goal is only to read from an URL, the HttpURI should be enough (it uses the urllib and the same code than the one you linked).

[LordKrabo]

Okay @aranega @tarelli I have tried restructuring this code as follows:

  class WebSocketHandler(tornado.websocket.WebSocketHandler):
        def open(self):
        # 1 -> Send the connection
       self.write_message({"type":"client_id", "data":"{\"clientID\":\"Connection1\"}"})
        # 2 -> Check user privileges
         self.write_message({"type":"user_privileges", "data":"{\"user_privileges\": \"{\\\"userName\\\":\\\"Python User\\\",\\\"loggedIn\\\":true,\\\"hasPersistence\\\":false,\\\"privileges\\\":[\\\"READ_PROJECT\\\",\\\"DOWNLOAD\\\",\\\"DROPBOX_INTEGRATION\\\", \\\"RUN_EXPERIMENT\\\", \\\"WRITE_PROJECT\\\"]}\"}"})

def on_message(self, message):
    jsonMessage = json.loads(message)
    if (jsonMessage['type'] == 'geppetto_version'):
        #Where do we get the geppetto version from?
        self.write_message({"requestID":jsonMessage['requestID'],"type":"geppetto_version","data":"{\"geppetto_version\":\"0.3.5\"}"})
    elif (jsonMessage['url'] != 'null'):
        HttpURI(jsonMessage['url'])
    elif (jsonMessage['type'] == 'null'):
        self.write_message('Request not found')

def on_close(self):
    pass

Is this close to what you were thinking?

[aranega]

@tarelli @LordKrabo Hi, I released a new version of pygeppetto in the developement branch which is generated from the new code generator proposed by a PyEcore contributor. This new version introduces named parameters to each elements. This way, instances creations are way more practical and less verbose:

import model as pygeppetto

flib = pygeppetto.GeppettoLibrary(name='mylib')
root = pygeppetto.GeppettoModel(name='MyGeppettoModel', libraries=[flib])

Obviously, the 'old' way of creating elements (assigning attribute/reference value on separate statement) is still compatible. In the same time, I added some tests for this new way of handling elements and I updated the pygeppetto version from 0.0.1 to 0.1.0 (if that's ok with you, otherwise, I will revert it).

Also, the metamodel are now directly registered to the PyEcore global_registry in the developement branch as well as the metamodel developement and master URI. This also ease the code required for reading an XMI file.

Tell me if all of this fits you.

Also, I'm sorry, I had very little time this past few weeks (months), so I wasn't able to provide the best support for all of this.

[tarelli]

@aranega that's great, thanks so much for this!

[LordKrabo]

@aranega @tarelli I'm sorry to be a nuisance about this but I appear to be hitting a bit of a brick wall. I have tried importing the xmi files in the test repository here (https://github.com/openworm/org.geppetto.samples/tree/development/UsedInUnitTests/balanced) into org.geppetto.frontend_jupyter using pygeppetto, and I keep running into this error:

  tree = etree.parse(self.uri.create_instream())
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
  File "src/lxml/parser.pxi", line 1832, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118116)
  File "src/lxml/parser.pxi", line 1852, in lxml.etree._parseFilelikeDocument (src/lxmL/lxml.etree.c:118399)
  File "src/lxml/parser.pxi", line 1747, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:117187)
  File "src/lxml/parser.pxi", line 1162, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:111914)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105671)
  File "https://github.com/openworm/org.geppetto.samples/tree/development/UsedInUnitTests/balanced", line 35
 lxml.etree.XMLSyntaxError: Specification mandate value for attribute data-pjax-transient, line 35, column 91

I suspect it is possibly an issue with the way that I have called rset inside init.py for geppetto (as it is occurring across all of the XMI files that I am using) but I can't find any information on a data-pyax-transient error.

[aranega]

@LordKrabo @tarelli The error seems to come from the XML parsing. The 'Specification mandate value for attribute' supposes that there is an XMI attribute without value, which is not allowed by the XML specification. Could you push or give me a pointer to your current code? I will try to take a look at it. Also, the XMI import is done using the HttpURI dedicated URI? If so, perhaps the XMI code is not 'read' as it should? I will also look at the XMI tests to see from where the issue could come (in the error log you gave, it looks like the issue is located at the line 35, column 91).

[LordKrabo]

Hi @aranega This was the gist that I created to replicate the code that I was using.

https://gist.github.com/LordKrabo/344b9413ef6b0406bf4eed75303d4c7d

The strange thing is that none of the XMI files I was using to test were 35 lines long.

[LordKrabo]

I also placed the pygeppetto repository inside org.geppetto.jupyter_frontend and was also attempting to call rset.get_resource from inside this repository.

[aranega]

Indeed, all the XMI I saw are really small. In the gist you've posted, I only see the ResourceSet defintion, but not the get_resource call, could you post the snippet where the get_resource is called? Also, the HttpURI must be used with the get_resource as it creates a "link" towards the resource, but it does not access it directly, only when the resource is accessed. The basic schema would be something like that:

rset = ResourceSet()  # we create the resource set that we must keep in order to keep the relationship towards the loaded resource
resource = rset.get_resource(HttpURI(url))  # the HttpURI creates a link towards the resource while the get_resource read/decode it
model_root = resource.contents[0]  # if everything's fine, the model root can be retrieved from the loaded resource

[LordKrabo]

I tried replicating this from the python command line, but unfortunately ran into the same error:

  rset = ResourceSet()
  rset.get_resource(HttpURI('https://github.com/openworm/org.geppetto.samples/blob/development/UsedInUnitTests/SingleComponentHH/GeppettoModel.xmi'))

[aranega]

Ok, I think I get it, I suspect the issue coming from the fact that the link points to the github page instead of the raw resource (the raw content). If we look at the source code of the github HTML page for https://github.com/openworm/org.geppetto.samples/blob/development/UsedInUnitTests/SingleComponentHH/GeppettoModel.xmi, we have

<meta name="request-id" content="9378:7448:384B832:54116D1:58FC9FEC" data-pjax-transient>

Instead, you should use the raw resource content located at this adress: https://raw.githubusercontent.com/openworm/org.geppetto.samples/development/UsedInUnitTests/SingleComponentHH/GeppettoModel.xmi

so, the call becomes:

rset = ResourceSet()
rset.get_resource(HttpURI('https://raw.githubusercontent.com/openworm/org.geppetto.samples/development/UsedInUnitTests/SingleComponentHH/GeppettoModel.xmi'))

[LordKrabo]

Okay I think we're getting further. This might be down to how I have implemented the connection between pygeppetto and frontend_jupyter though...

 Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/james/geppetto-sources/org.geppetto.frontend.jupyter/src/jupyter_geppetto/pygeppetto/pyecore/pyecore/resources/resource.py", line 28, in get_resource
resource.load()
 File "/home/james/geppetto-sources/org.geppetto.frontend.jupyter/src/jupyter_geppetto/pygeppetto/pyecore/pyecore/resources/xmi.py", line 24, in load
 modelroot = self._init_modelroot(xmlroot)
 File "/home/james/geppetto-sources/org.geppetto.frontend.jupyter/src/jupyter_geppetto/pygeppetto/pyecore/pyecore/resources/xmi.py", line 71, in _init_modelroot
eobject = self.get_metamodel(nsURI).getEClassifier(eclass_name)
File "/home/james/geppetto-sources/org.geppetto.frontend.jupyter/src/jupyter_geppetto/pygeppetto/pyecore/pyecore/resources/resource.py", line 233, in get_metamodel
raise KeyError('Unknown metamodel with uri: {0}'.format(nsuri))
KeyError: 'Unknown metamodel with uri: https://raw.githubusercontent.com/openworm/org.geppetto.model/development/src/main/resources/geppettoModel.ecore'

[aranega]

So, it only misses the metamodel registration. This is quite strange because it is registered in the __init__.py from pygeppetto. To do it manually (while I search why the registration is not perfomed automatically), you can do this:

import model as pygeppetto
from pyecore.resources import ResourceSet, global_registry
from pyecore.resources.resource import HttpURI

# We register the metamodel in the global registry (could be in the ResourceSet)
global_registry['https://raw.githubusercontent.com/openworm/org.geppetto.model/development/src/main/resources/geppettoModel.ecore'] = pygeppetto

rset = ResourceSet()
rset.get_resource(HttpURI('https://raw.githubusercontent.com/openworm/org.geppetto.samples/development/UsedInUnitTests/SingleComponentHH/GeppettoModel.xmi'))

I'm searching why the metamodel registration is not done automatically.

[aranega]

I search a little bit, and the auto-registration is actually performed. The issue I had came from ipython which was having trouble with my virtualenv (ipython does not seems to deal very well with virtualenvs). To fix the issue, I install ipython from pip inside my virtualenv, reload it (exit it then activate it again), and everything's working without the manual metamodel registration. With the default python interpreter I had no issues at all.

So:

• if you have an issue using ipython in a virtualenv, tries to install it using pip inside your virtualenv, then reload your virtualenv.
• if you have an issue using python with or without a virtualenv, please check that you are using the pygepetto version 0.1.0 and if so, please check that in the __init__.py in model if the last lines are related to the metamodel registration in the global_registry.

Also, I noticed that you copied PyEcore directly in the PyGeppetto code, you can avoid that if you use pip as PyEcore is available through Pypi. A better management of this dependency would be to directly install it:

$ pip install pyecore

so you can manage upgrades simply by typing:

$ pip install pyecore -U

This way of dealing with the dependency would change

# before
from .pygeppetto.pyecore.pyecore.resources.resource import ResourceSet, HttpURI

# after
from pyecore.resources.resource import ResourceSet, HttpURI

[LordKrabo]

Right, from what I can tell I'm not running pygeppetto in the virtualenv:

import sys
if hasattr(sys, 'real_prefixc'):
 ... 
File "<stdin>", line 2

^
IndentationError: expected an indented block

Which I'm guessing is a good thing? Also, I managed to import the XMI file, but with this error (but I think I know what causes it, so all good).

AttributeError: 'NoneType' object has no attribute 'getEClassifier'

I'll push those changes tomorrow.

[aranega]

@LordKrabo That's great new! If you're not in a virtualenv, I guess the pyecore version that you have directly embedded is probably outdated. That could explain the metamodel non auto-registration in the global_registry.

[LordKrabo]

@aranega I'm really sorry for going dark on this for a while, I've kind of gotten lost with my own research. Anyway, I thought I might have another crack at this. For whatever reason, the metamodel_registry is still not being populated. I have tried changing the registration:

 self.resource_set.metamodel_registry[self.prefixes[prefix]]

To

 self.resource_set.metamodel_registry.get(self.prefixes[prefix])

Following these instructions: https://wiki.python.org/moin/KeyError

But I keep running into an str type error. I'm so sorry, I'm terribly lost here.

[aranega]

@LordKrabo Don't worry, we will found the root of the issue. First, could you tell me if PyEcore is installed using pip or do you have the PyEcore source code direclty copied where your own code is?

[LordKrabo]

PyEcore is installed using pip. I tried modifying the embedded Pyecore folder, but I ended up getting nowhere so I tried using the version available on pip instead.

[aranega]

I've published new versions so, update your pyecore version:

$ pip install pyecore -U 

Just to be sure we are on the same version, if there is a bug in PyEcore, or a missing stuff, it will be easier to indentify and patch it. Then check in the model/__init__.py that you have this from line 121 to 132:

# Manually register all the URI and master URI in the global registry
# This registering is performed outside the previous 'for' to ease future
# code merging (in case of new metamodel versions)
geppetto_master_uri = ('https://raw.githubusercontent.com/openworm/'
                       'org.geppetto.model/master/src/main/resources/'
                       'geppettoModel.ecore')

global_registry[nsURI] = model
global_registry[geppetto_master_uri] = model
for subpack in eSubpackages:
    global_registry[subpack.nsURI] = subpack
    global_registry[geppetto_master_uri + '#//' + subpack.name] = subpack

[LordKrabo]

That last 11 lines of code were not in my version, so I've added them in and also imported global_registry and that appears to have resolved the problem!

<pyecore.resources.xmi.XMIResource object at 0x7efed985b2e8>

Thank you so much @aranega! Should I push that change I made to model/init.py?

[aranega]

\o/ I'm glad everything loads ok now!

These lines are already commited/pushed in the repository (on the master and develop branch), perhaps your pygeppetto version is out of sync with the github repository?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.