Access to file raw contents (via a GET request) in activities

[arturboronat]

Problem

A model transformation is in production on a system deployed somewhere. The input/output models are stored on a github repository.

When this transformation needs to be demoed on the MDENet platform, the files need to be copied to a directory on the same repository where the activity configuration is found. Unfortunately, this setup forces the user to duplicate software artifacts, doubling the maintenance effort when the tranformation needs to be changed.

Solution

Allow activities to fetch raw file contents via a url via a GET request wherever file: is used.

The advantage of this approach is that the software artifacts (trafos and models) can be maintained in their native environment (say as part of a CI pipeline) and they can also be used for demo purposes via the MDENet Education Platform.

[jgsuess]

For EMF the built-in resource framework allows the use of URLs to load models. This seems like duplicating functionality. Epsilon also provides a driver for CDO, which is a model storage layer. CDO can also handle transactions in MDE workflows. I would avoid reinventing.

[agarciadom]

I assume that with file:, you mean when the activity JSON/YAML has a file: key? It may be best to instead use an url key, to clearly distinguish the intent of the activity author.

I'm not sure if we want to overcomplicate the platform by having it interpret various kinds of URLs. It may be best to leave it to the tool function (e.g. allowing tool functions to take in parameters of type url and interpreting them accordingly). Epsilon (through EMF) can load arbitrary URLs.

[szschaler]

I agree we don't want to reinvent wheels here. However, some minimal support would be required from the platform. Specifically, it must be possible to provide a URL as the value of a parameter and not just a current panel.

I think all that's needed on our side is to allow action functions to be called either with the file contents (which would come from a panel currently visible to the user) or a URL provided directly in the activity specification. Not every action function will know how to handle this, but this can be managed as part of the declaration of these functions in the tool specification. How the tool decides to handle such a URL would then be up to it. We wouldn't even attempt any conversions as we would do for panel contents and we would certainly not attempt to interpret the URL in any way.

At some point, we may offer a light-touch rewrite functionality (probably based on standard JSON/YAML variable interpolation) so that we can use URLs that are only generated as part of an activity -- for example, when a student edits an Xtext grammar and generates the IDE from it, we may then want to be able to refer to the generated metamodel so we can show it graphically to the student. The URL of the meta-model file depends on URL we generate for the Xtext editor (which will be different for each student and for each session), so needs to be partly interpolated from information available only in the current browser session.

[szschaler]

BTW, note that this isn't currently on the high-prio list yet :-) Given the growing number of issues, we should perhaps start being a bit more explicit and defining release milestones or similar to be able to be clearer about the prioritisation we're doing...

[arturboronat]

Actually, there is not much to do in this feature proposal. After looking at the code in detail, it seemed that the platform already has this functionality - I tried it and it works. By the looks of it, it seems that I was not the only one unaware of this 😉

To close this issue, the documentation of Panel needs to be updated to explain the possible values of the key file: in the yaml configuration of an activity. At the moment, it only mentions a relative path but it can also be a url. I wish I'd known this earlier!

An example of a configuration panel with urls below:

panels:
- id: panel-yamtl
  name: Transformation(YAMTL)
  ref: yamtl
  file: https://raw.githubusercontent.com/yamtl/examples/master/yamtl-linkedlistreversal/src/main/groovy/linkedListReversal/ReverseLinkedList.groovy
- id: smm-panel-composite
  name: Source Metamodel
  ref: composite-panel
  childrenPanels:
    - id: panel-smm
      name: EMFatic
      ref: emfatic
      file: https://raw.githubusercontent.com/yamtl/examples/master/yamtl-linkedlistreversal/model/LinkedList.emf
    - id: panel-smm-diagram
      name: Diagram
      ref: plantuml

[jgsuess]

It is worth noting that the [re-writing facilities are also available in the EMF resource framework](https://download.eclipse.org/modeling/emf/emf/javadoc/2.5.0/org/eclipse/emf/ecore/resource/URIConverter.html#getURIMap()), they just need exposing. So are features for protocol handling and configuration. I know EMF is obtuse in its description, but the features are there.

[szschaler]

@arturboronat that's good but only works if you're showing the relevant file in a panel. At the moment, I think there's an assumption that when we call an action function all parameters are sourced from a panel. I guess it would be helpful to be able to also use a URL directly as a parameter, leaving it to the action function to work out what to do with it -- for example using standard EMF URL management if that's appropriate for that tool.

@jgsuess I think EMF URL rewriting won't help with the case I've described above, because this is about a URL that's in an activity specification and about interpolating information that's only available in the browser. Doing a simple JSON interpolation is straightforward enough, though.

[arturboronat]

@szschaler An activity is a UI abstraction. The functionality proposed above bypasses the UI and it should be discussed in a separate issue. There may be a relevant use case for it but this is not what I had in mind in this issue.

@jgsuess Not all software artifacts handled in panels on the MDENet platform are Eclipse/EMF-centric. While Eclipse is a powerful and widely used IDE, it's important to consider the broader context when developing tools or applications. The 'platform:' URI scheme is specific to Eclipse and tightly couples resources to the Eclipse workspace and plugins. This means that any tool or application that relies on 'platform:' URIs will inherently have a dependency on Eclipse, limiting its portability and usability in non-Eclipse environments.

[szschaler]

@arturboronat I don't think of activities as purely UI abstractions. For me, they are the central abstraction I as a teacher use to define what I want my students to do. Some of this is about the UI. But a lot is about which resources and what the links between those should be. Not sure what this means for this issue?

[arturboronat]

@szschaler We may be talking at different levels of abstraction: your explanation at CIM level (abstract intent of an activity for students), whereas mine is at PIM level (actual design of an activity where we configure button actions, panel contents and their composition). Currently, the MDENet platform is implemented as a JS front-end that follows the PIM mentioned above faithfully.

It is worth considering a refactoring where the UI viewpoint is decoupled from the core domain abstraction at PIM level in order to facilitate a more general design. However, within the timeframe that we have, it may be worth agreeing on the scope of the project, and prioritise accordingly.

This issue was, however, about getting contents for activity panels from remote sw artifacts and that is already feasible.

[szschaler]

Understood. Does that mean, we can close this issue?

[jgsuess]

@arturboronat EMF is not dependent on Eclipse. And its resource resolution mechanism is not either. Please do read the documentation thoroughly. Hence, I find your argument lacking and superficial. While I appreciate that there are models in other technological spaces, the question is how the platform is going to deal with them. It would be good to make the top integrator explicit. Presuming that this is EMC, it also uses a URI-based resolver, so, again, there is no need for an additional mechanism.

The reason I am arguing so vigorously here is that reinvention is exciting, but it dilutes development energy available. Reading documentation for existing features and doing reuse is not so exciting, but it allows more progress.

With regard to considering "the broader context when developing tools or applications." Broadness is often an excuse for playing with the technology one fancies. I find that "generic" and "broad" are often veiled synonyms for "one size fits none". A tell-tale sign of this is an increase for requests for broad features for a product designed for a specific need. Good projects can grind to a halt this way.

Sorry for the extended and somewhat of-target remark, but I felt uncomfortable with the factual nature of the statement. I think there is no harm, given this issue is being closed.

I am quite certain that I would be able to cover similar platforms requirements with a combination of Che and RAP. Not pretty, certainly creaky and dated, but about right for me.

@jgsuess Not all software artifacts handled in panels on the MDENet platform are Eclipse/EMF-centric. While Eclipse is a powerful and widely used IDE, it's important to consider the broader context when developing tools or applications. The 'platform:' URI scheme is specific to Eclipse and tightly couples resources to the Eclipse workspace and plugins. This means that any tool or application that relies on 'platform:' URIs will inherently have a dependency on Eclipse, limiting its portability and usability in non-Eclipse environments.

[arturboronat]

@jgsuess The platform: URI scheme, used in the URI rewriting facilities available in EMF, according to the documentation link you sent above is Eclipse specific. At least this is what the authors say in the book "Steinberg et al. EMF. 2nd edition. page 83". If you found a way of using this rewriting mechanism without Eclipse dependencies, please share it with a concrete example.

I am using EMF models in YAMTL model transformations and YAMTL is not Eclipse-dependent, so I have experience with using EMF without Eclipse dependencies.

On the other hand, at the moment, the MDENet platform is a service-based architecture that allows for the loose integration of MDE tools. So far it is working without having to choose a top integrator technology. I don't see the benefit of constraining the current flexibility we have.

[arturboronat]

Understood. Does that mean, we can close this issue?

@szschaler Once the documentation is updated (as mentioned above). This should be a quick fix.

[jgsuess]

@arturboronat I am not disputing that platform: is Eclipse-specific, however, the EMF plugin when loaded without OSGI emulates it, for once. Secondarily, the EMF libraries are using file: and http: for input streams. As for providing you proof, rest assured that I have in my previous company implemented such a mechanism with custom schema remapping.

I am happy that you are implementing Yet Another Model Transformation Language, it will go well with Yet Another Personal Platform (YAPP?) design. I prefer reuse and hence may not use YAMTL. Thanks for the consideration.

[arturboronat]

Thank you, @jgsuess, for your perspective. I would have appreciated a concrete example from your experiences.

While I respect your preference for established tools, it's important to note that YAMTL, developed through comprehensive research, offers unique features not found in other model management tools. These capabilities are critical for certain applications and address specific needs. It's crucial to understand that diverse needs often require diverse solutions, and new technologies like YAMTL deserve fair consideration based on their merits. Although our views differ, I value this exchange of ideas and the opportunity for constructive dialogue that respects our professional endeavours.

[szschaler]

I've added a note to the documentation and am closing this issue now.