search

mdenet / educationplatform-tooltemplate

MIT License
0 stars 0 forks source link

yamtl-m2m-service #4

Open arturboronat opened 1 year ago

arturboronat commented 1 year ago

Hi Will, I created a first attempt of a YAMTL M2M service at https://github.com/arturboronat/mdenet-yamtl

This service executes a basic M2M transformation where one model is received as input and another one is produced. The tool service is flexible: trafo, metamodels and models can be changed. The example is an activity that transforms a class diagram into a relational schema.

The unit test case in com.mde-network.ep.toolfunctions.yamtl_m2m_function is working. Please check that they are working for you locally too.

Some help is needed for configuring the service/activity. The configuration should be fairly similar to the ETL service/activity. In fact, I have followed their configuration but it crashes with this output log. The issues is with the dependencies of the Groovy plugin I am using (org.codehaus.gmavenplus:gmavenplus-plugin). If you look at the project pom, the configuration is fairly simple. It should work on docker too.

Some additional feedback regarding the example project template below:

barnettwilliam commented 1 year ago

Great progress, I'll try it out and take a look at the error.

The reason for the separate activities webserver container is to replicate the intended environment that is the activities will be hosted separately somewhere e.g. github.

Thank you for the feedback I'll look at updating the example and documentation to clarify.

barnettwilliam commented 1 year ago

I found a couple of small things

I haven't yet got to the bottom of the docker issue.

Causes of the [ERROR] No plugin found for prefix 'function' in the current project ... can be not having the maven plugin for google cloud functions (fixed) or not being in the correct directory when running mvn function run ... I checked the docker configuration and the locations in the container image docker run -it --entrypoint /bin/bash mdenet-tool-yamtl and it seem the correct directories are being used.

It should be possible to run the services locally outside the container by 

cd services/com.mde-network.ep.toolfunctions.yamtl_m2m_function
mvn function:run -Drun.functionTarget=com.mdenetnetwork.ep.toolfunctions.yamtlGroovy.RunYAMTL_m2m_groovy -Drun.port=8001

But I get

[ERROR] Failed to execute goal com.google.cloud.functions:function-maven-plugin ... : Could not invoke 
function: com.mdenetnetwork.ep.toolfunctions.yamtlGroovy.RunYAMTL_m2m_groovy

So the function isn't being found. Looking inside the compiled jar files in the local maven .m2 repository for the yamtl_m2m_function there are no class files. One key difference between the yamtl tool is that it is using groovy so @arturboronat can you configure groovy to create the classes? I don't have groovy installed locally but didn't get any errors building using maven so I assume that build was successful and isn't the reason no class files were generated.

It should be possible to use google cloud functions and groovy however it doesn't mention running them locally as we are doing. I suspect if the classes are generated then it will work.

arturboronat commented 1 year ago

Many thanks for this, @barnettwilliam .

This commit solves the issue of adding binaries to the jar.

The configuration of the service is not working yet but I guess that you will be able to make progress on your side now.

barnettwilliam commented 1 year ago

Perfect that helped a lot, the following commit https://github.com/arturboronat/mdenet-yamtl/commit/daa28ebc79343e37c48a149a3314af354b26a554 fixes the issue with the docker container failing to run by automatically downloading the dependencies. So you should be able to try out your activity now.

I've added an issue https://github.com/mdenet/platformtools/issues/13 to apply the same to the mdenet repositories

arturboronat commented 1 year ago

Many thanks for the updates and encouraging words, @barnettwilliam

However, I'm not getting it up and running yet. Could you check the configuration in the last commit and let me know whether the URL I should be using is http://localhost:8080/?activities=http://localhost:8083/yamtl-cd2db-activity.json.

As an observation, I don't know why there are so many attempts to connect to localhost. See the output log.

barnettwilliam commented 1 year ago

Sure I'll try and run the activity, yes that is the correct url.

The yamtl tool port should be 8500 and the console is still part of the epsilon tool so you have to add http://127.0.0.1:8070/epsilon_tool.json to tools. There is an open issue to include common tools by default and remove them from epsilon.

I'll push some fixes for the activity config once I've tested.

I didn't get those Trying to connect to validate messages, how much spare much memory do you have available when running the platform, could it be running low?

barnettwilliam commented 1 year ago

There were a few things:

Fixes in commits https://github.com/arturboronat/mdenet-yamtl/commit/3c0173bdeec0b30196d0736b27ee876ae2352927 and https://github.com/arturboronat/mdenet-yamtl/commit/651f5cde7cc2b3959bb855be9b03bfc469d0d835

My output log after starting and invoking the yamtl function a few times just in case it's useful.

With these changes panels are displayed including the new YamTL one, the tool service is running when the action button is clicked, and the response is displayed. @arturboronat the response from YamTL contains an error about unextpected '\' but looks like it's from YamTL itself.

image

arturboronat commented 1 year ago

Hi @barnettwilliam ,

Fantastic job and congrats for the article!

Although I could not reproduce your error (I'll talk about this later), I managed to fix the issue that you mentioned.

The problem with the character encoding can be resolved in maven, by adding the following code to the root pom.xml: 

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  </properties>

To keep a consistent character encoding in the docker container, I added the following line to the dockerfile: ENV LANG C.UTF-8

I have added additional output in the log to display the REQUEST object at the start of the execution of the function and the RESPONSE object after the transformation. The back-end service is processing the tranformation well after fixing the character encoding issues.

I have pushed the code to https://github.com/arturboronat/mdenet-yamtl to the branch docker-build-fix.

Remaining issues

Output not displayed

I couldn't find any information in the output model or log. When examining yamtl_tool.json and the activity configuration in activities/yamtl-cd2db-activity.json, I'm unsure how to associate the output values with the respective panels. Even after reviewing the Epsilon examples, I couldn't identify these connections. In your example, you showed the error message is displayed in the console panel but this is not the case for me.

Models not displayed graphically

The rendering of metamodels and models as graphical models is not working yet. Is there any pre-built service implementing this graphical rendering?

I have fiddled a little bit with the parameter names in the configuration trying to fix this second problem - I hope not to have broken anything. The last commit shows the changes.

Thanks!

barnettwilliam commented 1 year ago

Thank you and great that you have the YAMTL backend service processing information well, I will have to try it out.

The platform's response handling is an area that needs some work. Currently only text via a ConsolePanel class or a diagram via an OutputPanel class are supported, these are also mutually exclusive.

Output type conversion is also not implemented so tool functions have to generate the desired output as an svg image or text for it to be displayed by the platform. The aim is create conversion functions for common model types so the the output from tool functions can be automatically converted for display in a similar manner that is done for inputs to tool functions.

The relevant documentation https://github.com/mdenet/educationplatform/wiki/Adding-a-Tool#output

Outstanding action for me to update documentation - https://github.com/mdenet/educationplatform/issues/4#issuecomment-1594820812

Sequence diagram of response handling - https://github.com/mdenet/educationplatform/wiki/Design#user-activity

If the output of the YAMTL tool outputs a standard metamodel and model file? it would probably be worthwhile for me to implement the output type conversion and some display conversion functions for common model types. For quick testing, you could modify the YAMTL tool service to generate the desired svg images.

Also if you have any other particular display configuration in mind for an example activity that your creating which is currently unsupported let me know and I'll see if we can support it.

arturboronat commented 1 year ago

Many thanks for your prompt reply and useful pointers.

The transformation is now working in the sense that the target model is displayed in the panel-tmodel. Check the last commit on https://github.com/arturboronat/mdenet-yamtl (branch docker-build-fix).

As an idea for improvement, adding a column description to this table, would ensure that the field descriptions are consolidated, rather than being dispersed across various subsections. I believe this could improve clarity for users.

ConsolePanel vs OutputPanel

As far as I understand a tool function is defined with a single returnType (text in the case of yamtl_tool.json) and this corresponds to output. I tried to use the other response parameters as shown in the response specification but I could not make use of them in the function definition. So far output is working.

Having both output and console text would be helpful in YAMTL, to show how much time a transformation took on the server, for example.

Visualization of models and metamodels

For improved user experience, it might be beneficial to associate an action with the recycle icon, enabling it to convert XMI to an image and vice versa, similar to the functionality found in the Epsilon platform. Ideally, this action should operate independently from the transformation. In other words, while the transformation might output XMI, users should have the flexibility to select their preferred format. Creating separate activities for each output format introduces unnecessary redundancy.

Given a service that generates plantuml from a metamodel and I create the action that calls that service, how would I indicate that the output is of type puml? I would like to reuse an existing panel, which may be tagged with a different type (say xmi). Are there examples available that involve puml or dot?

Some questions

educationplatform-docker-mdenet-educationplatform-1  | 172.19.0.1 - - [22/Aug/2023:17:59:47 +0000] "GET /?activities=http://localhost:8083/yamtl-cd2db-activity.json HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
educationplatform-docker-mdenet-tool-emf-1           | 2023/08/22 17:59:47 [error] 10#10: *1 open() "/usr/share/nginx/html/highlighting.js" failed (2: No such file or directory), client: 172.19.0.1, server: , request: "GET /highlighting.js HTTP/1.1", host: "127.0.0.1:8073", referrer: "http://localhost:8080/"
educationplatform-docker-mdenet-tool-emf-1           | 172.19.0.1 - - [22/Aug/2023:17:59:47 +0000] "GET /highlighting.js HTTP/1.1" 404 555 "http://localhost:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
educationplatform-docker-mdenet-educationplatform-1  | 172.19.0.1 - - [22/Aug/2023:18:06:40 +0000] "GET /?activities=http://localhost:8083/yamtl-cd2db-activity.json HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
educationplatform-docker-mdenet-tool-emf-1           | 2023/08/22 18:06:41 [error] 11#11: *2 open() "/usr/share/nginx/html/highlighting.js" failed (2: No such file or directory), client: 172.19.0.1, server: , request: "GET /highlighting.js HTTP/1.1", host: "127.0.0.1:8073", referrer: "http://localhost:8080/"
educationplatform-docker-mdenet-tool-emf-1           | 172.19.0.1 - - [22/Aug/2023:18:06:41 +0000] "GET /highlighting.js HTTP/1.1" 404 555 "http://localhost:8080/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" "-"
arturboronat commented 1 year ago

I left the demo working yesterday (https://github.com/arturboronat/mdenet-yamtl/tree/docker-build-fix) but I tried to execute it today and I get stuck with the login screen on http://localhost:8080/?activities=http://localhost:8083/yamtl-cd2db-activity.json

I have created a second branch with another activity that gets emfatic metamodels instead of ecore metamodels but I can’t test it for the same reason.

Do you know what may be happening? I'm a bit baffled since this was working yesterday.

Another question: what parameter types are supported when defining functions in https://github.com/mdenet/educationplatform/wiki/Adding-a-Tool#parameter?

barnettwilliam commented 1 year ago

For the activity not displaying

After taking a look at at your activity on branch cd2db-emfatic, there are a couple of typos:

Browsers can cache the files so errors do not always immediately occur. Force refreshing e.g. ctrl + f5 after making changes can help show them.

At the moment there are no user friendly messages displayed and is definitely something that we need to add.

I still get this error reported back after trying to run the activity:

startup failed: 

CD2DB.groovy: 1: Unexpected character: '\' @ line 1, column 31.
barnettwilliam commented 1 year ago

Additionally the last commit removed the epsilon tool "http://127.0.0.1:8070/epsilon_tool.json" under tools, which is needed for the console panel. This should fix the not loading issue. There is an open issue for the general purpose panels to be moved into the platform itself.

Please could you commit the missing CD and relational files for me to test the full example. Thanks

mde-docs commented 1 year ago

Done: https://github.com/arturboronat/mdenet-yamtl/tree/cd2db-emfatic/activities

I have reinstated the epsilon_tool line in the activity configuration file.

barnettwilliam commented 1 year ago

Thanks I have tested and its loading.

On my linux machine I still see the 'unexepected character' error https://github.com/mdenet/educationplatform-tooltemplate/issues/4#issuecomment-1707944927 even with the settings committed in https://github.com/mdenet/educationplatform-tooltemplate/issues/4#issuecomment-1688091668. Running linux2dos on the activity files resolves this and I get the yamtl generated xmi so it seems like the settings https://github.com/mdenet/educationplatform-tooltemplate/issues/4#issuecomment-1688091668 aren't taking effect for groovy in the yamtl toolservivce?

For the display of models I'll fix the panel render button to support standard formats e.g. xmi https://github.com/mdenet/educationplatform/issues/79 and fix the platform response handling so the output field can be displayed in a console with other fields https://github.com/mdenet/educationplatform/issues/80

arturboronat commented 1 year ago

Thanks, Will. It's working on my end too.

The end-of-line (EOL) problem is specific to each operating system. Each OS has its own convention for representing the end of a line in text files: '\r\n' in Windows and '\n' in Linux/Unix. Git can be configured to handle line endings automatically:

git config --global core.autocrlf true    # For Windows
git config --global core.autocrlf input   # For Unix/Linux/MacOS
barnettwilliam commented 1 year ago

That's good! Yes, the .gitattributes file sets the line endings so they should be switched automatically to the current os except for shell scripts which are always LF.

The platform components e.g. tool services are built in linux containers or virtualised linux containers when building on windows, so I'm not sure why the yamtl tool service expects crlf endings.