search

ifml / ifml-editor

The open source IFML editor - Based on Sirius
MIT License
43 stars 33 forks source link

DSL evaluation for IFML - please take a look #29

Closed dzw1999 closed 2 months ago

dzw1999 commented 6 months ago

Hello, greetings from the software engineering group from Beihang University, China. We have been working on the evaluation of Domain Specific Modeling Languages (DSMLs ), the main purpose is to find out the flaws of the design of the DSML. To test our evaluation framework, we evaluated your software design, mainly based on your ecore and odesign files, and found out that even though the overall quality of your software design is very good, there still exists some minor problems. The details are as follows:

rule-based evaluation

Firstly, we did some rule-based evaluation, mainly focusing on the graphic part.

Evaluation of Semiotic Clarity

We've checked your odesign file according to Goodman's theory of symbols[Goodman N. Languages of Art: An Approach to a Theory of Symbols[M]. Hackett publishing, Indianapolis, 1976.], and there are some minor problems that can decrease the semiotic clarity of your design as follows:

  1. Deficit Elements

    There are some elements in your design that are not represented by any graphic symbols.

    Certainly, we know that some elements do not need to be assigned symbols to, but we believe it is best to check whether these elements really do not need symbols or they were just overlooked.

    • Direction
    • SystemEventType
    • ContextVariableScope
    • InteractionFlowExpression
    • DomainModel
    • NamedElement
    • ContentBinding
    • DynamicBehavior
    • Viewpoint
    • ContextDimension
    • BooleanExpression
    • InteractionFlowElement
    • Constraint
    • ConditionalExpression
    • InteractionFlowModelElement
    • ModularizationElement
    • BPMNActivityConcept
    • ContextVariable
    • SimpleContextVariable
    • DataContextVariable
    • DomainConcept
    • FeatureConcept
    • BehaviorConcept
    • BehavioralFeatureConcept
    • UMLBehavior
    • UMLBehavioralFeature
    • UMLDomainConcept
    • UMLStructuralFeature
    • ActivityConcept
    • DomainElement
    • ValidationRule
    • Position
    • UserRole
    • Device

Evaluation of Symbol Similarity

We believe that overly similar symbols are not conducive to users' understanding and use of language. Therefore, we use image features such as SSIM to detect the similarity between the symbols you use. Below are some pairs of symbols that may be too similar:

  1. ifml-editor\plugins\IFMLEditorDesign\img\icons\activation_expression_obj.png & ifml-editor\plugins\IFMLEditorDesign\img\icons\output_port_obj.png
  2. ifml-editor\plugins\IFMLEditorDesign\img\icons\activation_expression_obj.png & ifml-editor\plugins\IFMLEditorDesign\img\icons\module_obj.png
  3. ifml-editor\plugins\IFMLEditorDesign\img\icons\output_port_obj.png & ifml-editor\plugins\IFMLEditorDesign\img\icons\port_obj.png similar symbols-1

LLMS-based evaluation

Secondly, we did some LLMS-based evaluation, mainly focusing on the model part. Generally speaking, we try to make LLM act as a domain expert to provide multidimensional evaluations of your language design.

Model Completeness

We give your ecore design to LLM and ask the LLM to guess which domain your language is designed for. And then we ask LLM to add possible missing elements to your language (perhaps not taken into account in the first version of the design, but can be considered for inclusion in future versions). The results are as follows:

These metamodels belong to the field of Interactive Front End Modeling Language (IFML), with a particular focus on the design of user interfaces and the interaction between users and applications. IFML helps describe the views of an application and the interaction processes between these views, as well as the events and system operations triggered by users. This model language is crucial for detailed design of user interfaces and application usability.

Other key meta models:

Element:

ViewPattern: name;
InputField: isRequired; ValidationRule;
OutputField:
NavigationBar:
Footer:
Leader:
ModalDialogue: isDisassible;
Breadcrumb:
TabView:
Carousel:
DropdownMenu:

Relationship:

ViewContainer (1)<- header ->Header (1)
ViewContainer (1)<- root ->Footer (1)
ViewContainer (1)<- navigationBar ->NavigationBar (1)
Form (1)<- inputFields ->InputField (n)
Form (1)<- submitEvent ->OnSubmitEvent (1)
InputField (1)<- validationRules ->validationRule (n)
ViewComponent (1)<- outputFields ->OutputField (n)
ModalDialogue (1)<- triggeredByEvent ->Event (1)
ViewContainer (1)<- modal Dialogues ->ModalDialogue (n)
Breadcrumb (1)<- navigationFlow ->NavigationFlow (n)
TabView (1)<- tabs ->ViewContainer (n)
Carousel (1)<- items ->ViewComponentPart (n)
DropdownMenu (1)<- menuItems ->ViewComponent (n)

These supplementary meta models and relationships help to more comprehensively describe and model complex interaction designs, especially for highly interactive web and mobile applications. By using richer view components such as modal dialog boxes, navigation bars, footers, and headers, navigation elements such as breadcrumbs, label views, and drop-down menus, as well as input and output fields, the layout and functionality of the interface can be more finely planned, providing a better interactive experience for end-users.

We are not sure if these issues actually constitute a problem, the decision to fix them or not is still up to your team, looing forward to your respond, thanks!

xixiAndpipi commented 2 months ago

Thank you for your thorough examination of our extended IFML. Regarding the first point on the deficit elements, we acknowledge that there are several elements for which we have not yet identified appropriate graphical representations. However, we are uncertain whether elements such as ViewPoint, DomainConcept, and others should have graphical representations, and if so, how these should be depicted. In addition, "InteractionFlowExpression" can be modeled when you right click on the interaction flows. Next. for the symbol similarity and model completeness issues, We think it is a quite interesting work! We would like to replace those figures that are too similar to each other, when we have enough time. For the new elements given by LLMs, we think they can indeed further enhance our extension to IFML! Thank you again for your amazing ideas and practice! D:)

dzw1999 commented 2 months ago

Thank you for your reply!

As for the first point on the deficit elements, we also think that elements like ViewPoint and DomainConcept do not necessarily need to be assigned symbols. We only did rule-based evaluation on the Semiotic Clarity part, and the final decision about whether to add symbols rests with the author.

As for the rest evaluation results, thanks for considering our suggestions.

Wish you all the best!