search

IvS-KULeuven / MTCS-models

Models of the Mercator Telescope Control System, written in the Ontoscript DSL.
GNU General Public License v3.0
0 stars 3 forks source link

Ontomanager generating code. XML file structure #24

Closed jperezpadilla closed 3 years ago

jperezpadilla commented 4 years ago

To be checked on Testing-Ontomanager brach: https://github.com/IvS-KULeuven/MTCS-models/tree/Testing-Ontomanager

Updating the software.coffee file for a subsystem, for example "services" subsystem. The compiling or generating code process works fine in OntoManager, then the new XML file is loaded into the PLC software and compiled without errors. The "problem" is that after starting the PLC, running the new version, it lost the configuration about this subsystem, the config variables related with "services". Entering in the "Config" tab we can see that most those variables are at 0.0 and checkboxes at "False". It seems to be related with the struct of the config section in the XML file generated by Ontomanager. Twincat and PLC XML parser seems to be sensitive to variables' order.

I'll try to explain my tests:

Here a screeshot of TwinCAT after loading mtcs_services.XML file for the two previous tests. The variables' order in the STRUCT is different. Ontomanager-Models-Issue-01

Therefore, could generated code be different, even being generated from an identical source? Comparing two XML files generated by Ontomanager from the same source: TREE VIEW Ontomanager-Models-Issue-02 XSL VIEW Ontomanager-Models-Issue-03

Finally, this is the section of software.coffee file for mtcs_services dealt in this example, that is the source code for generating the XML file. I think the XML file should always take this order to generate the XML file: Ontomanager-Models-Issue-04

Conclusions and Questions: 1) I think the system is sensitive to variables' order in the config structure. I can understand it and this has not be an important issue. We can loose the config if we change the variable's order or inserting new ones.

2) Why is Ontomanager generating the XML file with a more or less random order for the variables inside of a Struct ? I think that it behavior is new.

3) What is different? I don't know what things make sense. I only list things that can be different respect to a year ago,

4) Any other idea will be welcome.

WimPessemier commented 4 years ago

Hi Jesus, thanks for the detailed info. As mentioned in our previous e-mails, I think the Python side is OK: we use lists (not dicts) to store the members of the struct, and therefore the order should be preserved.

However, maybe the problem is on the javascript side. When you ran: image it seems that the order was not preserved, as you mention: image

In the past, the order was always preserved it seems, e.g.

MTCS_MAKE_CONFIG THISLIB, "AxesRotationConfig",
    items:
        motorEncoderOffset          : { type: t_double, comment: "Offset in degrees, of the absolute motor encoder, w.r.t. zero" }
        moveOutOfLimitSwitchDistance: { type: t_double, comment: "How many degrees should the moveOutOfLimitSwitch process try to move?" }
        positiveLimitSwitchInput    : { type: t_int8  , comment: "Which input (0-7) represents the positive limit switch?" }
        negativeLimitSwitchInput    : { type: t_int8  , comment: "Which input (0-7) represents the negative limit switch?" }
        minPositionLimitVirtualAxis : { type: t_double, comment: "Limit the minimum position of the axis to this value in degrees, of the virtual axis" }
        maxPositionLimitVirtualAxis : { type: t_double, comment: "Limit the maximum position of the axis to this value in degrees, of the virtual axis" }
        minPositionLimitPhysicalAxis: { type: t_double, comment: "Limit the minimum position of the axis to this value in degrees, of the physical axis" }
        maxPositionLimitPhysicalAxis: { type: t_double, comment: "Limit the maximum position of the axis to this value in degrees, of the physical axis" }
        velocityLimit               : { type: t_double, comment: "Limit the velocity of the axis to this value in degrees/sec" }
        accelerationLimit           : { type: t_double, comment: "Limit the acceleration (or deceleration) of the axis to this value in degrees/sec^2" }
        minPositionSetpoint         : { type: t_double, comment: "The minimum position setpoint of the axis in degrees" }
        maxPositionSetpoint         : { type: t_double, comment: "The maximum position setpoint of the axis in degrees" }
        maxVelocitySetpoint         : { type: t_double, comment: "The maximum velocity setpoint of the axis in degrees/sec" }
        maxAccelerationSetpoint     : { type: t_double, comment: "The maximum acceleration setpoint of the axis in degrees/sec^2" }
        quickStopDeceleration       : { type: t_double, comment: "Quick stop deceleration, in degrees/sec2"}
        quickStopJerk               : { type: t_double, comment: "Quick stop jerk, in degrees/sec3"}

resulted in:

STRUCT
    ///Offset in degrees, of the absolute motor encoder, w.r.t. zero
    motorEncoderOffset: LREAL;
    ///How many degrees should the moveOutOfLimitSwitch process try to move?
    moveOutOfLimitSwitchDistance: LREAL;
    ///Which input (0-7) represents the positive limit switch?
    positiveLimitSwitchInput: SINT;
    ///Which input (0-7) represents the negative limit switch?
    negativeLimitSwitchInput: SINT;
    ///Limit the minimum position of the axis to this value in degrees, of the virtual axis
    minPositionLimitVirtualAxis: LREAL;
    ///Limit the maximum position of the axis to this value in degrees, of the virtual axis
    maxPositionLimitVirtualAxis: LREAL;
    ///Limit the minimum position of the axis to this value in degrees, of the physical axis
    minPositionLimitPhysicalAxis: LREAL;
    ///Limit the maximum position of the axis to this value in degrees, of the physical axis
    maxPositionLimitPhysicalAxis: LREAL;
    ///Limit the velocity of the axis to this value in degrees/sec
    velocityLimit: LREAL;
    ///Limit the acceleration (or deceleration) of the axis to this value in degrees/sec^2
    accelerationLimit: LREAL;
    ///The minimum position setpoint of the axis in degrees
    minPositionSetpoint: LREAL;
    ///The maximum position setpoint of the axis in degrees
    maxPositionSetpoint: LREAL;
    ///The maximum velocity setpoint of the axis in degrees/sec
    maxVelocitySetpoint: LREAL;
    ///The maximum acceleration setpoint of the axis in degrees/sec^2
    maxAccelerationSetpoint: LREAL;
    ///Quick stop deceleration, in degrees/sec2
    quickStopDeceleration: LREAL;
    ///Quick stop jerk, in degrees/sec3
    quickStopJerk: LREAL;
END_STRUCT

On a side note, it seems that the most of the generated files for the Services subsystem are not in git, See https://github.com/IvS-KULeuven/MTCS/tree/master/MTCS/MTCS/MTCS/Services/Generated/mtcs_services and compare to e.g. https://github.com/IvS-KULeuven/MTCS/tree/master/MTCS/MTCS/MTCS/Axes/Generated/mtcs_axes

So it's hard to know if previously the order was preserved, but most likely it was.

The real problem might be that Ontoscript expects that looping through a json object preserves the order, while in reality it is not guaranteed. When you run 

MTCS_MAKE_CONFIG THISLIB, "ServicesWestControllerConfig",
    items:
        unitID:
            type: t_uint8
            comment: "The address or UnitID of the controller"
        update:
            type: t_bool
            comment: "True to automatically update the setpoint"
        warningMessage:
            type: t_string
            comment: "A warning message (empty = not shown)"
        measurement:
            type: COMMONLIB.MeasurementConfig
            comment: "The measurement config"
        variables:
            type: THISLIB.ServicesWestControllerVariables
            comment: "Configuration parameters"

then the MTCS_MAKE_CONFIG function is called: https://github.com/IvS-KULeuven/MTCS-models/blob/master/coffee/models/util/softwarefactories.coffee#L405 which basically calls the PLC_STRUCT function, which is equal to MkStruct here: https://github.com/IvS-KULeuven/MTCS-models/blob/master/coffee/metamodels/iec61131.coffee#L60

The MkStruct function loops through the items object:

    for name, details of args.items
        #CHECK_ARGS("PLC_STRUCT.items.#{name}", details, ["type", "comment", "initial"])
        HAS ATTRIBUTE(details) name

The for ... of idiom in coffeescript transpiles to for ... in in javascript, which is not guaranteed to preserve the order (according to e.g. https://coderwall.com/p/_kakfa/javascript-iterate-through-object-keys-and-values)

Almost always however, the order seems to be preserved, e.g. check yourself: fill out 

args = 
  items:
    e: "5"
    d: "4"
    c: "3"
    b: "2"
    a: "1"

for name, details of args.items
    console.log(name , " : ", details)

in this online editor: https://repl.it/languages/coffeescript and the result is ordered:

CoffeeScript v1.10
Copyright (c) 2016, Jeremy Ashkenas

e  :  5
d  :  4
c  :  3
b  :  2
a  :  1

Maybe your node.js has changed in the new VM, and does not preserve the order anymore. (Which is entirely correct of course, as ontoscript should not rely on something that is unspecified.)

The only easy solution I see:

  1. Check if the above reasoning is correct (because maybe it is totally wrong :smile:). The easiest might be to add STOP (defined in ontoscript here: https://github.com/IvS-KULeuven/Ontoscript/blob/master/ontoscript.coffee#L209 ) on a new line just behind image, and then copy-paste the last 50 log lines or so here. Is order preserved already in javascript, or not?
  2. If the problem is indeed in the javascript, check if the node.js version has changed. If it is the case, maybe changing the version of Node.js could solve it. (it would be a lousy solution, but still worth a try)
  3. if the problem is indeed in the javascript, then we could order the struct members alphabetically, so that the order is deterministic. This would be a fundamental solution, but it will change all existing configs, and give you a hard time to recover all previous settings (in the XML files of the FTP server).
jperezpadilla commented 4 years ago

Hi Wim, Thanks for your explanation. Now I am pretty sure that node.js was updated together with the Virtual Machine in the last January. Maarten (Software engineer in Leuven) said me yesterday that he updated the Virtual Machine when it was moved to the new computer.

I've checked the version:

administrator@goya:~$ node -v v10.16.3

Having a look in Internet, this version was published on 15-08-2019.

For me the easiest way it to ask for returning to the previous version (node.js, java...) it always worked consistently, because I am not able to update the coffeescript code to provide a predefined fixed or alphabetical order. However, I understand that the best solution is to improve the coffeescript code to guarantee a predefined order. You know... I always prefer the best solution ;-) I would generate again all configurations to save them in the XML files of the FTP server. There is not problem for me at this item. The only one problem is that I don't know about java or coffee script to update the software. I understand that we should modify some functions (MTCS_MAKE_CONFIG, ...) to provide an alphabetical order. Is it a big change? too many lines to add? The critical items are MTCS_MAKE_CONFIG and/or PLC_STRUCT. If it is not too complex and you can guide me with the exact places to add some lines, I could do it or to ask Maarten / Wim De Meester to do it.

THANKS for your help Wim. I'm sorry for disturbing you with all this.

WimPessemier commented 4 years ago

Hi Jesus,

OK, if node.js was updated, that could make sense!

I think you can replace the code 

    for name, details of args.items
        #CHECK_ARGS("PLC_STRUCT.items.#{name}", details, ["type", "comment", "initial"])
        HAS ATTRIBUTE(details) name

with 

    sorted_keys = Object.keys(args.items).sort()
    for key in sorted_keys
       HAS ATTRIBUTE( args.items[key] )  key

but I'm not 100% sure, I cannot test it out.

I hope the problem is really what we think it is. Because I still find it strange that Node.js would all of a sudden not respect the order of a json object anymore (even though it is unspecified in reality), maybe I'm totally wrong and the problem is something different.

But the above code could help to quickly verify this. If the order in the generated code is alphabetically now, then I guess it is solved and it makes sense to re-generate all code. Alternatively, as said above, you can add STOP behind a MTCS_MAKE_CONFIG call, and then check the logging to see if the order in which the javascript is handled is the same as in the generated code (which would prove that the problem is already present in the javascript part of the procedure).

Also it makes a lot of sense to add all generated files of mtcs_services to the git repository, because currently many files are missing there, it seems! (My mistake at that time, for sure :wink:)

And you're not disturbing at all of course, I still want the telescope to be in perfect shape, just like 3 years ago, this has not changed at all :wink: If the above code doesn't solve the problem, let's continue to investigate.

jperezpadilla commented 3 years ago

The root cause of the problem was found by Wim Pessemier. It was in Ontoscript: https://github.com/IvS-KULeuven/Ontoscript/pull/1 It was pretty subtle, and very different than what we thought.

Some background info: When ontoscript (the ontoscript.coffee “compiler” from the Ontoscript repository) converts the models (like mtcs/dome/software.coffee) to a standard file format (like mtcs/dome/software.jsonld), it inserts a number (called “counter”) that is continuously incremented. For example, the first attribute of ServicesTimingConfig in the model gets 1234, the next one gets 1235, and so on. This number is then used later on, when the jsonld file is queried, to know the correct order.

The problem was a bug in ontoscript: it inserts the numbers using the “counter” term of the ontology (the “vocabulary”) “http://mercator.iac.es/onto/metamodels/ontoscript” instead of “http://www.mercator.iac.es/onto/metamodels/ontoscript”. But the Ontomanager only knows the “counter” term of “http://www.mercator.iac.es/onto/metamodels/ontoscript”, so when it queries for the “counter”, it cannot find anything. Because the counters cannot be retrieved from the model, the order is random. (I think by coincidence this problem was not discovered before, because the wrong URL was over-written by the correct one due to the random order inside some Python dict. I can explain in detail if you like, but I would need to investigate more.)

So if we simply fix the 4 characters (=add the missing www.http://www. In the ontoscript.coffee file), and re-build all models with it, and then re-generate the PLC code, then everything will be fine. The order of the STRUCT members will be the same as how they are defined in the coffeescript models. Which is the best solution (better than alphabetical order).

Below this mail is the full procedure on how to fix it.