search

BallAerospace / COSMOS

Ball Aerospace COSMOS
https://ballaerospace.github.io/cosmos-website/
Other
360 stars 129 forks source link

xtce_converter doesn't support byte order list #268

Closed severn closed 8 years ago

severn commented 8 years ago

To be fair I haven't done a true in depth look at this, just a quick run through of a single packet xtce file I know is correct from a mission. The xtce_converter doesn't support several elements, which is fine overall -- but the byte order list element is pretty important to us as although we are principally big-endian, this is not always the case for all mnemonics in our database however (and to follow through on it here -- I have to double check whether it is the case in this one packet in which case I can remove the element which we probably generate by default regardless). As the program runs, it produces a fair number of issues along those lines before dying as follows (I'm guess it's the poly exponent which can be 1.0 in straight XTCE 1.1 -- and wonder if you have a variant as this is being changed in xtce 1.2). Anyway thanks for reading...

 Ignoring Unknown: <ByteOrderList>
  Ignoring Unknown: <Byte>
  Ignoring Unknown: <Byte>
/root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:584:in `Integer': invalid value for Integer(): "0.0" (ArgumentError)
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:584:in `block in xtce_process_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:840:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:582:in `xtce_process_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:217:in `block (2 levels) in process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:840:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:216:in `block in process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:215:in `process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/bin/xtce_converter:71:in `<top (required)>'
    from /root/.rbenv/versions/2.2.2/bin/xtce_converter:23:in `load'
    from /root/.rbenv/versions/2.2.2/bin/xtce_converter:23:in `<main>'
ghost commented 8 years ago

Your crash is an easy fix, I was requiring something to be an Integer that is specified as a Float. Already fixed. I'll look into supporting ByteOrderList next. Any other required elements?

ghost commented 8 years ago

ByteOrderList is now supported in the handle_xtce_byte_order_list branch. I'll wait for your response before merging.

severn commented 8 years ago

I have 4 large xtce file to run through it ultimately. I've paired it down to 1 representative but simple-ish packet each. But I can simplify these further still. For example I can remove limits and I can flatten or unroll things like arrays. I can "rename" aggregates -- dotted named mnemonics also as a temporary measure. Ultimately I may want them but for now many items can be worked around.

LocationInContaierInBit is another that comes to my mind. Our translation software calculates and hardcodes the packet entry offsets. Generally though they are all actually "packed" although not necessarily on byte boundaries if their actual telemetered length is < 8 bits. Anyway I could probably strip the location in container bits out on a known good xtce telemetry description, etc... that's likely to be case at least.

severn commented 8 years ago

It's taken me a bit to get back to this. I clearly don't quite know what I'm doing with Ruby but I managed by hand copying paket_config.rb from the branch into the installed Ruby subtree for cosmos -- that I see that the new ByteOrderList is being exercised. I can't really verify that it works but there is less complaining in the unsupported element messages from the converter.

Unfortunately I have lots of other elements it does not like as well:

-EnumerationAlarm & related -AncillaryData (which is fair to not support) -ArrayParameterType -AggregrateParameterType

Those are the top items. But I can make these go away for an initial cut, so I'll do that first...

(note: again this 1 packet from a 1000 packet database, xtce file)

severn commented 8 years ago

I removed the EnumerationAlarmLIst because this is not supported. Now for my 1 packet scenario I'm down to (I believe) the absolute addressing (LocationInContainerInBits) in container parameter entry.

Does the cosmos target DB support addressing its packet/container definition? (I'm poking around for documentation)

severn commented 8 years ago

I suppose this is a new issues... let me do that.

severn commented 8 years ago

Ryan what I get after finally have a chance to look into the COSMOS telemetry database description language is that byte order is only supported on a per packet basis...

Whereas I was really after it being supported on a per mnemonic basis...

Am I missing the obvious?

What's your feeling on difficulty of adding that support: change to ITEM syntax of telemetry (& command but that can wait) to add a byte order list option, change to internal "decom" code to support it... (I assume this exists but I have no idea where it'd be in the code tree)...

Anything else?

ghost commented 8 years ago

Bit endianness is supported per item in the COSMOS config files, though it appears the web documentation is out of date. Each of the ITEM, PARAMETER, etc keywords takes another optional parameter after description of endianness. So just add BIG_ENDIAN or LITTLE_ENDIAN after the description and you can change the endianness of the specific item.

severn commented 8 years ago

Thanks for getting back to me. Someone in between posting my posting this earlier and your comment suggested the same to me based on a command definition they had or had seen. I haven't had time to try it.

I believe this would of course cover %90+ of our byte orderings. Most of the time we strive where we have control for big endian. However I am aware that sometimes we have other byte orders. Little endian would be likely.

Even so I believe this is not guaranteed and historically we've had to support other strange orders -- on a per mnemonic basis.

Given that: both of our in house tool chains support giving the byte order basically as a comma delimited list against the byte length of the mnemonic.

For example a 16 bit mnemonic might be byte order 1,2 or 2,1 and so forth.

Any feeling on where in the code base one might add such? Level of difficulty (for me to try?)

ghost commented 8 years ago

The only other endianness I've ever seen is sometimes when using 1553 you'll see the two 16-bit fields that are each byte swapped individually, but together make up a 32-bit word. (If that makes sense).

Binary Accessor is what does the reading/writing of values and what handles endianness. So that would be the first place to make the change.

lib/cosmos/packets/binary_accessor.rb.

Most of the code is actually written in C at:

ext/cosmos/ext/structure/structure.c

I would rate the change as pretty hard and not worth it. For weird endianness cases you can always write a few lines of Ruby in a conversion to handle the weirdness.

ghost commented 8 years ago

I've just checked in support for LocationInContainerInBits and Arrays into the handle_xtce_byte_order_list branch. If you could do a pull on that branch and try out the changes it would be appreciated.

ghost commented 8 years ago

Closed by pull request #291

severn commented 8 years ago

I will very soon give it a go, sorry for the delay!